I think the plot governs what the fantasy world should be. That and a good map! The details enhance the story with twists, adventure and everything else.