If you know what you want your characters to experience, what matters to them, and what kinds of conflict your story needs, you have somewhere to start.
And when you get stuck on what to build next, look at what you know about the story. There are always #worldbuilding ideas hiding there.