What your characters are wearing might seem like a small consideration in the grand scheme of your story—and you may very well be correct. But there are many books where fashion is integral to the world-building. One of my favorite examples of this is how the difference in clothing between the districts and the capital in The Hunger Games series is a tangible way to show the corruption of their government and differences in social status.
But like with all world-building, you don’t want to get trapped into hours and hours of designing clothing for your characters and not actually writing their story. To avoid this, it might be helpful to focus on the