I think it helps that I've been a pilot since I was a teenager. Spend some time where you can see a third of a state at a time you'll just develop this sense. You get a bigger picture of how things are oriented relative to each other that's sort of like, wherever you are in your home, you can probably work out which way the road outside goes, likely parallel to one of your walls. I can do that over much greater distances. If you've ever stood in the middle of a parking lot in a strip mall, and gone "the highway is over there, the Belk is over there, the J.C. Penney is that way, the furniture store is down on that end and I know the Red Lobster is just on the other side of it though I can't see it from here," I can do that with the major cities in my state.

Orienting yourself if you've gotten turned around is another habit to build up. Yes "the afternoon sun is in the West" but also if you're in the Northern hemisphere, your shadow will point North at noon. I also have a pretty good picture of the highway system in my head and can orient myself by knowing the general heading of a nearby highway.

From both my time as a pilot and as an amateur radio operator I'm familiar with the various towers across the state. I've used those to work out my approximate location and heading both in the air and on the ground. In medium sized cities often there's a city center with a few tall buildings that can be seen for several miles around, orienting yourself to them can help you develop a sense of direction. I've started doing that almost subconsciously.

Now if I were to wake up in a cave my gyros would be tumbled until I managed to get out. I don't have an actual built-in compass. But it wouldn't take me long to orient myself seeing how the daylight hit the cave entrance.