I have a lot of indirect experience/knowledge with caves. I don't have to be one of the people who directly explored one to know one of the first things one learns about caves is how ill-suited for wandering around they are. Slopes can change on a dime, it's incredibly rocky inside, and they fill with whatever falls from the sky. Imagine hiding in one of these, as someone with less intelligence than us, and not expecting to even stub a toe, let alone fall or suffocate. I assume it must be a concern because all the cave hobbyists I know (even if I only know a couple) say they have to split up based on their physical skills (so one chooses a slope-free while the other a rain-free one as would be the case for me as someone who never learned), though I admit I'd be amused if cavepeople didn't ignore these slopey checkpoints and instead it caused them to make some inventions.
Another thing that sticks out to me is the rule against fire. When cave explorers (not me) venture in, the most important rule of all time they learn is that it's an absolute cardinal sin to light so much as a match in a cave, let alone a torch (in contrast to Indiana Jones movies where that's the first darn thing he does). The heat from fire is enough to interfere with the stone composition, which in turn threatens to collapse a cave. Imagine having just discovered fire, and you go running to show your family but everyone dies before you can say "hey Mario look what I made". I wouldn't expect a caveman to know about that rule, but I would expect them to feel tempted to find out the hard way.
Finally, there's the fact they're filled with disease. Most notably from the cave animals; while things like ticks and rabies are not common in cave animals, they do happen. If that wasn't enough to outright stigmatize dwelling in a cave, even the environment itself is viral. There are caves where the reservoirs are like 100% condensed bacteria. That's got to send awful mixed messages to seek refuge from an oasis and suddenly you have a fear of water like me.
How did caves become such a go-to and one where nobody is depicted as having any serious accidents in?
Never heard about collapsing a cave from heat, but soot from fires would damage a cave's aesthetics, which is a cardinal sin in spelunking.
Yes, so basically picture a cave like a giant jawbreaker. When a jawbreaker is left in the sun (and yes, Mythbusters proved this), the different layers of candy expand at different rates. This causes pressure buildup and eventually explosion.
A cave isn't all just one solid type of mass, it's not all a single boulder like many people depict due to the limited coloring of old cartoons. You'll have many different forms of sedimentary, igneous, and metamorphic rock all around the cave. And a fire is hotter than depicted on TV, that's the whole point of a campfire (a part of why the original question in the OP intrigued me). So the simple act of starting a fire can cause a chain reaction which destabilizes a cave section and causes a collapse. And with caves being as intriguing as they are, you don't want to ruin humanity's chances of finding a cave section.
People upvoting a lot of the confusion-based replies shows both sides here have things they never expected they didn't know (while downvoting my own confusion-based replies, for whatever weird reason).
I wasn't doubting you about the collapse from fire heat, I just had never heard of it before so wasn't commenting on that aspect.