Nostalgia is really interesting in that it's inherently bittersweet. It's nice because it grounds us in a shared timeline and focuses on mostly positive aspects of some past point in time, but it's also sad because it means thinking back fondly on a time that will never be again.
So maybe it's the bitter half of that bittersweet feeling that you're subconsciously averse to? Either that or maybe your past/childhood was mostly negative or even traumatic? I'm no psychologist, so really I don't know what I'm talking about.