That's true, except people from Baltic countries, all of the Eastern Block, and notably Finns love the narrative of bad bad barbaric Russia that always oppressed them, and bad bad totalitarian USSR that was "worse than Nazis".

Just recently certain Linus Torvalds expressed a interesting sentiment about being a Finn and knowing something about "Russian aggression", well, Soviet-Finnish conflicts didn't start with the Winter war, and the Winter war was preceded by a few suggestions ending in an ultimatum. By those suggestions Finland would receive far greater amounts of territory (in the areas it claimed before at that) than the stripe of land and a few small islands in artillery range of Leningrad it would be giving away. That's rather soft if you consider the character of the preceding Soviet-Finnish war. And Finland's participation in the blockade of Leningrad while allied with, well, Nazis makes the "worse than Nazis" argument more easily understandable and still wrong.

Percentage-wise Latvia had more % of population in SS than any other country including Germany so i think the "Latvian" part did played at least some role.