Humans have a certain reservation against newcomers yes. But the way this is excarberated by politicians and right wing media outlets is very much crafted.

It is first crafted on the level, that immigration is not aided with integration, e.g. giving people fast access to education and working opportunities. Then the resulting poor social status is correlated with higher crime rates, as it is with the native population too. Then few high profile crimes are plastered all over the media and it is claimed that this would be intrinsic to the newcomers and it is them being newcomers and not them being held at arms length and deliberately kept in poverty that is at play.

It is classic divide and conquer, to pit the lower social classes against each other instead of the leading elites. This also shows in your example from the US. The problem wasnt Irish catholics. The problem was Irish catholics integrating and taking positions in government offices and exercising their right to vote. It threatened the established elites power and they reacted by targeting them with xenophobia.