Not liking immigration isn't racist, not liking immigration because of a persons skin colour or race would, by the majority of people, be considered racism.
I'm in agreement, mass immigration is bad for any community as it can be overwhelming and cause side effects, like people feeling like they're losing their own country. The whole point of immigration in my eyes is to assimilate said people into our community, and as such, it should be done over time, not in mass flocks.
Put it this way, if you have a large group of friends you've always known and you put them in a room with a group of strangers, and are then told to approach a group, who do you think they will approach? Their friends obviously. Putting the not quite so perfect analogy aside, my point is that, if you put a massive group if immigrants into the country, they're not going to try and join the community, they'll stick with people that were/are a part of their community already. If people don't mix and bond it then gives an us vs them persona, especially when there are people in countries that ARE racist towards people. With the media running around screaming "ALL WHITE PEOPLE ARE RACIST" it can easily put negative thoughts into people who entered this country to escape whatever bad state theirs is in or even just because they fancied a change.
In my opinion, if you put a small group into the community, over time they'll start getting used to the community, talking with them etc etc. and before you know it, they've just become a part of daily life in that country.
(In these examples I'm talking about your RUN OF THE MILL person, I'm putting aside the people who enter the country to promote extremism or are here under some other kind of motive.)