If we take this line of reasoning, then even regular population growth is bad and doomed to wreck the economy. It's the same argument made against women entering the workforce: that they're taking jobs away from men. In fact, a bigger workforce means that more people spend their earnings, which creates more demand and more jobs.
That said, it's probably fair to make the case that a certain class of very low-skilled individuals are going to lose their jobs because of low-skilled immigration. Very few of these people are going to learn to code and I think that we can all agree on the need to find them some help.
Edo Japan was an agrarian society that would have eventually been subjugated by more technologically advanced countries. These were technologically more advanced because they were more open.