There are fringe, exceptional cases where this is a legitimate fear, but in the general broad-brush case, it's largely a self-fulfilling fear, brought about by the way people are treated when they come to a society that believes this. If they are given no opportunities when they arrive, and are segregated off into culturally de-facto ghettos, resentment brews over generations. In the cases where more open immigration is abused, such as the 9/11 terrorists who came to the United States, it's throwing out the baby with the bathwater to question immigration as a whole, rather than focusing hard on how to mitigate the exceptional cases of abuse.