It's not exclusive to the rich/advantaged. IMHO the biggest opposition to unrestricted immigration in richer countries comes from workers, because "they're taking our jobs". Just look at the H1B mess in the US for an example (where a lot of company managers/owners want to bring in more foreigners, not fewer).
EDIT: Maybe I misinterpreted, your post reads like "the rich are preventing free movement". I want to point out that anyone who has a well-paid job is "advantaged", not just rich people.
...within a gated community that encloses those who pay the access fee.
While free roaming would be a net win for society, the currently advantaged part of society does not feel like sharing much.