It's no different in the good ole US of A... a good friend local builder with a great business has a son who has no interest in taking over the firm. He wants to be a rock star / programming genius etc etc... We are reliant on immigrant labour for many skilled jobs - carpentry, plumbing etc as no one wants to do them.
The challenge with eu freedom is that those countries who want it are those that want people to freely move into their country and then quickly out to the chosen destination which is the UK. There seems to be little fair share around.
We of course have the issue, like France, of a commonwealth who can settle here too... so the pressure comes from everywhere.
I do have much sympathy as an expat worker here, the hoops to jump through are many and rightfully so. I should not be able to make my home anywhere without being able to pay my way and be a positive contributor to the community.
The issue of human rights is a tough one, but ultimatly we must manage the issue as source rather than provide a release valve. I might be attacked for this, but if the determination to leave was applied to the determination to make changes the world might be a better place.