A lot of things are perception! Fox hunting was banned as it was seen as a sport of the "landed gentry" The upper classes! Fishing was not that was seen as a pastime of the masses.
You only have to see the stream of Caravans or motorhomes some costing in excess of a small aircraft to realise how strange these perceptions can be.
The media like to portray aviation as the sport of the mega wealthy.
Local MP flying to work in a clapped out 30 yr old mooney becomes Local MP using private jet. etc.
Part of the problem is political especially in Europe. On the roads you can have police, you can have all manner of speed devices to regulate and control.
In the sky its different. While its regulated there is only a certain amount of control and government dont like that.
Once in the air you have freedom of movement and god knows what you may see from your lofty viewing point
Pace