It's really country dependent. It's different below some weight (eg 5 kg, 25 kg, 150 kg etc). Height matters (eg 500 ft). So does line of sight or distance (eg 800 m). There are also privacy issues. Some of the regulation changes or additions categorize these things into tens of different classes based on the criteria above. Based on that I've seen official suggestion for forbidden zones like a few miles parallel or tens of miles in distance from airfield runways. From some point you do need an VRF or IFR license, CAA approval etc as proposed.
For example here in Estonia the flight rules say that you can fly unmanned flight systems below 500 feet, after that it's under ATC rules. But like almost anywhere in Europe it is mostly unregulated and vague, if it's outside Regulation (EC) No 216/2008.
That said, I agree it is easy to fly these things dangerously either on purpose or accidentally. A friend just had a quadcopter fly away yesterday to disappear. Most quadcopters are pretty dangerous when the propellers hit a person (cuts fles like a knife), or when it falls down on you (being a few kg in weight). Better not to thing if one wonders into an airfield traffic zone or gets sucked into a jet engine.
The official EC page seems pretty outdated:
European unmanned aircraft systems (UAS) - Transport
and here's another one:
Remotely Piloted Aircraft Systems (RPAS) - Aeronautic industries - Enterprise and Industry
Here's another take on it listing some EU countries:
Drones: Privacy implications across the EU