In the UK you are restricted to VLOS (in effect about 500m) and 400ft altitude. However, you can apply to the CAA for dispensation to operate outside these restrictions provided you have good reason to do so and an Ops Manual that shows how you're going to ensure safety.
I've worked with everything from Black Hornet to Reaper; although this sounds a bit like the NRA re handguns, the problem is not the UAV per se, but the person(s) that operate it. The guys that are professional are not the ones you need to worry about. It's the naive or genuinely nefarious (e.g. paparazzi) who simply don't understand or deliberately ignore the regulations that are the greatest hazard.
Until a rogue operator actually kills someone very little will be done. The problem is very similar to mid air collision; there are those that think that TAS, TCAS and ADS-B will solve it - conveniently forgetting that a large number of air users in open FIR don't have transponders. The answer, for both manned and unmanned aircraft, is passive/active collision warning systems.....