Flying drones is completely safe if you use basic common sense and abide by the rules. Don't fly over 400 ft, out of sight, over crowds and respect peoples property and privacy.
The rules are clear about flying near airports, restricted areas or near any aircraft. Most have GPS so won't fly into a restricted airspace or even take off if you are in one. Suggesting banning them with no reason because the word drone sounds scary is silly. Besides being a lot of fun to fly and do aerial photography they are being used a lot more to benefit all of us.