yellowperil, it's not the limitations that cost lives (well, they do, too, sometimes), but the discovering of them. The more you rush into this, the more limitations you're going to discover rather than anticipate--in a situation where you can't just stop the vehicle and wait for help.
One known limitation: we already know that humans need some time to get their bearings when entering an unfamiliar situation, so when a plane calls a remote pilot for help, that better be some minutes before pilot input is needed: but if the plane is going to try and anticipate that, there are going to be a lot of false alarms, which are going to lead humans into treating alarms as not so serious.
The unexpected challenge in avoiding accidents with the Google car has been to prevent other cars from hitting it: it needs to conform to human expectations when moving in traffic, or a fender-bender can result. If you do not research beforehand how automated planes can intermingle with traditional aircraft in congested airspace, you're going to have some avoidable accidents.
Before we know of the limitations, can we really make an informed decision on whether it's worth it? You can dodge the question by saying "it's going to be inevitable", then you don't need to have a discussion on the merits; but you could be making a bad choice. (Well, it won't be bad for the tech companies driving the change--their bottom line is going to be assured.)