Of course absolute perfection will probably never be achieved but pilotless aircrafts will happen, and I'd wager to say sooner than most expect.
It won't happen until the world's insurance markets agree to let it happen, when they feel it is an acceptable risk, especially the third party element, (pilotless aircraft involved in a mid-air collision over the CBD of a major city). The leaders in the aviation insurance market notch up a lot of air miles and will have definite opinions on this subject.
Before the skies become full of pilotless aircraft, dependent on ground control all threat from sophisticated terrorism will have to have been eliminated.
Don't hold your breath.