Terrorists have had little UAVs for a long time.
Very specialised ones that go really fast and are in fact specifically designed to shoot down aircraft. They are called SAMs.
Yet a remarkably small number of civilian aircraft have been shot down despite many attempts.
My point is that even with the right tools, it is surprisingly tricky to do.
I'm really not that worried about chimps with quad copters, at least not compared to how worried I am about the vast number of SAMs that have disappeared from Libya/Syria etc in the last few years.
Tell me. Do you worry about meteor strikes too?