In some ways, testing UAVs is much more difficult than testing manned aircraft. As far as the raw airframe goes, either is straightforward. Design analysis, structural load measurement, flight control system limitations, etc. are very similiar activities whether or not there's a filled seat on board. Perhaps easier without the filled seat. Once you start talking systems, a huge increase in complexity results. Manned systems have a very reliable back up system on board - the person. UAVs have to have their own backups, and have to be proven to be safe enough for the intended work.
When the work is to fly over a battlefield and gather intel, that's straightforward. The risks are easy to calculate and mitigate, airspace is controlled by the agencies fighting, and one backup is to let it crash (no immediate loss of life, risk acceptance for what is on the ground below).
Shift that into civilian airspace or over populated areas and the reliability cannot be mitigated by accepting erratic flight profiles, destructive aborts, etc.
Now consider the concerns both in the battlefield and in civilian airspace when the UAVs are armed. The risk in using short range ground attack weapons is much more easily mitigated than the risk in using medium to long range air to air weapons.
One litmus test in the perceived level of safety of UAVs is whether you would put your children in the back of one. I won't. An interesting extension of this litmus test is would you rather have your children in a UAV flying in airspace protected by manned aircraft, or in a manned aircraft flying in airspace protected by UAVs. If neither is not an option, then I'd prefer that the fighters are manned.
Cheers,
Matthew.