A GA aircraft doesn't have to be full of terrorists to be a "threat".
Surely a more plausible threat from a GA aircraft is to accidentally bust airspace and then fly blindly near/through a scrum of media helicopters, police helicopters, UAVs, etc. I remember Dale Farm got NOTAMed for that reason and, sadly, I think the Olympics will be an even bigger nuisance than Dale Farm was.
Shutting down vast areas of airspace is overkill, but it isn't totally stupid to consider the threat.