Taking Nimmers post a little bit further - class D is to protect IFR traffic arriving at / departing from an airport but permits VFR access when cleared by the controlling authority (i.e. the bloke (or lady) at the other end of the headset).
Putting Gapsters sentiment in a slightly more constructive way, IFR jockeys have to jump through an awful lot of hoops before they can sit there and push buttons. They are professional people, trained to an international standard, constantly being checked, monitored, reported on etc.
I'd not discount the amount of training and checks Mr Smith in is PA28, indeed the CAA make it hard enough I think. However I've been on the receiving end of multiple life stories over the RT filled with irrelevant information, and blocked frequencies, and incorrect routings, which builds up a doubt in my mind - does this person really know what they are doing - after all not everyone with a licence to drive on the road can handle a car like Stirling Moss.
I've never refused VFR access to class D based on "this guy sounds a bit dim so I'm going to tell him to remain outside and pick up my Daily Mail again" but I have experienced some - how should I say? - unusual airmanship (similar to Gapsters story) one of which resulted in reporting action, so Timothy please understand that us controllers aren't out to screw over the GA by keeping them out of "our" airspace, but we may be hesitant on occasion based on what comes across on the RT.