Yes Adam, I have had similar experiences. I think the key question is, are they afraid/reluctant to go flying with you or are they afraid/reluctant period?
The latter group as IO540 suggests, is perhaps not for you to deal with. I just let such folk know that, if they ever decide they do in fact want to fly, then I'd be happy to take them.
If it's you that's the concern, and it's their first time out, they probably can't accept that you're not a professional, as you suggest. This group will probably come round, given time. I have taken such slightly nervous people up in a hired club 4 seater along with an instructor whilst having a check-ride and it helped.
If it's that they won't come back for a second go, then DC3's comments are about right I guess. I haven't met anyone in this group yet, but then again, I've not been flying long and there's still time :-)