I've generally had the opposite experience, it may depend on how you put the question: "Would you like to sit in the aeroplane and pay half?" or "I'm going for a trip round <somewhere> tomorrow, spare seat if you'd like to come"
Obvious though it is, I'd also stress the importance of a calm relaxed attitude both before and during, explaining plenty and making it seem everyday/simple, rather than talking up how clever one is. I made a gliding mate (also a pilot) my first victim - he pointed out that he never wanted to hear me say 'oh cr*p' in an aeroplane, even if it was because I'd left my wallet in the car. For some reason that stuck with me!
Another thought is that while many of us know people we wouldn't fly with, at least we're in a position to judge - for most, it's a matter of blind trust.