I like many others i'm sure, always feel safer when i am in charge.
I have to agree, I will happily sit in the left hand seat of a PA28 and bump around merrily at 2500ft on a hot summer's afternoon. Even in the right hand seat with someone else's hands wrapped around the controls, lumps and bumps will go very much unnoticed.
At 35,000ft things change somewhat (for me as a passenger at least), I have no problem with the turbulence, however with the monotony of a 4 hour flight with little to do, comes the propensity to notice the smallest of differences. I suppose sitting without an instrument panel and constant ATC chatter leaves one to speculate further over insignificant changes and apparent oddities.
I can vividly recall a flight back from the Dominican Republic to London (this was when I was around 11, so the memories maybe slightly enhanced...), it was a Virgin A340. Nothing was too eventful at first, however a couple of hours into the flight we experienced the clear air turbulence which is likely to only distract a more nervous flyer. A couple of minutes later, the seatbelt sign went on - I imagine even a frequent flyer may start to wonder what maybe in store after the ominous chime.
Anyway, over the next half an hour, the intensity of this turbulence increased to the extent the cabin crew stopped service, and were asked to be seated. By this time, it was no more than moderate, however instead of dying out, this turbulence went on for at least 4 hours. I wouldn't describe this as scary (no unexpected hundred foot drops or the like) but the continuity certainly managed to turn my stomach upside down.
I think everyone has their own experiences (I'm fairly young, so I haven't many yet), but I think it's interesting how situations differ due to the psychological situation the person is in.