The only time I've ever been really scared is when I was on a KLM 767 out of Schipol. The skipper had just released the crew in perfectly calm air when we seemingly fell out of the sky. It was as if we'd driven over a huge hole in the air, that's the only way I can describe it. I remember thinking 'oh crap, this is the big one' as the floor fell away from my feet. No sooner had it begun than it was all over and we were climbing normally. The skipper commanded the crew to buckle in again but the rest of the flight was smooth as silk. That was in the pre-9/11 days and upon visiting the flight deck the crew told me it was pretty severe CAT event, totally unforseen.
Since then, my seat belt remains fastened, always...