I used to share similar concerns, but discussion on here of an incident (Continental MD82 I think) illuminated the fact that modern airliners have warning bells for when the driver tries to take off without flaps. I learnt even more after my first Fokker 100 flight (they don't usually put the flaps out for take off!).
Now I just cr*p myself when it gets slightly bumpy - though interestingly in the days when cockpit visits were still allowed I sat in the jumpseat for a rollercoaster ride into Schiphol and wasn't bothered in the slightest. I think it must be a control thing (or at least being able to see the guys in control). I liken sitting in the cabin to being given a lift in a car, but having to sit on the back seat with a blanket over my head - unnerving.