I can only speak from a gliding point of view...I spent many happy years...20+, straight and level (and a few turns to!) with no problems...but as soon as somebody wanted to turn things upside down...I was a bag of nerves!
My only dream I seem to have (I stopped gliding 7 years ago)...and fairly often, is that I have turned up to fly...but don't have a medical!
...probably because I feared those nearly as much as aeros!