Flying has never been an issue. But then I haven't done much of anything that could be considered 'interesting'. My worst experience was trying to maintain formation on an aircraft being flown by the unit CO - it could all end if I get it wrong.
For me, if I am taking a risk and it's only exposing myself to that risk, I'm ok, but I'm totally different if I'm exposing anyone else to the same risk. Responsibility I suppose.
As I've got older I've developed some 'knowledge induced' vertigo. I also have become fearful of overbalancing and falling down escalators when I get on them.