My personal survey of pilot friends shows that 80% can't stand heights in buildings, and most won't go up a ladder to their roof.
My experience is that above about 2,000' with the doors off starts to get a bit spooky, and 20,000 AGL in a Gazelle (with doors on, parachute and O2 even) was also getting a bit dicey.
Not sure what it is, but it is prevalent.