Speaking as a pax, I dont feel comfortable flying high (AGL) in a chopper either, unless I'm occupied with track/balance equipment or t/shooting some sort of fault thus keeping me distracted from dwelling on it.. .If the helo has little or no airspeed at altitude then I start to sweat, no matter what I'm doing.
The reasons for the scary sensation must be to do with some sort of yet un-named phobia maybe?