It's all in the mind. Flying both F/W and Heli's I find it's directly related to perceived groundspeed. Heli's up to 2000' no problem. Hovering at 2400', big problem!
Now with F/W altitude is much less of a problem as we are wizzing along so fast. Right up to FL390 no big deal. But, start going into the 40's and it gets worse. The perception of transit across the ground gets less and less. FL470 over equatorial regions looking down the side of a big CB in turbulance is no fun for anyone! I also found altitude in a high wing cessna worse than a low wing piper. Something to do with the sensation of being suspended rather than sitting on something.
Try walking on a 6" plank balanced on two bricks. Now try it with a 2000' drop underneath you. The same 6" yet only a fool would try it.