Thank god - I thought I was the only one!
Having done 98% of my flying below 500' and quite often at treetop level doing utility work - I could never get comfortable flying above about 1500'
For me it seems to be about seeing the detail on the ground - although I know i'm safer higher up, I feel more in control lower down. Because the horizon is all around rather than below, its easier to judge attitude and therefore everything feels more stable. Once I lose ground detail and i'm looking down at everything I become more prone to that tumbling sensation. Interestingly, If i'm flying at night in mountains i'll go way up, and it doesn't bother me because i'm using the instruments more.