I have exactly the same problem. Above 3000 feet, I start to feel anxious about the height that I'm at. Steep turns and spiral dives have never been a problem. However, i have been reduced to a gibbering ball of nerves on a couple of occasions, both of which were away from flying. The first was when climbing the CN tower in Canada, and trying to convince myself to cross the glass floor. The second was climbing Sydney Harbour Bridge.
It's a perfectly natural reaction to have.