I spent 20+ years climbing ladders and telephone poles with no issues, and can't remember having problems on any tall buildings. But when I started learning to fly in a flexwing microlight I certainly wasn't happy. In fact I quit after a few lessons. However I had another go and persevered. My instructor was quite correct that I would feel happier when I was doing the flying, rather than just being a passenger.
Now 20 years and 1200 hours later I have packed it in (money & health, being the main reasons). Over this period there were a number of occasions when I was genuinely scared - principally during turbulence, and in this situation I would always descend to low level, where I felt somewhat safer. Why, I don't know, I just did - that & "fear of heights", maybe? On the other hand I've been up to 10,000 ft in calm conditions and been OK (if bitterly cold). But the slightest "bump" would set my heart rate rising! I was perfectly happy chucking it around at up to 2000 ft or so, but no way would you find me practising "unusual attitudes" at higher levels... I've had a few short flights in R22's and can understand the "twitchyness" comments. I wonder if a lesson in something a bit heavier might help you?