No point worrying about heights. Height never killed anyone, it's the ground you want to be scared of.
I've done climbing and abseiling, free fall parachuting, and skiied some pretty steep slopes. I learned to fly in 1974 and enjoyed fairly inept aeros from which I would commonly emerge the wrong way up and going backwards. None of it bothered me.
However a few years ago I developed the same symptoms you describe. I think it was at its worst when I was in a glider, looking across at someone else in another one and realising how much empty space there was below it. Happily that feeling has now gone and I hope it does with you. I think it's to do with the business of being "at one" with the machine, feeling that you're wearing it rather than sitting in it.
Took someone for a flight last wekend who had been in the Air Cadets and wanted to join the RAF. Trouble was he got vertigo when he was flying. He was happy as Larry as a passenger. When I asked him to have a go he was distinctly uncomfortable while handling the controls and quickly handed them back, following which he was fine and enjoyed the flight.