I think even the most experienced pilot's experience the feelings you describe at one time or other.
As a very (inexperienced) pilot, I used to sometimes see the small wheel dangling pathetically somewhere over a big expanse, and think "Oh sh*t, what am I doing up here?"
It's not that unnatural if you logically think about it. 'spose it only causes problems if it affects your ability to land the thing, of fly into something.
Funnily enough if I was having a bimble, and feeling a little agrophobic, I'd take the low level route, because it was nearer the ground. Which, logically, is daft.