Why do you fly so high? I think your fear is not of being alone, but of heights. I rarely go above 500 ft/agl
It's definitely just what you're used to. For the first few years I flew/taught, i didn't tend to go that high and doing so would often be nervous (hey, it's a pretty different sight picture, especially in a R22 with a VFR panel).
Then I started to do a lot of instrument flying/instructing where you're often at 4,000-6000 ft or more and you just get used to it.
After a little while you realize that there are lots of benefits - less noise for the people on the ground, lots and lots of places to pick from for an autorotation, and arguably fewer birds/wires/towers.
There are mechanical failures where you would be better off close to the ground, but of the dozen or so failures people I know have had, they've all been failures where extra altitude would have been a benefit (i.e. where they weren't going to lose control of the aircraft if they didn't get it right down).
So, while there's always the risk that you might get one of those types of failures where time is of the essence, I think the odds are against those types of failures and that it's more typically safer to fly higher and a lot more neighborly.
When I fly with someone who likes to fly low I'll roll the throttle off on them (so, say, 500 feet) and then after the (very short) autorotation I'll have them do the same thing from 2,000 feet. Having 16 times the area to pick from for their forced landing, and 4 times as long to think about stuff usually convinces them that higher is better.
That said, if you're afraid when you go higher than 500 feet it's not something you'll quickly cure yourself of - it's not fun flying while scared! But it's something you can gradually work yourself up to.