I quite regularly take passengers who ask how high we'll be flying - and i'm sure most of them would be surprised to know that i'm not joking when i reply "don't worry - we'll not be going very high - i'm afraid of heights too!"
Like Nigel, I feel most comfortable around 500' - high enough to give you a fighting chance but low enough to still get some detail. Strangely, flying cross country at night i'll happily fly at 4000agl as there is no detail on the ground to see - just horizon/instruments. purely psychological i think - my last job flying powerlines probably didnt help - i thought anything over 200 was high...
plus, like jemax - if something goes catastrophic - i really don't want to get too long to think about it!