IMHO ground operations are the most dangerous bit of flying.
I've never had as much as a scare when airborne, but had one or two major mishaps (none my fault) on the ground, plus a load of sticky situations (usually to do with negligent airfield surface maintenance at some of the places I've been to) which luckily did not result in damage.
As for flying itself, one should always have an escape route. Therefore, I avoid SE night flight for long periods because there is no escape route. Other stuff e.g. over water flight is handled by carrying a raft etc. Mountains are OK - fly high above and have a GPS displaying the topo chart so one can glide into a valley if there is an overcast.