Interesting question
CAVOK, well maintained aircraft, fuel to tabs, brimble to an airfield half an hour away that you can visually see after take off, well that's incredibly safe.
Awful visibility\mist\haze\fog, low level thunder clouds, not flown for a few months and not instrument rated = almost certain death
As a pilot you analyse the situation, look at where between those poles you are and then make your decision to fly based on that. As others have said, aircraft have been flying so long now that design issues have been fixed so very rarely does the aircraft fail, it's almost always the pilot. The reason you die is because you where closer to the "thunder storm" pole than the 1st.
To put it into some context. I fly, in fact I love flying and fly over water and abroad. On the other hand, I wouldn't ride a motorcycle (apart from my 50cc) on the road.