If you're that concerned about leaving your wife and kids behind, just ensure your insurance is firstly adequate in terms of financial support and secondly that it actually covers you, should the worst come to the worst.
I made sure my life insurance covered me for flying, if anything happens, as long as I'm not negligent, the wife and kids will be adequately provided for, then it's down to me to ensure I minimise the risks as best as possible. The question is: How far are you going to let minimising risk interfere with your fun? If you can't, you might as well stop right now.
As an example, I flew to the UK earlier this year from Hamm, crossing over via Dunkirk - Dover. On the way back, I decided to fly direct across the North Sea from Great Yarmouth to Middenzee in Holland. Now, I could have reduced my risk by flying down to Dover, across to Dunkirk and back, at an altitude where, in the event of an engine failure, I would have been able to glide clear of the water. So why did I choose to fly direct across the North Sea? Because I wanted to. If Lindbergh can cross the Atlantic in a single engine back in 1927, I'm sure I'm reasonably safe to fly across the North Sea in a modern C172.
However a healthy portion of fear means you probably won't make the mistake of flying into poor visibility when prudence dictates you should have stayed on the ground, so enjoy your hobby....