There's something else - an X-factor of some sort involved. It's not skills alone
You could start with picking a suitably nice day to fly
Yeah, the X-factor is called "luck"
But seriously, Adam, point taken - life, and the general progress of human civilization is all about taking risks.
I guess my point was, that I don't equal light aircraft with family transportation, both for feasibility and numerous practical reasons as Peter explains so well, and in terms of risk.
Actually I have to conclude, that most of the flying enjoyment is really for our own personal selfish reasons as pilots of the aircraft. Passengers tend to have a separate experience.