Gotta agree with Shunter. I drive a motorbike, and if I only took up pastimes that I had been convinced were safe, I'd never get on the damn thing.
but yes, a combination of training, care, not trying to show off, and a balanced approach to quality of life all go into it.
If you want to know the truth, flying isn't safe, in as far as once you are more than about 10 foot off the ground, if you drop out the sky, you could be in real trouble. Of course, leaning out a window isn't safe, nor is crossing the road or eating in a restaurant. Just got to balance any risk in life with reward.
I know a chap that skis 3 or 4 times a year at least. The idea of bombing down a hill at breakneck speeds with a small plank on each foot doesn't sound safe to me, but that's because it doesn't appeal. I go sailing, and he thought it was the daftest thing ever, because if a storm hits, you can't just get off in the middle of the ocean. Fair comparison, I think! (oh yes, and we both love flying, and have motorbikes!)