To put this in a very blunt manner, would you rather risk dieing doing something you love or risk living the rest of your life wishing you'd had a go?
Which is not to say the risk is that high, if you fly within your capabilities and those of your machine, make sure the machine is properly checked and maintained, etc etc.
My partner and parents (and family) were initially a bit upset when I started flying. Ditto when I started riding motorbikes. Ditto when I learnt to skydive while at school (my partner missed that one). Ultimately, though, they saw how happy all those things made me, how seriously I take safety, and now support me 110% - we've just got back from a holiday in South Africa with my parents, where I took them all flying.
Hang in there - only you can decide it's not for you, and it does keep getting better!