Andy, I know what you're going through. I'm 33, married, 3 young kids, own business, earning over £100,000 p.a. Got it all, you might think. But I've always wanted to fly, and life is too short to stick at doing something you don't love.
For me, it's not about the money. It doesn't buy you happiness. But it does enable the family to have a nice life. My wife thinks that if doing what I'd love to do would make me happier, then I should do it, because the family will benefit more from a fulfilled husband/dad than they will an unhappy provider. Great in theory, and wonderfully supportive.
I agree with other posters who say it's not worth upsetting the family situation. That has to be the most important factor in your decision. If they are 100% behind you, then go for it. 90%, then don't. Remember the very real possibility that you may have to move across the world to secure a job, and this will have a big impact on the family.
For me, I think I've almost decided (is that suitably non-committal?) that I'll keep my job, get PPL, maybe some further ratings, and buy a part share in a plane. Keep it a hobby. That said, why am I going to the Flyer show?