How about the part-time route? That way you can hang onto your day job and pay the mortgage? A friend of mine works in IT during the week and flies joyrides at weekends in Tiger Moths and a Pitts S-2C, then trades his flying pay to buy more hours. He's currently getting his Instructor's rating and ATPL. He probably flies 5 times the hours I do in the Pitts, and he doesn't lose any pay for doing it. Mind you he doesn't have any family so it's more achievable. When I was his age I was either working weekends in a commission only sales job, or navigating a racing yacht (yacht-racing is the most expensive and the cheapest sport in the world - cheapest if you're crew; most expensive if your the owner).
Had I worked it out I should have done what he did... after the RN declined to fund my hobby any further I decided to go and earn the money to do it myself. I may be able to afford to pursue my passion financially but finding the time is much harder now...