bryris,
It's all about perspective. You'll have the best of both worlds. The efforts you put in will have a direct benefit to yourself and your family. Also, you'll be able to chose when you fly and in what weather. You'll also be there for your family which will give you the greatest enjoyment and fulfilment.
There's nothing stopping you from combining your accountancy talents with aviation. Plenty of pilots need good financial advice and you'll know first hand exactly what they need.
Good onya.