Funny thing about free advice...
I had uncles who were doctors who told me not to do it, "it's not what it once was".
I had uncles who were Architects who told me not to do that either - same reason.
Bloody good thing I didn't know any pilots.
I don't know if a "passing interest" is enough to get you through the tough times but I do know a few pilots who weren't total aviation anoraks and have done just fine.
It's only a job at the end of the day.
...and by the time you have been in it for 10 years you would have had any burning passion bashed out of you anyway. If you can muster the cash and the time to take on a new career, why not?
Go for it I say. It's more fun than most other jobs despite what the whingers on here will have you think!