Have a go, you can always stop if you don't like it or if it becomes unaffordable.
Flying has been my hobby since my early twenties. I, like you, have never wanted to fly as a career but have enjoyed the lifelong learning, challenges and social aspects of flying. I am now the wrong side of fifty but still go for a flight nearly every weekend and still immensely enjoy the experience.
In terms of expense, if you are really passionate about flying you will find the funds. I did stop flying for a period of some 10 years from the age of about 34. This was partly due to funds (mortgage, kids etc) and partly due to a demanding career.
I am now in the fortunate position to be able to afford it. In any case, I have never regretted spending the money on something that is solely for my enjoyment and benefit. If you are really passionate and interested in flying then have a go - I am certain you won't regret it.