Perhaps another way to look at this question would be to ask "Are there other pursuits in life that can give the same sense of reward/satisfaction that flying has?
What I mean by this is that it is right and proper to ask others if they thought that their path was "worth it," but since no one's path is exactly like anothers then it might be best to ask why is aviation exerting a draw on me and what do I expect (or would like) to gain from it? I know that is a difficult question to answer, but if you can then you can compare it to other pursuits that you have in mind.
Sounds like a rather verbose thought, but if one is tipping toward middle age and is thinking of taking a step toward flying, one must ask what exactly is one trying to find? What is the motivation. If you ask others if they think it was "worth it" then be sure to also ask what motivated (or still does) them to take up the calling.
Just MHO