I'm about half-way through my PPL training. I half-jokingly say to people that I hope I'll get bored of flying in about three years - that's about the time it has taken me to get bored of every other hobby I've ever taken up - because my long-term financial outlook is much rosier that way!
I know a couple of guys who have given up flying. For both of them, it's a combination of having achieved their dream and "got it out of their system"; and the arrival of wife and kids meant that it was difficult to find the time to stay current and even more difficult to justify the cost, which had become a much heftier proportion of their (greatly diminished) disposable income.
If you're not on the salary level of top lawyers or investment bankers, it's quite a big financial sacrifice once you're no longer free and single.