I learnt to fly in the RAF, when I left I did nothing for 15 years. My wife bought be a microlight lesson and I did the PPL(M) in 25hrs. I flew about 300hrs in ML and became more and more frustrated at the lack of capability. I got bored of the UK and wanted to do what being able to fly gives you the greatest gift - to be able to get off this Island under your own steam.
So I went and did a PPL(A) and flew the club spam cans and as IO correctly points out I quickly realised that sticking around a club environment would strangle the life out of my flying so joined a group and discovered even more versatility. Liked it so much that I did the IMC and night, then an IR and an Instructor rating, accompanied by a few thousand hours of touring around Europe and the list goes on Now I fly a multi engine turboprop but still fly privately for fun as well.
Although cost is a driver for most people, I think the biggest reason people give up is a lack of confidence which leads to a lack of understanding of the real utility of aviation.