I can't find the reference, but I seem to recall reading somewhere (not very authoritative, is it!) that >75% of all student pilots quit before completing the private license, and that >75% of all PPLs quit flying within five years of becoming licensed.
This article on AvWeb provides some ideas about why the student dropout rate is so high:
http://www.avweb.com/news/columns/182185-1.html
I don't know why some many pilots quit after they have qualified, but it's probably a combination of money pressures, time pressures, and boredom. The last problem is the key, since most people will somehow find the time and money to pursue almost any activity if they are sufficiently interested.
When flying is so interesting, how can people get bored? Well, for one thing, most of us never graduate beyond flying circuits and perhaps the odd 1-2 hour cross-country flight (the famous '$100 hamburger'). Unless you own your own airplane, or at least a share in one, the minimum daily rentals are a significant disincentive to more adventurous flying. Also, some instructors deliberately or unconsciously educate their students to fear cross-country flying (see the first chapter in Frank Kingston Smith's wonderful book, "Weekend Wings"), or crosswinds, or what have you. If people are afraid, it's not surprising that they quietly bow out.
There are plenty of opportunities for challenges and fun beyond the PPL (air racing, soaring, float flying, aerobatics, glider towing, overseas 'aerial safaris', etc.), but you do have to look for them.