I'm going to be cantakerous and try to inject some positivity into a seriously depressing thread.
The pilots I've met and mix with have been flying on average a lot longer than three years, the time loosely cited as a typical drop-off point. The pilots I mix with continue to find it enjoyable, or they would have stopped long ago and spent the money on yachts or repaying the national debt. They may not all be as anoraky as me and mine, but they revel in touring/bimbling/gentle aeros/introducing the great British public to the sky/strip hopping/etc as much as we revel in our flying, which includes a least a few of those items.
I don't think most people drop out because of flaws in the flying world, however flawed it may be, but because the flying world no longer fits their particular world. Domestic arrangements, health issues, money all are significant causes but most know there are ways to continue, even if their flying habits have to change. As someone above said, they move on. We do move on from hobbies, quite often because we want to try other ones. I once enjoyed being part of a chess club, but that was then. I used to enjoy making my own clothes, but it's a time-consuming occupation and flying consumes all the spare. I've chosen flying because it fulfills and stretches, gives experiences I'd never have acquired and having got sucked in, I'm not struggling to escape. I don't think I will until, one day, I get too poor, sick or shot down by regulations and it'll need at least two of those to make it a serious consideration.
I'm one of the 'lucky' ones. I have an old tailwheel aeroplane, fly from a wonderful farmstrip style airfield and enjoy the help and support of wiser men. I suspect too many people these days have to train at places that inadvertently give the impression that such airfields/aorports are the norm and they never or rarely get to experience the full joys of the puddle-jumping world....
One size doesn't fit all, I know.
"I think the biggest single reason is that the whole scene is decrepit, full of anoraks, has few interesting women which ensures that most "modern" men (who still have time and money) will go elsewhere".
Well I've found plenty of interesting men in the scene.
What's a "modern" man?