Outside of the wannabe pilot circles, I suspect the average young person that could be a potential GA hobbyist just simply isn't aware of its existence- most airfield with a flying club are in the middle of nowhere after all, and GA rarely makes an appearance in the mainstream media unless there's a fatal accident of course.
You're right there. Young people with money live near the center of town. If your airports near town are so overpriced such that there is no GA operation based there, your only option is to go to the middle of nowhere. And that's where the older guys with money live.