What's the difference between caning a car for a couple of hours on a track day and flying a light aircraft?
There's a simple and vitally important difference: Culturally, nearly everyone [in the western world] grew up with motorcars/cycles as a part of everyday life. It's a natural extension to get involved with them as sport. To support this, nearly everyone already has the license required (at least at the entry level for racing), and the urge to drive whatever it is, beyond the rules of the road for pleasure and excitement.
Conversely, aviation is a big leap for many people, it's not already running in their veins. Then the devotion to earn even a basic PPL, and expense to operate a vehicle which seems of prehistoric design philosophy compared to a modern car.
We can watch dozens of movies and TV shows, where driving is glamorized, and people can get their heads into that. Glamorizing piloting in movies has faded, and getting one's head into the cockpit of the Concorde or an F35 seems virtually unobtainable for the average person.
Experienced pilots in our generation have to promote GA, to prevent its fading into a rare pastime a generation or so from now. The general public will not support us - they have their cars!