I think it's because flying is unusually demanding - there are so may aspects that have to be right before the wheels leave the deck -
Licence
A/C documentation
A/C condition
A/C availability
Any Handheld devices charged up
Clouds - base - type - forecast etc.
Wind - and forecast
NOTAMs
etc
etc
Personal availability - must co-incide with all of the above.
The last point maybe is the cruncher - how many days ahead do people ordinarily plan? OK if you're IMC and night rated etc, but if you're daytime VFR like me, then lots of mental energy goes in to trying to schedule a flight for when a) I can do it, and b) when the weather is likely to be OK. After lots of missed attempts, I am sure it can seem like hard work.
And then when the wheels leave the ground, it is a high-demand pastime.....so I think a lot of people find they are not getting the enjoyment return for the effort expended.
And unfortunately, that's the nature of the beast - some people thrive on it, most won't!
A