Actually I have to conclude, that most of the flying enjoyment is really for our own personal selfish reasons as pilots of the aircraft. Passengers tend to have a separate experience.
For my experience, the passenger's desired experience is usually nothing more than getting there. This, from flying with each of my kids over the years (three of whom are in their 20's now). They have zero interest in flying the plane, they just wanted to get where I could take them. My youngest can be coaxed into a bimble, but after 30 minutes is usually declaring boredom, if there is no clear objective. I have done many 1000+ mile trips with my wofe, who really seems to enjoy the charms of a
fair weather trip.
I have considered the "x-factor" for decades, trying to hone and optimize it. I have reached the following conclusion:
The "x-factor" is the opitimum point in between very well informed decision making (personal skills and equipment capability considered) and complacency.
I have seen new pilots, who I obsevered to be complacent right from the start - they would have a hard struggle to overcome that to become well informed decision makers.
I have seen new pilots who seemed to be great decision makers, but had yet to develop the broad experience upon which to base those decisions - they have time to improve, and have th potential to be great.
I have seen (and fear becoming) complacent pilots, who were now flying only on luck, 'cause they were not applying skill very much. I've been to a number of their funerals. Others I have sat down and chatted with....
At the height of fearing personal complacency, I trained for my helicopter license - that straightened me out!
Flying which is high challenge, but low risk is excellent for skills development and warding off complacency. I do not notice new pilots attempting to refine their skills after being licensed, they are sometimes riding the high of being licensed, and imagining the vista of opportunity in front of them, without realizing what those who are doing the bigger vista, have put into that flying.