I've gone through various phases.
I've never been afraid of trying things, working on the basis of "lots of people do this all the time". It's worked for flying, gliding, free-fall, diving and abseiling.
However any time I want I can induce nervousness by imagining a wing falling off. Wearing a parachute in a glider helps!
The confidence thing seems to ebb and flow a bit. I'm very at home in the Luscombe now but every now and then you find something you didn't anticipate. Sudden reduction in power earlier this year caused me to divert into Compton Abbas and was eventually traced to a valve guide breaking up, causing that cylinder to stop firing. Sudden total power loss just after take-off during my SEP renewal caused me to put it back on the runway sharpish (diagnosed as some water that had escaped the drain check). Landing at Bembridge with a crosswind, kicked off the drift, perfect! However I was just a smidgen faster than I should have been and she floated, during which we started to drift sideways and ended up rocking uncomfortably from one wheel to the other.
I don't think it a bad thing at all that we get our confidence dented. It makes us think about something that is in a glider pilot's checklist, Eventualities - what shall I do if.......?
No, it doesn't affect my enjoyment.
Mike