I share your frustration Ollie, but some skills do atrophy unless they are regularly practised, ask a musician.
However, like you, I personally don't have a problem going manual thrust, and on all the types I have flown, from Shorts 360 to Airbus A330, it was not difficult. Possibly because the first four types I flew commercially did not have auto-thrust; I learned to constantly scan properly and adjust thrust manually, so I have that "motor programme" fixed in my brain. But I am certainly not Chuck Yeager - just an ordinary Joe.
However, manual flying does use up more brain power and scanning time, and is therefore not always appropriate when in busy airspace, or when tired. With auto-thrust in, when I see the trend arrow and speed dropping, I flick between the speed tape and the Engine N1 or EPR gauges, expecting to see a small increase in thrust. If that does not happen I am ready to intervene, but 95% of the time the Airbus gets it right.
The 5% of actual interventions have mostly been certain A330s with older software getting slow on short finals on thermally days; where literally two seconds out of the CLB gate and back sorts it out.