Of course the other guy would be loaded up more than when the Auto Pilot is engaged, but that's good to practice too!
So you pick appropriate times to hand fly eg. fine weather.
You don't hand fly holding patterns in IMC in the middle of a storm etc.
At least then, when the Auto Pilot or Auto-throttle does drop out one day, you won't even blink an eye when reverting to manual control.