Things can get confusing very quickly.
Would it make sense for Airbus pilots to have thought in advance that if things start going the way they did here(or something similar), then don't just disconnect the autopilot, but also take the thrust levers out of any detents and manually fly. Otherwise, you might get these situations of thinking you are at CLB thrust when it is really at idle.
It sounds like the PF on this flight got the "get rid of the automation" in terms of autopilot part of the automatics correct but not the autothrust part of the automatics.