TOGA is always TOGA; idle is always idle.
At the first sign of anomaly that is taking away your flight control, disconnect the autopilot, auto thrust, and turn off the flight directors. Set 85-90% N1 and recover level flight, crosschecked against standby instruments.
I've been flying for a living for more than 40 years, the last 10 in A-320's. The above is my basic boldface for just about everything that can come up on the Bus. That is "flying the airplane".
When the dust settles you can get out the QRH, talk to each other, and figure it all out.
For God's sake, though, just look around if you find yourself climbing or descending abruptly for no good reason. Pickle everything off and level the flight with a reasonable power setting.