if you don't press the red button first, you will screw it up further
I agree... oh, wait a minute! Why would anyone want to take over without pressing the take over pb?
Taking over the control is always a risky procedure. It is even riskier when done close to the ground. And risk gets fairly unacceptable when it's not done properly. Proper procedure for FBW Airbi is to press 'zee red button' and then do whatever you want to do. Not pressing it is not an option. This procedure is not intuitive for pilots coming from classic types and it has to be well taught and ocasionally rehearsed (in sim, of course).
Your A320 is not 737, it's not MD80 and it's definitively not Tu-154. Don't fly it as if it were on of these, fly it in the way your FCOM tells you to.