Excellent question about something which also confused me.
I found the answer here on pprune a few years ago after I posted the topic but I can not find it anymore. Maybe you can find it.
Basically I was told not to look at the engagement conditions of the Go Around Mode (which we both did) but to look at at the disengagement conditions for the SRS mode, which are :
SRS (SPEED REFERENCE SYSTEM) :
– Disengagement conditions
The SRS mode disengages :
• Automatically, at the acceleration altitude (ACC ALT), or if ALT* or ALT CST* mode engages
(above 400 feet RA).
• If the crew engages another vertical mode.
• If the crew selects a speed while in SRS mode : SRS reverts to OP CLB mode, and a triple-click
aural warning is heard.
Note : In Engine Out conditions, the SRS mode does not automatically disengage at EO ACC ALT.
Refer to Engine Out procedures.
Therefore if you apply TOGA whilst in SRS, the SRS mode will NOT disengage, therefore the Go Around mode will not ENGAGE !!
I think this is the answer.
Just another Airbus trap/Airbus manual trap which also caught me out.
You think there would be a note saying something along these lines under Go Around Engagement Mode conditions. Being Airbus, you have look for the answer under a different topic nothing to do with the Go Around Mode.