Ah ok, it all becomes clear now. You were accelerating already. The fact that you were accelerating means you pushed V/S or the ALT button, because the A320 does not accelerate out of its own on single engine. If it did, I dare say that it is a sim fault. But nevermind that, the fact that you accelerate means that SRS is not active anymore, you were probably in V/S or ALT mode. That means that if you subsequently select TOGA....you get the G/A mode. I know, I did the same not too long ago when I wanted to speed things up after the n'th engine failure for the day
That's how you learn!
I'm not sure what your instructor is saying about the acceleration altitude itself having any effect on things. It's too early to open my FCOM.