No need to get crazy on these switches either
Ok,if you press TOGA once,you will get the FD into GA mode and reduced GA thrust.( your FMA will read GA/TOGA/CMD) GA is reduced GA thrust,TOGA is for your pitch,and CMD is cause your AP is engaged.
When the thrust is stable, click one more time:and you will get full GA thrust(FMC N1Limit) and your FMA will read N1/TOGA/CMD.
In brief, when you see N1/TOGA, stop clicking
I think the FMA won't change as you're still on the GA N1 limit. You're not changing the thrust limit by the second TOGA press, just how much of that mode you're getting, whether it's a thrust for 1-2000fpm or full GA thrust.
You shouldn't get an N1 indication until another vertical mode or you manually select N1.