No... Pushing TOGA with the A/T disengaged will result in F/D popping up and commanding a G/A without any increase in thrust. If the A/T is left armed then its a different ball game. Problem is, you may get the low speed protection kicking in at 30' with the intent of recovering a slighty low speed, but at 27' it forgets all about that and retards the thrust levers (ouch). Thats why some folks above have talked about removing it at 50-100'.
Me? I leave it in all the way. If the low speed protection does anything below 100' then it goes.