A sudden application of TOGA may not be such a good idea at low speed, if you have taken off in FLEX your speeds are based on this power setting. It would be good practice to ensure you have control and won't be destabilised by a sudden increase in asymmetric power.
We use FLEX for take off and would first look for a positive rate of climb, raise the gear and apply TOGA in that order, in the event of an engine failure. By this stage you should have directional control sorted out and would only need a small increase in rudder, and be able to stay within drift limits.