either consciously or sub consciously has finger poised to hit TOGA
A dangerous habit but common. A more prudent technique is to manually advance the thrust levers towards go-around thrust while pitching up as part of the GA procedure. Once it is evident that the thrust levers are in the high power range, only then press TOGA which in most aircraft will place the FD's in the TOGA mode (if you need them) Depends on the aircraft type.
This reduces the chances of inadvertently selecting an incorrect mode while fiddling with the TOGA buttons while at the same time giving a moderately high enough thrust setting enabling a safe initial climb away gradient.