You might know this already strIA, if so skip to FCOM stuff at the bottom.
There are 2 modes for the FPV, with and without the Flight Path Director. Operationally we (my airline) do not fly with the FPV+director, so I'll assume for the moment we're talking about FPV alone.
For takeoff and go-around we are concerned primarily with the performance and obtaining the best angle of climb. In both EFATO and G/A cases we assume the power is fixed (at or close to maximum) and the attitude is the principle driver of airspeed and therefore performance. The FDs will give you an instantaneous, clear and accurate indication of required pitch to achieve the speed you require. The FPV on the other hand is not a command, it will show only your angle of climb (or descent!) thus removing an important pilot cue for pitch required. For this reason Airbus recommend against using it for takeoff.
That being said if you suffered an unexpected AP/FD failure it won't do any harm to have the FDs turned off and the FPV displayed.
PRO-SUP-31
There is inevitably some lag in-between the pilot's raising the nose to commence the go-around and the aircraft's responding by changing its trajectory. For the same reason the pilot does not use the FPV on take-off.
DSC-22-30-20
FD bars are automatically restored [G/A modes] at go-around engagement. If FPV/FPD was previously selected, it reverts to FD bars.