If you are in a situation having lost System B and then go-around, the last thing you want is to need to hold your hand on the switch to retract the flaps.
In very worst case: if an engine failed and you need to go around without System B...
Retracting the flaps must be done immediately when terrain is a factor especially and flying the aircraft requires a PM not to be distracted or preoccupied holding a switch in the UP position