Sincerely don't think a "direct" CONF1 selection (from CONF3) would be safe for a one engine inoperative go-around, cause of the sudden lift loss combined to a relatively low thrust; instead it would only be detrimental on a/c performances. Here's my reasoning:
- Unpredicted wind-shears
- High airport altitude
- Low pressure
- High OAT
- High weight ...
It's perfectly safe and it is the normal technique used in the simulator for LST and LPC. Bear in mind that the acceleration is done level, not climbing (Push to Level Off - V/S 0). The likely exception, as I wrote above, is for close-in turns in the EOSID or GA procedure with an associated speed restriction when it might be appropriate to select flap 2 due to the increase of indicated A-prot speed in manoeuvres.
Windshear CONDITIONS are hardly unpredicted (Cb, frontal passage) and if that were the case, the advice is to fly a flap 3 approach so the go-around would be initially in flap 2.
High altitude, low pressure, high temperature and high AUW are all taken care of by the landing performance calculation and provided the aircraft is below the calculated maximum weight and can make the necessary Missed Approach Climb Gradient there is ample performance on one engine in TOGA power. Generally speaking in Europe there is no GA weight limit and a 320 can make 5%+ gradient. In the Middle East in summer it may well be different.
HTH