I too ofen see the other pilot get a bit phased by selection of a speed below GD or F or S. As far as I understand it, as long as you are above VLS, you are safe. GD is just min drag and F or S are just selection references.
From A320 FCTM 02.080 - Holding Speed and Configuration:
'If green dot speed is greater than the ICAO or state maximum holding speed, the crew should select flap 1 below 20,000' and fly S speed. Fuel consumption will be increased when holding in anything other than clean configuration and green dot speed'.
From A320 FCTM 02.100 - Intermediate Approach:
'If selected speed is used to comply with ATC, the requested speed should be selected on the FCU. A speed below the manoeuvring speed of the present configuration may be selected provided it is above VLS'.