He sort of has a point. You can fly below S speed, as u demonstrated, but from an operational point of view, I agree that u probably should have extended flaps, although I can see ur point for wanting to keep them in.
Green dot, S and F are maneuvering speeds in the selected config. So although the plane can safely be flown below the speeds in that config, you may not have adequete protection against stall for all the maneuvers that the plane is certified for. It's highly unlikely that you will have to push the plane to the edge of the envelope like that in a terminal though. He should have said that u SHOULD not fly below S speed.
VLS is the lowest speed that the autothrust will maintain. You can fly below this speed in manual flight.