The Airbus Instructor support handbook I have clearly says this in Intermediate approach section
You may fly BELOW the maneuvering speed of the current configuration, provided you fly ABOVE VLS.
If I am given a speed to fly by ATC whch happens to be below O F S speed I just querry ATC and ask them if it is ok to fly O F S speed. In most cases they have no problem.
Do not forget along with alpha floor we also have alpha lock in case you retract the flaps below S F speeds.