This will happen, do you need to worry about it... no.
FCOM states Flap retraction at s speed AND accelerating.
S speed is 1.29 Vs, while Vls in clean config is 1.28 VS so on a normal day there shouldn't be any overlap.
Only my opinion but consider a Config 1 + F Take-off. At you ACC ALT you are well below S speed but the aircraft will accelerate towards S speed. This is similar, but you are accelerating to and past you manouvring speed of Green Dot instead.
At S speed when you select Flap 0 it may be just below VLS, though it is about 56kts above the stall speed. Just relax. you will find its one of those French Autoflight laws regarding negative G's. It has fully authority but usually wont command the full and abrupt pitch down until you get to about VLS -5 or so.
If it doesn't fix itself up, yeah sure follow the golden rules, disconnect, and manually pitch down as you did... good call. In my experience, it would "eventually" pitch down, just beware the "Garfield" (Alpha Prot) during the turbulent and hot conditions in this scenario because this may make things a little bit trickier.