Think about what your "minimum speed" was in the takeoff configuration BEFORE you started retracting the flaps. You got airborne shortly after VR, and didn't achieve V2+10-15 until you were well airborne. So you were flying - but accelerating - well below the initial takeoff flap manoeuvring speed as well.
No-one questions whether it is "safe" to fly in the early stages of a takeoff below the "manoeuvring speed" and for the same reason, flap retraction as you have described it is perfectly logical and consistent.