No. Assuming a Flap 5 departure, as you pass "5" and you are accelerating, you select Flap 1. As you pass "1" you select Flaps Up. Main points to note are that you must be accelerating and if you fly 'at' any of the selected flap settings you need to fly at the corresponding speed for that setting (i.e. if you stopped retracting the flaps at Flap 1 for some reason, you need to fly at a minimum of "1"). I have had pilots just call for the start of flap retraction without checking that their attitude actually has the aircraft accelerating.
If you get a flap malfunction during flap retraction, then you need to fly the minimum speed for the minimum amount of flap you definitely have, say if they stopped moving towards Flap 1 upon selection from Flap 5, then you need to fly the Flap 1 ("1") speed as a minimum, as that is the maximum amount of flap you can say you definitely have... more of an issue if you are decelerating and have already moved the speed bug to the next flap speed. When decelerating we always teach PF to call for the flap required, PM select it, PF (and PM) check the Flap indicator to ensure the flaps are running before PF selects the next speed with the MCP speed bug.
PP
Edited to add that 5150 beat me to it!