distaff, I am taught not to remove any flaps until immediatley after becoming airbourne again. This is in a Pa28 with the 'handbrake' flaps you talk about.
I think the idea is to avoid, (a) the distraction of reaching for the lever, (b) avoid taking too much flaps off, and (c) just in case you pull the gear retract by mistake in a more complex type.
The procedure is to go from full flap to 20 degrees (2/3) as soon as airbourne as this last stage is nearly all drag. You can actually feel the speed build as you do so, without any noticable loss of lift.
I must admit though, that I regularly put that third stage in whilst still on the ground (but only when there's no instructor watching).