Pilot DAR, if I maintained attitude while retracting flaps, I completely agree with your assertion. I had hoped that maintaining speed would avoid the settling, etc, but I dare say I am still missing something. When I get the a chance I will try it, (at a safe altitude).
24Carrot, perhaps this explanation may help a little.
When retracting flap in a glide, there is a reduction in (coefficient of) lift assuming all else remains the same. Due to inertia, it takes time for the aircraft to accelerate to restore the lift force. As the flaps are retracting and the aircraft is accelerating, the aircraft will descend more steeply due to the loss of lift. There is therefore a momentary sink associated with flap retraction.
If you are close to the ground, avoid the momentary sink associated with flap retraction as it will be far more significant than the slightly improved glide angle associated with less drag.
If you are still quite high, you might accept the (relatively) minor sink associated with flap retraction in favour of improved glide performance for a prolonged glide to a still-distant landing site.