The difference in stall speed and hence available lift is not much at the later stages of the flap settings - but the difference in drag is enormous.
This isn't correct, as you can see on the following illustration (B787), the later stages of flaps are having a lower impact on the L/D performances than the earlier. This means lift benefits are more predominant over drag in this stages than in the earlier.
While L/D improvements resulting from flap retraction are visible on this illustration, you have to keep in mind that this is valid provided that the aircraft flies the appropriate Vref. In other words, compare the little L/D improvements that you obtain through flap retraction with the major L/D reduction caused by flying at the minimum speed and you will find out what was the most efficient way to control gliding performances. The lack of speed control caused greater performance deterioration than the flaps retraction could improve.
It's exactly like trying to clear an obstacle during a climb by retracting the landing gear while the aircraft's speed is far from Vx. It only reduces the "negative" effects.