Why can't they just pull the throttling thrust levers back a bit?
Because jet aircraft have a minimum speed they can fly at in clean configuration, for the A320 this is generally in the 190-220kt range depending on weight. A jet has a flat drag curve and if you were to fly too slowly drag would increase slowly with little warning and more power would be required until you were at full thrust just to stay airborne. Then you could only accelerate by losing height.
Flap life limits are based on them only being used for take off, initial climb out, approach and landing. A controller can't give you a speed en route which would require flap extension to achieve it, minimum clean speed, holding or radar vectors are used.
There used to be a Cathay interview question regarding what distance out from the field a controller could give you a speed which required flap extension.