Plus of course you must meet the minimum speed requirement for the flap retraction schedule for the aircraft in question.
If you are low energy you may not be able to take even the last stage back in.
Ultimately if you are very low energy then ground contact may be inevitable.
At the risk of pointing out the obvious, this is why we don't fly dangerously slow in the approach..... because it's dangerous!