If you are above the minimum flap retraction altitude before the engine fails, then it's highly likely that you are far above the minimum climb gradient, which was predicated on climb performance following an engine out at V1. There is no harm in levelling off immediately and retracting the flaps, so long as you are following the engine out path until reaching MSA or the minimum radar vectoring altitude.