I've never HAD to do it, but I have practiced it a few times when I'm certain I'm the only one in the circuit.
Two things impressed me:
1. C152, during initial PPL training, Golden Turn exercise with my instructor. With an engine failure above 800 feet and a USAF runway underneath, we were able to make it back onto the runway without too much problem. I don't remember the wind conditions but I don't think it was more than about 10kts. We were making steep turns and not hanging about in the exercise.
2. Wind, climb rate, and runway length are HUGE factors. To whit, practiced the Golden Turn in a 95 hp Super Cub on a 2,000 ft runway with a decent headwind. I climbed at such a steep angle that, had I to turn back, there was no way I was going to make the runway - I was going to overshoot and end up in the field on the arrival end. I had climbed, routinely, to such an altitude because of both the aircraft's performance and the headwind, that the altitude and headwind became liabilities and there was no way I was going to be able to descend quickly enough to put it back on the runway without significant maneuvering.
My learning point from all this - it's a very complex question, the answer to which can vary enormously from day to day, location to location, and aircraft to aircraft.