If you want to know the right answer, ask a glider pilot.
As one has already replied that they always have a plan of action which changes with height, I'd like to expand a little.
I case of an engine failure what you need is a decent flat bit of land. What you've just left was a very small 'licenced' area of what is in most cases a very much larger area of very suitable land which a rescue crew, can probably get to within three minutes or less. See the difference it made at Sioux City!
The plan you devize for your departure field may not require a 180. Maybe just a 90 degree turn to a cross runway. Perhaps a gentle 270.
Just as long as you maintain flying speed; and this is the big killer in GA, stall/spin accidents whilst making a forced landing.