The perspective, visual horizon, and ground rush are enormously significant.
Only turn back if you have planned to do so on the grounds that you knew it was safe. This requires a highish performance aircraft (either in terms of climb angle, glide performance, or both) and a skilled and determined pilot.