I believe the RAF have looked into this on several occasions; perhaps someone can shed some light on their findings. While I don’t have their detailed conclusions to hand, I recall that the consensus was that it was fraught with danger – even when flown by experienced Instructors who were expecting the manoeuvre. They tended to support Old Smokey!
Of interest, one very significant issue is that of speed control as the aircraft rapidly turns from a headwind to a tailwind at very low level. The resulting overwhelming visual cue from the rapidly increasing groundspeed as the turn takes place at very low level causes an instinctive tendency for pilots to reduce airspeed to compensate. Given that such turns often lead people to fly at significant angles of bank, the danger is obvious. This illusion is not helped by the fact that their attention is more on the runway that they are trying to get back to, rather than the ASI which would reveal the extent of the illusion. There is a school of thought that there is a significant risk that many aircraft would be lost in a low level "turnback" due to a stall/spin at low level.
As a glidist, we had a simple rule for a winch launch. The only time a “turnback” was flown was from 700ft. At that height, you actually flew a normal circuit joining the downwind leg midpoint at fairly well a normal height for that part of the circuit anyway. All the rest were modified “land aheads” which were designed to keep the glider inside the airfield boundary. Obviously not possible for powered aircraft but the concept is sound. If you are high enough to fit in to a traditional "glide circuit” go for it. If not, pick a field ahead. In reality, I think the Space Shuttle would be the only craft which could meet the criteria!
For those odd airfields where there is literally nowhere to go (ie a takeoff path over a large built-up area with no obliging playing fields or rivers under the climbout) you may have to consider a "turnback" but, forget the runway. Just head for the first available space behind you. If taking off in such a situation, the pre-take-off brief must specifically deal with this possibility each and every time!