I made the mistake of mentioning this on my MEI checkride and the examiner - a legend of sorts in the area and a career that dated back to WWII said "What nonsense. Watch me."
Then we did a series of steep turns, in both directions at pattern altitude with him pointing at the ASI saying "So long as you keep your speed up, it doesn't mattter which way you turn. It's all about speed."
Quite an informative lesson really.
On another note, with an EFATO, other considerations come into play, such as the already mentioned avoiding built up areas. Others might include turning INTO the stiff crosswind to reduce the amount of turn needed, heading for a parllel runway, heading for an alternate runway, heading for an alternate airport or lake or parking lot, etc etc etc. For instance, if you leave Love Field on 18, you've got 17 at Redbird directly ahead of you. No turns necessary.