I very much doubt the performance penalty is even close to engine-out performance, but it could be a problem on a RTOW take off in terrain. Under those circumstances, following any emergency turn procedure may be prudent or even necessary. I think it's what I'd do. On a take off without terrain issues, though, or one well under RTOW, I'd follow the SID. I think it's one of those things where you need to (quickly) weigh the circumstances of the event rather than have a rigid rule.