I wouldn't try to do it all in one formula; rather, I'd split the problem in to multiple cells. My way involves using formulas to split off the overtime hours. Rather than try to explain it, I threw an example together,
here. (The IF formulas in C2 and D2 are because you don't want negative hours popping up.)