There are a few questions here indeed.
The aircraft, if indeed it was in the middle of the Gulf, would have been just outside the Malaysian radar cell and about to enter Vietnam's. Modern birds like the 777 use ADSB (which is what you see in apps like Flight Radar 24, it is NOT a radar feed).
The thought of the aircraft turning back comes not from a radar observation but from ACARS. More investigation is needed. If it was from radar (assuming the return was spotted), means very little. Radar does NOT tell you which direction the aircraft is facing, it simply gives you the next return. For example, you can be going north and move in a sideslip to the right, the radar will show your return to have turned.
If the data came from ACARS it is a different story, but still inconclusive. It could be showing a violent breakup. Without other data such as speed and amount of correlated data showing the "turn", there is not much to go by.
The gulf does not have a lot of strong currents, so any debirs, once found, should be (hopefully) localised.
I am starting to wonder if the initial hunch to look to the north-west and possibly even terrain south west of Ca Mau and around Kota Bharu is correct!