I'm not saying leave the controls alone entirely, just that the first point of order should be to see if the aircraft will remain relatively stable without encouragement - which in most cases it should!
If I recall, a Lion Air Indonesia B737 went into a gradual spiral at night with devastating consequences. Seems the crew were having difficulties with one of the to IRS. in fact there were many reported snags on that IRS previously.
Again, from memory, the crew decided to trouble shoot and switched off one of the IRS and in turn this caused the automatic pilot to disconnect. Either the crew didn't realise the autopilot had disengaged or if they did, neither pilot was allotted the task of flying the aircraft manually.
While one pilot read from the QRH about IRS non-normal, the other pilot selected the perceived offending IRS to the Attitude mode. This blanked out the PFD on one side. Meanwhile the 737 wandered around the sky at night with no one watching the flight instruments. It eventually went into a gentle spiral which got worse. By the time someone exclaimed WTF, the aircraft was in an unusual attitude.
The captain didn't have a clue how to recover from a UA and instead pulled back hard on the control column while in the spiral. It doesn't take too much imagination to guess why the aircraft eventually broke up in mid-air.
the point is one has to handfly routinely in order to be better equipped to deal with more exotic flight conditions...
And therefore the Lion Air crew went in...