for me one of the most worrying times is getting the "leans" when single pilot IFR without autopilot when you know that you have no choice but to get over it because you have a good amount of time before being VMC again. The biggest fear has to be a fire onboard that means getting on the ground asap!