Part of the problem is the way avionics are made. Boeing and Airbus doesn't have an HSI mode which supports FMS navigation, so once you're given direct to some fix, you've got to have the FD on. In my old airplane, the FMS could drive the CDI, and it was not unusual to hand fly an RNAV SID/STAR or approach. The only time you really get to practice hand flying without the FD nowadays, is while you're getting vectored, or on an approach. At least once a week, I try and turn off the FD/AT and do the approach raw data. Day/night/VMC/IMC shouldn't matter.