I've hand-flown around 10 sectors due to either Yaw Damper INOP or Autopilot INOP. There is a difference.
Apart from being pretty shagged at the end of the day, the best way to describe it is that at the end of the day your body "remembers" the slight left-right-left-right rocking motion.
IMHO, frequent flyers may notice, as well as Cabin crew.