Hardest thing for me is getting called out of standby at 6pm for a 2-sector 2-crew night turn and trying to land at 8am having being on duty for 12 hours and awake for nearly 24 hrs.
Otherwise the hardest bit is keeping fatigue at bay and finding time to do all those things which work gets in the way of: eating healthily, exercising and sleeping.
When the aircraft doors are shut it's the best job. When the doors are open you have no authority or control.