Definitely the working hours/ earlies /lates /jetlag / not being home when you want to..
That all depends on your company/fleet though.
Been flying for 4 years and the job is still just real fun for me. I'm always relaxed.
Most of the work is definitely on the ground, especially when dealing with delays/technicals/pax trouble etc but you grow into managing it.
Once up there, it can get a bit boring sometimes. I have seen some people doing nothing in cruise for hours, it's something I can't do but to each his own way of killing time.
Sometimes it really comes down to who you are with, be it in the cockpit or downroute. The right people can make any day really enjoyable and relaxing.