By far and away the biggest hassle is the constant tension between work and social/family life. The best mitigation is part time and planning to retire as early as you are able. The aggravating factors depend on the airline, but poor rostering, inefficient leave systems, commuting, sleep management, job insecurity, stress around sim/medicals and poor management are all to be considered.
I have rarely met a pilot who doesn't enjoy the job between doors closed and doors opening, but I have met some who find any of the above factors unbearable.
Perhaps, it is a career most suited to single people!