I think it is more to do with the fact that most people don't understand the cumulative effects of shift work, siting still for 11 hours with only a brief break to stand up, cabin pressure at 8000ft, constant moderate noise, high mental processing demands in short bursts, and an expectation of consistent high standards of performance combined with significant responsibility and the toll that takes over a period of months and years.
If I show my neighbour a roster that has absolutely bowled me over he will most likely look at the numbers and start a story about how much he has worked this month and won't ( can't ) factor in all f the influences I listed above as he has never experienced them all at once. Other pilots understand. Even cabin crew find it hard to believe our job is taxing.