As I don't work for an airline, I havn't expereinced crewing calling me, although it does happen to some extent with the flying I do.
However, I do have alot of experience of it happening with my (pre-flying) care jobs. I have often used the tecnique suggested by fireflybob of saying I will call back in ten minutes if I got a call to work in off time. It gave me time to consider whether I wanted to do the extra work, if I was legitamatley supposed to be somewhere else, or time to think of a good excuse! Like fireflybob says, it puts you back into control. Obviously, most companies (in care work and in airlines,) hopefully wouldn't call you in unless it was neccasary. Some are less thoughtfull and then perhaps its time to move on. However, it is much easier (especially if you are experienced) to find a better care job than get a flying job!)
In a round-the-clock operation like a care-home as in aviation, anti-social hours and personal life distruption are part of the job. (Bear in mind please, that even a relatively low paid T-P FO gets around twice what a good care worker earns!)
Friends that have decent 'Nine to Five' jobs often have disruption, overtime, meetings that entail overnight stays in some ****e business hotel in the midlands, and get to the pub on a planned night out only to go home early because they're too shagged out after the week to enjoy themselves. I'm not yet earning much as a pilot should (about the same as I was as a care worker) but they often say to me "I wish I was doing what you are doing. I hate my job"
I think life is tough outside of aviation too!