Just don't answer the phone!!! If you accidentally pick it up - just say you don't know where HE/SHE ie YOU is. 'Back on ..... (your next rostered duty)' - and say goodbye.
There is no voice recognition, you may be thousands of miles away on a bad line. Just don't get involved in a chat.
Easy really............