" and put the phone down"
There's an even better response on a mobile.
If you don't recognise the number, don't answer . Those that are genuine will leave a voicemail message so you can simply call them back.
Being a bit thick, this seems to work very well for me. Along with checking and blocking the number if no message is left.