Actually the response is fairly simple;
"I couldn't legally take off from an equipped airfield, therefore I can't legally or safely take off from this garden"
No matter what the owner says/does the pilot is covered, I know the risks of being jobless, but at least if you can provide yourself with irrefutable evidence then he will think twice as you the pilot are less likely to take his threats.
SND