The ANO provides the answer to a pushy passenger.
Section 10, Article 87.
Words along the lines of "My apologies but as commander of the aircraft, in my opinion the flight cannot be safely made" is all that is needed.
I'd willingly go to court or to any employment tribunal having said that.
I have had cause to say it in the past, thankfully very infrequently and I've not had to take it any further. I have walked away from the aircraft though.....