The safety issue is a refusal to follow crew instructions, which amounts to disruptive behaviour and is a threat to discipline and good order on the aircraft. A common precursor to airborne Air Rage incidents.
I'm really surprised any airline pilot would side with this guy, given your responsibilities if it later goes pear shaped. If you like taking risks on human behaviour then go ahead - I hope I never fly on your aircraft and end up diverting into some hole in the back of beyond on a dark winter night thanks to a 'nice old guy' gone berserk.
Get him off - let him sue. No longer your problem. That's the correct course of action.
The means are his choice.