At which point the Captain should have pushed back and not let him off!
And risk having to return immediately to the point of departure to disembark a disruptive passenger? I think kicking him off was the best course of action, I hope he gets banned for life from the airline.