With the greatest of respect, and i mean that so try to take this as a constructive post.
The situation you faced was poor, BUT, your behaviour was also problematic. Pilots have a process called NUTA - Notice a problem, Understand what difficulty it may cause you and, Think Ahead - ways to avoid the problem. Sounds like your wife did this and you overrode her sensible suggestion. It is far less stressful sitting comfortably watching someone else have a problem than to experience it yourself.
As others have suggested this is one to write off to experience and avoid similar problems in the future. Your problems could be a lot worse and more expensive once you have been off-loaded. I don't think many captains would have been so tolerant of one passenger delaying his flight. He is responsible for ALL his pax and a fight one vs the rest is one you will always lose. Chalk it up to experience and move on.