Thanks, this has been helpul in discussing with everyone.
What was concernign me was that I had failed in my role, by effectively reaching the conculsion that the guy was not just a total pain but counter-productive and adversely affecting our flight safety and performance.
Worse still, I began to doubt my ability to control myself in the situation and 'turn it around.'
My gut feeling is that I dont want to ever fly with this guy again, but it's inevitable that we will at some point. I will be having a very frank (but calm) conversation with him in pre-flight.