Er, I think Airbus might have thought of this before. I'm not rated, but as far as I know the stick's can do something like this:
LH pilot flying.
RH pilot wants to take over, presses the override button and immediately gets control.
LH pilot can then do the same, once only. If he takes it back, no more swapping is allowed.
Something like that. The mention of Airbus seems to get everybody all excited about 'designing pilots out', but from what I gather speaking to my international friends, it all boils down to A and B doing something along similar lines anyway nowdays.