I'd always give another pilot that I take flying a chance to fly the aeroplane.
Equally, I'm very clear when it's NOT their turn.
Can't say I've ever had a problem - maybe you need to think about how you agree to do the flying, and how you do the brief?
G