DFC has this is right IMHO. I am happy to fly as a passenger with a few (very select few) pilots who I trust....For the rest I make it clear that I am the PIC and all in flight decisions will be either made directly by me or take place after I have given my concurence. Furthermore if I at any time I declare "I have control" they agree they will instantly and without question hand control over to me. If they are unhappy with these conditions I will respectfully but firmly tell them I am not the right person to fly with them.
It is absolutely vital that the cockpit rules of the game be firmly established before takeoff...and there is always only one PIC !