The only time these days that I'm flying with another pilot who would think to "control" the aircraft without my request to do so, is when I'm flight testing something (modification approval, not pilot skill). This was a problem before I started doing really thorough preflight briefings. Now (having learned the hard way) I brief:
" I will fly, and I will continue to fly no matter what. If you need to fly, you say "I've got it" and you'll have it - all of it. But, if that happens, the flight test is over, and we're going home - incomplete"
I've had exactly the same issue, and come to the same basic solution.
One thing you have to remember is that mosts PPLs have had no training in CRM, nobody has discussed issues like cockpit authority gradient with them, and they really don't understand how dangerous these behaviours can be.
I do invite people in the right hand seat to advise me of anything that concerns them at any point, but that they must ONLY do so verbally.
G