The only time these days that I'm flying with another pilot who would think to "control" the aircraft without my request to do so, is when I'm flight testing something (modification approval, not pilot skill). This was a problem before I started doing really thorough preflight briefings. Now (having learned the hard way) I brief:
" I will fly, and I will continue to fly no matter what. If you need to fly, you say "I've got it" and you'll have it - all of it. But, if that happens, the flight test is over, and we're going home - incomplete"
Since I brief that, I've never had a problem. If I were to, I would end the flight, and that pilot would not be right seat to me again.
If you are willing to have the other pilot touch things, discuss it first.
On the obverse, I can think of three times that, having left it too long anyway, I suddenly announced "I've got it", and took over, without negotiation. At that point, I considered it to be life and death. Apparently, for whatever reason, the other pilot did not disagree, as nothing was ever said.
But, I go to a lot of effort to have the the other pilot fly as much as possible. It's only the very demanding things where I might request to fly, by advanced briefing, and agreement. A very tight landing into a small lake, of go around, for example.