Potentially a very dangerous situation! If you are the pilot in command and another pilot is fiddling with things you have to speak up and you have to be clear and unambiguous.
The best policy would be to anticipate the situation in your pre-flight briefing, eg "I would like you to be responsible for lookout/radio/navigation/writing up the PLOG (whatever you think) and I will be responsible for all flying tasks, if you see something which does not seem right please bring it to my attention and I will decide what to do. Do you have any questions?"
A simple briefing like that sets clear boundaries and should avoid any "task confusion" without giving offence. If the other guy does take offence then you can off-load him or cancel the flight before you even start!
If you are in flight and he is fiddling (whether or not you gave a briefing) then you must speak up and you must be firm. eg "John, please remember that I am flying this leg, if you see something wrong please let me know but do not interfere with the aircraft controls!"
Happy landings
3 Point