This is the catch, isn't it? If you, either implicitly or explicitly, agreed from the outset that the "passenger" sitting next to you could intervene if he saw fit at some time, then he's not really a passenger and you're not really "in command", are you?
(a) Change of PIC mid flight?
(b) Safety pilot?