(unless they feel have to intervene)
This is the catch, isn't it? If you, either implicitly or explicitly, agreed from the outset that the "passenger" sitting next to you could intervene if he saw fit at some time, then he's not really a passenger and you're not really "in command", are you?
does any one actually care...will any one ever find out ?
Usually not, no. Usually the instructor will not intervene, the flight completes successfully, nobody ever looks at all three logbooks (2 persons + plane) together.
But if something goes wrong, regardless of whose fault it is (the pilot, the instructor, the aircraft/engine or the bird) the FIRST THING the authorities do is ask for all paperwork regarding the legality of the flight. And if you've logged PIC but at the same time you've agreed that the instructor could take over if and when he saw fit, well, there will be a lot of frowning.