My youngest is a fine instrument pilot, by necessity as she first started taking the controls when too short to easily see forwards !
My hands are never far from the controls, I'm obviously PIC, and as has been hinted above, should anything happen, who would know
There is actually a safety advantage in having passengers know how to handle a plane.