The only legal definition of pilot in command refers to being responsible for the safety of the flight, there is no mention of having your hands on or manipulating the controls.
There is nothing in the law that says it is illegal to let a passenger manipulate the controls, you as pilot in command are still the one responsible for the safe operation of the flight.
This is different from driving or any of the other comparisons drawn as they don't have dual controls, and usually have a provisional licence requirement.