For Robinson R22 and R44 the Pilots Operating Handbook gives a sub title "Caution" and states;
R22: "For helicopters with removable controls, remove left seat controls if person in that seat is not a rated helicopter pilot"
R44: "Removable controls should be removed if person in forward left seat is not a rated helicopter pilot."
Whilst this is not in the limitations section of the manuals, but in the normal procedures, it would no doubt be deemed to be irresponsible if you went against this advice. Your local Air Navigation Order/FAR OPS will go into detail on which sections of the flight manual are law, and which are advisory.