I would go along with that.
The PIC is responsible for the safety of the flight, and (assuming a successful outcome), he is the sole judge of that.
Clearly there are some aircraft where most would feel it was safe, eg a PA28 S&L at 3000ft with CAS and wx OK.
Clearly there are other aircraft where it would not be safe, eg small helicopters.
From the R22 POH:
CAUTION
For helicopters with removable controls,
remove left seat controls if person in
that seat is not a rated helicopter pilot.
But even here it is a caution, and it certainly does not read as though they are describing a legal issue.