It is a fundamental principle of criminal law that if something isn't expressly prohibited, it isn't illegal. As no one has been able to point to any specific regulatory provision, I'd say that it is fairly safe to permit a pax to handle the controls.
Now admittedly, 'they' could go after a pilot under some sort of catch-all provision prohibiting "reckless" or "negligent" conduct, but absent unusual circumstances it would likely be a tough sell before any fair-minded judge or tribunal.
BTW, who among us hasn't asked a pax to take the controls when you need a short break or have to check a chart or approach plate, etc. That's what happens in the real world, surely? I wouldn't let a non-pilot handle the takeoff or landing, but cruise is another matter.