If you were speaking about letting a passenger in your car take over you might have a point, but you are taking an unnecessarily strict line, in my view
I've often let passengers take the controls. I brief them to give back control when I ask for it and have never had a problem. After all the most they can to is to go off track or up and down and I have hundreds of feet available to recover the situation. In one case I let a blind person fly and they loved being in control.
I would say that if I am flying in congested airspace or close to CAS then I take over