Whyeman
Is it your opinion that hand flying an airliner in a VMC day, non-busy airspace, not being fatigued, etc... Poses any risk?
If so, visual approaches should be banned, let alone circling approaches!
I have hand flown like that... With my wife and my brother seating in the pax cabin. I feel no remorse, really.
However I see your point.
I would not handfly in IMC, busy environment, etc... Because when you can't see, you never know if you have missed something and may be not readily apparent you did until it is too late. I would not hand fly in many circumstances with or without my family on board.
When a pilot is too passive tends to become sleepy, well back in the performance curve. Switching off AP/FD wakes you up and moves you to the optimum in the curve. In that case you are safer than auto flying. It might be the difference between detecting or not an A/THR subtle disconnection, for instance.
Furthermore, in the event of a failure involving handflying, non skilled pilots will have a lot of extra workload and stress, putting them too far to the right of the curve. Not so with pilots skilled in hand flying.
I think Automation is great, that we must master automated flight... and keep skilled in handflying so that safety is at a maximum.