It becomes a self-fullfilled prophecy. The less hand flying we do ->
the worst we are at doing it -> the more dangerous it becomes to do it.
My take on it is that since neither the autopilot nor the autothrust is a MEL no-go item I must be proficient in flying the AC without them. I wasn't born knowing how to fly a jet and practice is the only way I can be any good at it. I do it regularly, if the situation allows it and not just before a sim, and I don't feel overly taxed by handflying or by monitoring a handflying pilot. Equally I am not too proud to reconnect the autopilot halfway through an approach if it gets too busy or complicated.
As for pax comfort I bet anything they don't notice. As for efficiency at least I don't fly a four mile level segment at 3000' or do geometric descent paths between two altitude constraints.