Here is the big thing - what is safety?
If you fly all approaches on AP and mandate its use, when something different happens (even like a visual approach) then you are eroding safety margins MORE because you are out of practice. What happens when you fly into Tel Aviv and they give you a visual onto 30 because the wind is out of limits? I have flown that with a very nervous captain on a gin clear day. He was worried we would screw up - its only a visual approach for goodness sake.
The guys who I fly with who regulary practice hand flying with and without the flight director are much better at everything, because their capacity is higher and they understand what is happening more.
The day before yesterday I turned the flight director off because my flying was better without it. Nailing the flight director got me levelled off 100 feet below my chosen altitude - OK, I should have scanned more, but that is the problem with a flight director - it erodes your scan, hence the fact that I turned it off, scanned more and flew more accurately.
I never want to fly with someone who thinks his hand flying is so dangerous that he shouldn't do it with passengers on board. If that is true he shouldn't be flying. Full stop.