I can't imagine that pilots in general, or very many at that, are afraid to hand fly the airplane. We normally hand fly through at least FL180, and I usually hand fly my approaches. We have operational requirements when on RNAV arrivals and when flying by RNAV to have the flight director operational and the autopilot available...doesn't have to be engaged.
The flight director can be a help or a hinderance. Our standard policy is to bias the vertical bar out of the way so it doesn't form a distraction. After takeoff, we'll usually call for airspeed-hold on the FD, but the rule as always is to fly through the flight director, rather than follow it. One should still be flying the airplane, rather than simply following behind it.
I find that if I have questions regarding what the autopilot is doing, my first action is to disconnect the autopilot and handfly. I see that nearly everyone around me does the same.
The autopilot or FD are nothing more than tools; the airplane flies fine with or without them.