The second that happens, I'll leave the industry for good. A fair few will be doing the same I wager.
Whilst you could argue that it is safer to have the automatics in all the time, we have a pretty good record here in the UK whilst allowing hand flying where appropriate. (He says whilst looking for a wooden object to touch!)
How safe is safe? There is always some element of risk that cannot be accounted for. Hand flying is not the biggest issue, CFIT still happens when "George" is in.