I'm on the fence about this issue. It came up on this forum fairly recently, too, and I didn't make a comment even though I usually have very strong opinions about unusual attitudes. The reason to get rid of the FD is strong and obvious. But on the other hand, every second in a UA counts, especially a nose-low and/or inverted one. You can ask the other guy to get rid of it for you, but having the mental bandwidth to make that request verbally, or for him to understand and fulfill it, also may not be there.
And "just fly through it" is naive, for the very reason you posted.
Policies requiring the FD to be always switched on are idiotic, and can only be sensible under the premise that any remaining manned aircraft control is only a temporary stopgap until complete automation arrives and the yoke/stick is replaced by a mouse.