It's not a golden rule, but it is Airbus SOP for the reasons listed above. It's not that the FDs are a problem, it's the interaction with the autothrust.
FDs should be off whenever you aren't going to follow them. Examples include TCAS RAs, visual approaches, NPAs once you are below MDA, and hand flown departure procedures with no coding in the FMS.