Both FDs OFF
If you will not use the FD and follow their orders then you should turn them off....because as long as one of them is still on, it directs you to what is set on the FCU, and if you dont follow that then you risk getting what is called FLIGHT MODE REVERSION.....that is a by-product of the flight safety envelopes installed in all the Airbus modern A/C starting from A320....
and its stated clearly in the FLIGHT CREW TRAINING MANUAL :
MODE REVERSIONS
GENERAL
Mode reversions are automatic mode changes that unexpectedly occur, but are designed to ensure coherent AP, FD, and A/THR operations, in conjunction with flight crew input (or when entering a F-PLN discontinuity).
For example, a reversion will occur, when the flight crew:
• Changes the FCU ALT target in specific conditions
• Engages a mode on one axis, that will automatically disengage the associated mode on the other axis
• Manually flies the aircraft with the FD on, but does not follow the FD orders, which leads to the aircraft to the limits of the flight envelope.
Due to the unexpected nature of their occurrence, the FMA should be closely-monitored for mode reversions.
also it has been stated
USE OF THE FD WITHOUT THE AP
When manually flying the aircraft with the FDs on, the FD bars or the FPD symbol provide lateral and vertical orders, in accordance with the active modes that the flight crew selects.
Therefore:
‐ Fly with a centered FD or FPD
‐ If not using FD orders, turn off the FD.