Without F/D, PF can fall into the trap where they assume autothrust in speed mode. However, if the PNF has their F/D on say in 'Op Des' mode, then autothrust is in idle mode, NOT controlling speed.
Hence the rule, for simplicity is, both on (and followed) or both off.