As I understand it the altitude is one of the inputs required for the calculation of the FPV.
The FPV is an indication of what the a/c is doing over the ground whereas the flight director is (mostly) related to what is happening with reference to the air mass. The IR knows the groundspeed and drift but also needs an altitude input to work out what FPA the a/c is achieving.
One of the failures that can occur is a partial ADR failure. This is (according to Airbus) quite possible and will result in failure of either the altitude or the speed outputs. If the alt on the #1 PFD fails the FPV will also go off. There is nothing (of course) about this in the FCOMs!
Incidentally the wind readout on the ND gives absolute nonsense readings if the a/c is unbalanced (beta target/lower trapezoid not centered). Therefore I must assume that the FPV in similar circumstances would be useless too...must try it tomorrow!
As always edited for spelling!