Maybe this stuff has been discussed again, sorry to repeat again.
As far as I am aware, VS uses both inertial and baro data. Inertial is for initial movement & disturbance. Where this is not available, the VS value is boxed amber. This is visible when gusty on ground with ADIRS not aligned yet. These changes in pressure make the VS going up and down like crazy. On the other side, baro data is used for the subsequent constant rate of change that is achieved (altitude change).
Thats why FPV is affected. Because FPV uses both VS and groundspeed to calculate the angle displayed in the PFD. If the Baro altitude is unreliable, VS is unreliable. And while instantaneous VS may be correct, the subsequent value would be a non sense one.
Slightly off topic but this conversation reminds me another common debate and misunderstanding: Many confuse the ISIS diagram on FCOM because it shows a data feeding from ADIRS3 to the ISIS. So many are interpreting wrongly that ADIRS3 feed the ISIS with IR data, but, this is only for the optional mod that incorporates heading information to the ISIS (yes, from IR3). ISIS is feeded with static and dynamic pressure from the STBY probes and ports. It generates its own attitude, and thats the purpose of a STBY instrument. No involvement of ADR3 here.