The FPV needs air data to compensate over time.
Inertial data will only provide accurate accelerations. Velocities will always drift with time and to accurately indicate the vertical component of the FPV, you'll need to correct for those errors and that's where air data comes into play.
Just like the IVSI uses inertial data for the initial and fast pointer deflection, the long term averaged vertical speed comes from air data.
The sim replicates this nicely when the statics are blocked: you'll see the VSI fluctuate if you climb / descend but it settles back down (due to the blocked ports).