I'm not an expert on engines, but IMHO it's not the inertia of the fan that governs the spool-up time. The fan is driven by a turbine that is driven by the power generated in the gas generator, and the surge margin of the gas generator does not allow it to spool up faster.