Your observation is correct, in certain cases. For example, if the primary governor failure resulted from any failure in the components that drive the hollow gear-shaft and turn the flyweights, that will also take out the Nf governor.
However, if the failure of the primary governor to regulate oil supply to the propeller arises from any form of impaired movement or breakage of the pilot valve - the pilot valve being the shaft that moves vertically within the rotating hollow gearshaft - then the Nf governor will still be functional. The Nf governor will continue to function as long as the hollow gearshaft and the two flyweights are still turning in correct proportion to Np.
The cut-in Np of the Nf governor is determined by the preload pressure exerted on the coil spring in the top of the primary governor assembly (not the spring you see on top of the exterior of the assembly, but another spring inside the very top of the dome). This is the spring that we, as pilots, adjust preload on when we operate the propeller lever. The pilot valve itself is physically independent of this coil spring, although its range of movement is directly affected by spring preload.