This is a common problem with automatic recording of engine parameters. Manufacturers are normally only interested in the 'eyeball' figures as there are errors normally associated with the airborne data gathering.
I don't know what system you are referring to, but most systems are very accurate. In fact the eyeball errors are just what the manufacutrers want to get rid of.
Most data gathering systems will only record figures to 12-bit accuracy, this leads to numbers being rounded up (or down) depending on the installation. Without going into the boring stuff, if your engine is doing 29,994 RPM gas generator, the recorded number might be anywhere from 29,982 to 30,006 RPM.
I think most if not all monitoring systems use Hertz as their units, then 29994 RPM would be read as 499.9 Hz and 30,006 RPM would be read as 500.1 Hz, their accuracy would be within the 12-bit accuracy you state.
I know what you are going to say now "How come they don't accept those figure when my mark one eyeball is not accurate enough to tell the difference from 29,960 to 30,000 even on a good day?"
I am not sure what you mean here.
Another problem is hysteresis of the cockpit indicators. This is taken into account on the graphs, but the onboard equipment does not. Figures taken during a transistion where the engine is accelerating can mislead the automatic algorithms and give a fail.
The algorithms should be using steady state and probably use averaging anyway. You are right about the hysteresis of the cockpit indicator. I have seen many times where gage accuracy has caused a failed power assurance indication due to Ng speed. A proper check of the gage showed it reading higher than it should have by a couple of percent. Many aircraft gages have a 2% tolerance because of hysteresis. This is where I agree that you can't set the limit on an onboard system to have a tighter tolerance than the gage as the gage is all the pilot can fly it to. However, if the onboard system differs than what the pilot reads from his indicators then a check should be made of both systems using a calibrated input to determine which system is in error greater than allowed.
Regards