I have never noticed a mis-match, but then I normally don't looks at the FFPM for my N1 setting.
So are you saying you calculate and then use a different N1 from the FFPM, instead of using the N1 the FMC is using? Then you manually set the N1 you want?
Are you using OAT and airport elevation or sensed TAT and p-alt (what the FMC is using), as they/that might account for the difference. If so, is your TAT reading accurate before the engines are running?