Your tracking of the radial will be less influenced by any wind changes than the IRS drifts themselves.
In reality, it is quite common to see an IRS drift after a couple of hours:
Hence yor bird will "only" be + or - 1 or 2 degree accurate.
So when correctly tracking a radial or an ILS, it's normal to see you get slowly off track, even with no wind. A very small correction (+/- 1 or 2 degrees) on the FCU selected track should compensate that drift.