I'm not sure, but I think that if IRS and GPS positions differ in more than 12 miles, GPS cannot be primary.
Also, in GPS fitted 320s, FMGC position is not GPS position. It is GPIRS position. Therefore you need the IRS, too. Actually, GPIRS position is computed by the ADIRU, not by the FMGC.