O rings very certainly can swell. Particularly if the O ring composition is not compatible with the fuel. In plunger primers, swelled O rings are usually first noticed as the primer being rather stiff, is it has more swelled surface area in contact with the bore. Primers also go stiff because they need lubrication restored with gasoline proof grease, but that's another story.
Primers are easy and accessible to maintain, so there really is no excuse for flying with one that's in poor condition...