I'm not sure about the Airbus family, but on many types the cross feed and refuel lines (gallery) are the same.
In theory, if pumps are on, then certain areas of the refuel 'gallery' could be fighting the aircraft pump pressure during refuelling. At best this could slow refuelling and at worst damage componants.
Maybe.