As you can see, the bush itself is not threaded (internally or externally), so rotating the bush to adjust incidence will NOT push or pull the rear spar laterally, so there should be no sweep changes when the incidence is adjusted.
I think that would be true if the bolts were mounted side on to the fuselage but I believe the bolts are mounted fore/aft so surely if the eccentricity was left or right it will have the effect of moving the rear spar inboard or outboard and thus affecting the sweep slightly?
Edit: beaten in great detail by Mr Dar!