and the 'swing' is sideways movement of the ball in flight due to the rough side of the ball experiencing higher pressure than the smooth or polished side,
Not higher
pressure but higher
drag. IIRC, the ball swings towards the rough side, away fom the polished side, or at least mine mostly did. In other words, the drag on the rough side pulls that side back more.
You could achieve a similar effect by making the upper surface of the wing more draggy, but that would defeat the objective. The goal is not an upward force at any cost, but rather the optimisation of lift to drag. Merely increasing drag doesn't help.