The movement of a cricket ball is connected to the way the flow seperates from the surface of the ball; any asymmetry in the flow seperation will induce a force upon the ball, causing it to deviate from the 'straight' line.
Roughness, and indeed also the seam, can influence the seperation characteristics, by changing the transition from laminar to turbulent boundary layer flow, which changes how the flow behaves. Turbulent boundary layers seperate less easily, so the flow on the rough side, or the side with the seam forward, will tend to stay attached longer. As a result the wake becomes larger on the other, shiny, side, and the asymmetric wake will tend to push the ball towards the rough/seam direction.
Incidentally, aircraft DO use this behaviour - but since it's concerned with controlling seperations, and the best solution is to have no seperation at all, we generally attempt to design airfoils etc to minimise them.
However, everything must seperate eventually, and so if the nature of the seperation is important various devices - like vortex generators, typically - are used to control the way the flow seperates - although usually we're striving for symmetric behaviour, not asymmetric.