Ahhh, the shed vortex; I knew it would be something simple like that.
When you transition, you're often flying past a hangar, and when you hit the turbulence coming off the building (the 'shed vortex') it buffets you around.
Dead easy, this aerodynamics!