I was given this explanation, I'm not qualified to vouch for its correctness:
Use of flap changes the spanwise pressure distribution of the wing. Since flaps tend to be on the inboard portion of the wing, the change of the pressure distribution will decrease stability in roll. This will cause obvious problems in gusty cross-wind conditions so a lesser flap setting may improve things.