If you were a weight lifter you would be far more stable if you held the weights with the arms, above your head, as far apart as possible. If the arms are brought closer together it becomes harder to stop the weights rocking. when flap 40 is selected the centre of lift moves along the wing towards the fuselage, therefore it becomes move sensitive to lateral displacement.
There are notes in Airbus manuals advising using Flap 3, instead of Full flap, in turbulent cross winds.