Superpilot - I haven't seen this behaviour on A320...On A319/321 it does it because that is the speedbrake extending further!
Flighttime -
Watch your speedbrake use on the 321 - It is inhibited in Conf 3/Full and also leaves you little margin when heavy and clean.