I've had similar vibration problems inflight with a number of older A320's.
It was bad enough to spill drinks in the aft galley.
Vibration could be eliminated by disconnecting the A/P and setting rudder trim to zero. When A/P was on the rudder would trim and vibration would return.
Source was the rudder actuators which wear with age allowing a small amount of play causing the vibration.
Even after the rudder was identified as the source of the problem the Engineering department still wanted vibration reporting forms filled out
Took several months to fix.