I have encountered that several times in the 320, and after consulting engineering department, they came to the conclusion that it was cause by a misalignment of the rudder trim, which itself was caused by aligning the IRS while having flaps 1+F. They issued the recommendation to always align the IRS with Flaps 0.
Whenever that comes up during climb I disconnect the autopilot and 'tap" the rudder trim to the left for a brief moment, without applying any input to the sidestick. That causes the airplane to slightly move sideways, and the vibration to dissapear completely.