Yes, the fan blades do rock gently from side to side in their mountings as they lazily windmill round (so not under any load to keep them still). The loudest contact may be between the mid-span shrouds - sometimes called the clappers - of adjacent blades, as they hit one another. This makes a loud tinkling sound.
Re the A320 family, if memory serves, the CFM-56 engines have mid-span shrouds on their fan blades, but the IAE V2500 engines do not.