Good question!
I would advise against switching AP’s for temporary PF/PM switching on the A320/A330.
It can create quite an unexpected jerk beyond of what the AP would normally command.
The reason for this is the presence of a minor FD mismatch due to onside Air Data / IR data not being 100% synchronised.
This is noticeable in cruise when ADR 1 and ADR 2 altitude are slightly different. (But well within RVSM tolerances)
Also the FD mismatch occurs frequently in managed descent where the vertical profile differs between Capt and F/O side, this is directly caused by differing onside IR data (wind, GS etc).
For prolonged taskswitching it would be advisable to switch AP when possible because this allows you to monitor the AP performance. I.e. is the FD being followed or not.
hope this helps