I used to hate when some skippers will insist to keep the headset on and loudspeakers open at the same time. Like you say, it creates very annoying interference making the communication more difficult to undestand. I really don’t undestand why people will do that. If my colleague or I forgot to tune down our loudspeaker, we notice immediately and we will tune them down straight away. You are 2 in the cockpit with 2 headsets so you already have redundancy.
Personnaly if my FO agrees, I ask headset off as soon as pratical especially on the A321 which is a very quiet cockpit usually around
15/20000 feet and put it back on about the same altitude. It’s doable to do the flight without headset on the A320 series. Done it before. A bit noisy during climb on A319 tho. If some people think this cockpit is very noisy they should fly an Islander. xD