I never notice this interference unless the speaker is wide open. My conclusion is that certain individuals must have some very sensitive auditory canals. Some FO's complain even when the volume is less than half way open. Fine, whatever rocks your boat, I'll turn it down, but it's odd!
The problem arises when both pilots take off their headsets and forget to open the speaker volume. Where is your redundancy then?