It must be the other pilot’s speaker that causes this effect. The difference in time delay between your headset earphones and your onside speaker is presumably too short to bother you, hence we often don’t notice if we have left our own speaker on. But the time delay from the opposite speaker is perhaps long enough to cause noticeable audio effects to the other pilot, depending I guess, on a combination of headset type and relative volume levels.
Note, I am talking Airbus FBW here. Relatively wide cockpit with the speakers next to the sidewalls, so furthest away from the other pilot. Other types, such as 737, have different speaker locations.
(Delayed audio added to the original, produces all sorts of effects, ranging from increased ambience, to echoes, to frequency cancellation, (e.g. comb filtering), depending on the duration of the time delay. Indeed, this is the method by which some electric guitar and recording studio effects are produced).