Yes. The cockpit of A32o is so quiet, the opposite side speaker is clearly distinguishable even if your own headset is on with ANR active.
The only time (a couple of) when I needed to lean over without a headset to hear the colleague was if he puts the boom on, closes his speaker completely, turns up his INT ACP all the way up and whispers silently into the mic stuck under his nose. An unhelpful habit that was.
Could not imagine wearing something on top of my skull on a 10 hour day. but then again I enjoy A/THR on to keep the nights easy, so not really a pilot at all.