Who the devil goes around on speakers and headsets all day? That's pretty annoying, unless the headset volume is off in an ANR headset. Like someone else said, at the companies I've flown, pointing it out is sufficient to get the other person to turn off the speaker.
If somebody flew around with the speaker on all the time, I'd ask if he listens to music via his ipod headphone and speaker simultaneously.