I wear earplugs when flying the 767. When I leave the headset of and the other pilot is deaf (wich is often...) I ask them to turn the volume of the speaker down.
Sometimes it is so loud that it hurts on your ears, but wearing the headset again won't help. I just keep on asking or just turn the volume down on their comm panel.
Nothing you can do. Just keep on asking, or suggest to wear the headset.