For those who are wearing ANR it would make a lot of sense to put back on their headset, the Telex wearers among us may not get its benefit though.
In light of the amount of crew wearing ANR I think it would only make sense not to use the speakers.
Also it can be extremely noisy and flight deck compartment speaker may not be heard as well as a headset speaker located a few cm's from your ear.