I was thinking of bringing this subject up a while back, then the B757/Tu154 mid-air happened and as it may have been a factor I decided against.
Personally I made sure that at least one person had a head set on all the time and both of us for T/O. climb and descent.
I have sat on the jump seat many times when both pilots have had their head sets off and were relying on speakers and have been alarmed at the number of calls from ATC that were missed the first time and had to be repeated, often causing delays in whatever action ATC required.
It is my view that speakers and hand held micrphones are possibly OK for only one member of the crew, with at least one member on headset all the time.