Always keep mine on, I have a lightweight headset that you can hardly feel and I find cockpit speakers very intrusive.
The Air Navigation Order prohibits the crew of UK registered public transport aircraft using hand mics. below 15,000ft in controlled airspace and during take-off and landing.