Hardly ever take mine off (DC H10-80), except to leave the aircraft, as too many years of no headset flying have caused hearing loss such that with airflow noises I miss too much radio traffic without it.
I'm not aware of any legislation mandating the use of headsets above/below a certain altitude/flight level.
Kingof theRoad what country are you referring to?