Ampclamp is on the right track
Your recorder is using an automatic level control or ALC. When there are no transmissions, the unit appears to be jacking the gain and you hear over boosted amplifier noise from the panel itself. See if the ALC mode can be switched off and you may find better results.
Master Tullamarine's suggestion of a resistor to attenuate the output of the panel also has a good chance of helping.
<<<---out from panel-----|
...................................Z
...................................Z<-----------to recording device>>>>>
...................................Z
<<<--------earth--------|--------------earth------------>>>>>>
the device represented by the | and Z's and <-- is a variable resistor.
1K ought to do it.