Without the KMA20 circuit drawings in front of me I can't comment on the 40db input muting both radios however I would assume that it should be working as you have tried another KMA20 box.
With greater detail that you have given in your latest post I think it is time for a back to basics check of the wiring, this has to be a full check on each pin of the whole audio system (including the NAV audio). the system needs to be checked for each pin going to the correct place and more criticly going to the wrong places.
Some pins are not connected in aircraft instalations but have bench test functions within the box so all the pins that are installed in the conector block in the back of the rack will need to be checked even if they have apparently got no function. It is a long and time consuming task but at this point I think it is the best option for finding your problem