It seems to me that if a passenger aircraft is flying on established routes and at established flight levels with its transponder deactivated, provided it does not trespass into secure airspace, it will be ignored by military radar and invisible to ATC. This is rather worrying if I am correct.
You are wrong if you are assuming military have only primary and civil have only secondary and they never speak to each other.
Military will almost always have both. They may not all have the latest Mode S & ADS-B, but certainly Mode A/C plus military IFF modes. So they will see any civil transponder that is switched on. If an aircraft on a recognised civil route is not transponding they will notice. Indeed it is one the key incursion methods they should be looking for.
Civil controller will often have both also.
And in many states it is not uncommon for mil and civil controllers to be sitting in the same room. Or even for mil personnel to be offering the civl ATC service.