Is it really so unreasonable to restrict access to military installations to EX-servicemen? Timothy McVeigh and Lee Harvey Oswald were both ex-servicemen.
If you feel hard done-by, try using your pass to go airside at any civilian airport as an ex-employee and see how far you get.