Correct. I think you'll find that the ASIC card is ONLY VALID whilst engaged in your employment/vocation. Even then you must have a valid reason for being in secure areas. For eg: Looking at a visiting a/c on the ramp is not a part of your job and you will get your ass kicked!!
ie, been there, done that!!
I can also safely tell you that arriving from an international flight in full uniform displaying the appropriate ID Card doesn't even get you easy passage through Customs/Immigration. Sometimes the opposite actually.