As an ASIC holding pilot my reason for being airside is largely any reason I deem legitimate and justifiable.
It's not actually, it's the airport operator who is required by law to apply and enforce access control, so it is up to them to decide if your reason is legitimate or justifiable. If they don't consider it either, you'll be the one explaining yourself to the higher authorities.
Airports operate under rules and regs just as much as pilots do. Just like you, they may not want to, but they have to. That is a point that is almost always misunderstood by some pilots, who seem to think that they get to pick and choose which rules and regs apply to them.