In terms of the access you have it depends on the requirements of your job.
As you say there is a colour system. Working for a retail outlet you would have an airside pass that gives you access to the departure lounge and gates only but not on the ramp, ie 'Internal Airside'. Even staff who have 'External Airside' access don't have access to all areas for example baggage buildup areas, departure gates, cargo areas, taxiways etc.
The Control Tower certainly isn't somewhere you have automatic access to just because you hold an ID, it would need to be required for your job.