To put the ASIC requirement a little more simply. In order to have unescorted access at a security controlled airport, a person must display a valid ASIC. If you only have an AVID, the only option available to you to get to or from your aircraft would be to have a Visitor Identification Card (VIC) and be escorted by an ASIC holder.
The main thing to remember is that the ASIC is NOT an authority to be in the secure area of a security controlled airport. The granting of that authority rests with the airport operator. Suffice to say in most circumstances, airport staff wouldn't generally stop you accessing the required airside areas.
I will say it one more time: For an Australian license holder, you have to have an AVID and/or an ASIC, but you do not need an ASIC, except in the security controlled areas of an airport/airfield with a security program.
In my experience, there are very few, if any, security controlled airports around that have areas within the airside boundary where ASICS are not required. The obvious ones being AF, BK, PF and JT which are exempted under the regs due to no RPT services being conducted to or from those airports.