OK Cec, I'll bite!
The big picture may have been different to what thought you could or could not hear. Nowra may have had Jindaviks flying around, who knows? Amberley may have had no-com aircraft buzzing about. And WLM approach may not have had anyone you could hear on freq, but maybe the aircraft were with the tower on final for an instrument approach? It is Approach's job to separate and sequence IFR arrivals from other traffic (usually). That does not mean they are on Approach's frequency. The circuit area, generally, goes out to 5NM but the aircraft will call TWR at about 10NM. Or they could have been on no-retransmit UHF operations! Who knows?
Point is, just as we can't tell what is exactly going on in the cockpit, sometime aircrew don't have the big picture either.
As a professional ATC, I can say without a doubt that no one is ever held without a reason. If you were genuinely concerned, ask for a phone number and ring ATC. But instead of asking 'Why wasn't I cleared through immediately?' start off with 'Not sure what the hold up was, but would there have been any way in which I could have assisted in getting a clearance more quicckly?'
Cheers,
NFR.