The Constitution of Australia lists certain things over which the Commonwealth has powers - and since Aviation did not exist when the Constitution was adopted in 1900, it is not on the list.
Originally, the Royal Aero Club movement issued licences which really had no legal validity. Class A or Private Licences or Class B Airservice licences !
Then in the thirties, the military decided they would run aviation, civil and military and took over issuing licences.
Come the middle of the '30's one skilled pilot named Goya Henry flew under the nice new Sydney Harbour Bridge which slightly annoyed the then so called Director of Civil Aviation, one Colonel Blimp...... no sorry, one Colonel Brinsmead, who promptly informed Mr Henry that his licence was cancelled.
Mr Henry no doubt responded in the gentlemanly manner of the times but he went on flying and simply said he did not need a licence - and the High Court agreed.
Then a bit of a World War came along and nobody did much Private Flying for a while.
In 1944, the Commonwealth of Australia signed on to the Chicago treaty that launched ICAO and that was in exercise of the foreign affairs power which was a COmmonwealth Power but it did not really enable the CofA to regulate all aviation.
I suggest you read Professor Ron Bartsch's works on Aviation Law in Australia for a more accurate and educated dissertation on the subject but as an aside, to this day you can see the history in the "Applicability" statements of our Civil Aviation Regulations, so carefully spelled out instead of simply "all aviation".