The way I read it
You have to seperate annual leave and any other leave.
Annual leave is not counted- you can spend that anywhere, including Aus.
The tax dept is saying that other days- days off, days in lieu etc.- are generated from the foreign service, and therefor if you spend them in Aus, they are not foreign service, and therefor costitute a break in the foreign service. The ATO has picked an arbritrary figure- 60 days- and said if you have any more than that in Aus (that is not annual recreation leave), pay us.
I may be wrong, but it seems plane to me. Very sad.