Well people there is your answer.
Thank you Titan. Did you respond to smack me down or answer the OP? The information I posted is a mix of my own experience, experience of other expats and reading of various Australian Govt websites.
I found the ATO were always very vague and would tell me they didn't know "my circumstances". My accountant would always say "just to be safe do this" which infuriated me.
One thing I do know. If the ATO have decided you owe them, it will be an uphill battle to prove that you don't.