Vpilot53, there are too many variables to give you an individual answer. Your pay, from whichever contract agency, is gross and it is entirely up to you to sort out your tax in your tax jurisdiction. Since you have paid no tax in Japan then no dual taxation applies and, so, your gross pay will be subject to tax......if you meet the resident criteria. The first thing to do is visit the ATO website and see if you do fall under the rules that govern residency. The second thing to do is call or write to a bunch of overseas tax specialists and see what is the best deal that can be provided. Some seem more liberal and 'educated' than others in which rules can and cannot be applied.
You won't get a clear-cut answer because we are all individuals but I think most have ensured they take full advantage of any ATO overseas allowances.