KP you are spot on. Problem is CASA people are reluctant to go out in the summer heat up north or the winter ice down south to do what you suggest. Also, as you say, by giving notice of their rare field visits they give everyone time to tidy up the immediate mess.
But CASA inspectors need to dig deeper than mere external visual inspections of hangars, aircraft and operations. Digging deep is something they appear ill-equipped to do, instead focussing on the state of manuals and paperwork which they know will be an easy target in your typical smaller organisation.