Well, they can't take away your hours of total flight time. Unless they invent time travel and retroactively sabotage your aircraft, or something like that.
The only thing CASA can do is say that the hours you've flown so far don't qualify to fulfill the experience requirements for a CASA PPL. In which case you would have to do the 40 hours all over again, but somehow under the oversight of CASA.