You can switch them off whenever you like!
However, there will be at least 2 issues.
1. You won't get any fire warning in the event of a fire post shutdown.
2. The APU inlet door won't close.
Good practice (and the AMM) dictate a 2 minute wait between actual APU shutdown and switching the batteries off.