Need to use your backups, not just test them
One lesson I think I learned in another industry: if you want your backups and protections to work when they are needed, you have to actually integrate usage of them into your standard operations. No amount of "really really careful" testing is an adequate substitute.
True story: the factory site at which I worked which at times was probably on the worldwide top ten list for dollar value added across all industries, was so concerned about the single point failure of losing utility power (yes, we had lots of stuff on UPS, but some big stuff of interest was not) that they paid to have several miles of high-voltage connection made to a second point within the utility network. There was a nifty switch on our premises which at need would transfer our load from the one string of power towers to the other.
Came the day we needed the backup connection to work--not because of a failure of the utility, but because a forklift operator on our own premises accidentally damaged a very late connection line by swinging a load up into it.
The post-mortem established that the nifty transfer switch had a battery which needed to be alive for the transfer to happen. And there was no maintenance plan for looking after the battery, which had probably been dead for some time by the day of our need. That one cost a very, very, large amount of money.
Yes, a suitable test would have caught that one, but I'll still hold out for the higher standard of usage. That way people take it seriously, and people notice and fix the troubles.