Originally Posted by
Cazalet33
The power supply excuse doesn't quite sound credible for such a wide ranging series of failures.
They only way I can see it happening is as follows :
You're running x number of data centers each geographically remote of each and each mirroring the active Datacenter and each designed to perform a seamless horse swap.
On the active data centre DC1 you have kicked a batch job consisting of a volume of updates within the database subsystem. The power cut hits with uncommitted updates still in full flow.
DC2 takes over, continues, but because DC1 didn't have time to communicate a synch point, and as a result of network latency, DC2 resumes at the incorrect moment of processing. Now may have a corrupt database. Worse, subsequent batch jobs in the queue may use it as input, spreading the damage. Even worse, this corruption has been faithfully reproduced on the other data centers, so you now have no hot swap capability.