Originally Posted by
Nialler
DC2 takes over, continues, but because DC1 didn't have time to communicate a synch point, and as a result of network latency, DC2 resumes at the incorrect moment of processing. Now may have a corrupt database. Worse, subsequent batch jobs in the queue may use it as input, spreading the damage. Even worse, this corruption has been faithfully reproduced on the other data centers, so you now have no hot swap capability.
Easy, you mirror the uncommitted database queue as much as is feasible.
One changeover you have a procedure to immediately process that queue, and if a few bits get missed then you deal with it in the field and accept that loss.
If you have to restart procedures for certain events (passenger boarding for instance) ... then these are isolated and expedited. Meaning 5 plane loads of passengers are disembarked and reembarked.
In other words every part of the procedure is planned for.
Nothing is out of process.
And practiced, a failure is just another day at work.
If airlines do not do this, they are completely irresponsible.
This is normal.