I'm not much for conspiracy theories but whatever root cause the simultaneous "unrelated" Canadian outage has, I'll be interested to hear. I mean, that's a heck of a coincidence.
Originally Posted by
Maninthebar
A corrupted database is apparently to blame. Though this raises questions as to
1. How the database became corrupted
2.What mechanisms are in pace to detect corruption
3.Whether any shadow copy is in place (or should be) for failover in such a circumstance
Wellllp, whenever one person making a mistake in a (software) deployment can cause a system to fail, it will fail sooner or later.
Decades of software engineering experience informing my opinion here, including multiple incidents creating regional/national/worldwide outages evolving right in front of me.
"Two person rule" is helpful in reducing the risk in deployments to production, but there need to be more "Swiss cheese" measures in place to reduce the likelihood that a single mistake will make it all the way to production.