Originally Posted by
Ian W
In a machine room from my past, that no longer exists, a security guard groped for the light switch and powered down the research data center using the emergency switch despite its switch cover - rather obviously the lights did not come on although several smaller ones went off.
Exactly same thing happened in my old company. Fortunately the room was only a small part of overall ops and was backed up.
Another time a maintainer tried to shut down one poorly performing widget using the maintenance screen; his mouse pointer slipped down one line in the menu from “shut down this widget” to “shut down all widgets” and he did not spot it, with serious results for the many operators using the widgets. In the aviation no blame culture it’s not the fault of the maintainer or security guard, it’s poor design or procedures. Proper analysis and testing in the design and build phase should find this.
I am surprised to hear here that servers can be so sensitive to mains problems. I guess the problem is that that their designers assume they are always fed from a nice clean UPS supply and, if the UPS fails to filter the surges, brownouts and spikes, the server has no designed in protection against them. The UPS manufacturers will probably claim it can never happen.