A few words on critical computer systems and automation from someone involved with regulated medical software, where a fatal outcome is an assumed possibility.
Systems and software are really not intelligent per se. In every system there is a perceived risk, designing complex inherently safe solutions is simply not possibly. That is why designs in general are approach with an obligatory risk assessment for every main functionality or mode.
The simplest formula to categorize risk is LIKELYHOOD x SEVERITY = CALCULATED RISK. Every single risk assessment then contains a mitigation or solution. It can be a required new feature or system, it can be training or in a perceived low risk simply acknowledge the risk, or try to lower the severity.
A long story short, icing of pitot tubes, pilot errors, system crashes, etc., etc. are (should be) all accounted for in design. No-one assumes to design an inherently safe machine, where a fatal outcome is even a remote possibility. The sad part: the only intelligence in design is learning from mistakes, testing, and yes, learning from real-life accidents.