I would like to hear from the guys who are actually really experienced in the automation and control side about this.
There is a lot of talk about programming and sensors and databases and systems that learn, which are progressing by leaps and bounds. I have no doubt that it will be possible in a relatively short timeframe to do far more things quasi-autonomously than we do now. BUT:
In the following I am using the term sensor and system very loosely, e.g. sensor simply means a "problem detector" and "system" simply means some aspect of safe flight. So it could be a sensor for a hydraulic leak in the airframe "system" or "conflicting traffic" in the air environment "system", or just about anything else, we want to stay conceptual here.
For the sake of argument imagine we have this device we'll call a super-smart box (SSB), and accept that SSB design is such that it "doesn't make mistakes". All problems can be detected by a sensor and the SSB provides signals with correct output to deal with it, with greater reliability than a human can.
But correct SSB output is not the end product we are looking for. A change of trajectory of hundreds of tons of aircraft is what we are looking for, and SSB output needs to be converted to physical machinery activity.
My question is what happens in this chain of events.
Sensors detect problem in system A > SSB chooses correct response which demands action by physical system B > physical system B does not respond as expected by SSB. It may do nothing or may do something entirely different. "Something entirely different" could trigger other sensors in system A, B, C, D etc.... ad infinitum.
What is the process by which the SSB knows what to do now, and who is responsible for the correct outcome of that process?