There is an overarching software design & architecture requirement that any "catastrophic failure" -- a failure resulting in the loss of the airplane and deaths of its occupants -- must be "extremely improbable".
For FAR 25 aircraft, "extremely improbable" is defined as a failure rate of no more than 1 per billion flight hours (1E-9), established by a quantitative safety assessment.
However, as we found out with the Challenger shuttle disaster, this kind of quantitative assessment can be a bit pie in the sky. Still, critical software do tend to be extremely reliable. Just remember to reboot from time to time........