Boeing 737 Max Software Fixes Due to Lion Air Crash Delayed

[DaveReidUK]

Originally Posted by Dan Winterland

There's something fundamentally wrong here. The AoA sensors on the Max are failing. They are normally very reliable - I've never seen one fail in about 10,000 hours of flying aircraft with the same sensors. What you do notice while doing the walk around check is that they are fully deflected up if reverse thrust was used on the previous landing. The Max has engines moved far forward of the original position. Perhaps the sensors are being damaged by use of reverse thrust.

Interesting post.

I don't buy either the "bad bit" argument, nor the theory that the sensor was fitted wrongly oriented (presumably after the locating pegs had been sawn off!), but physical misalignment of the vane relative to the armature would be another explanation for the consistent AoA offset error.

If the latter is the case, I wouldn't rule out the possibility of the damage having been sustained before it was fitted to the accident aircraft.

[FCeng84]

Originally Posted by Dan Winterland

There's something fundamentally wrong here. The AoA sensors on the Max are failing. They are normally very reliable - I've never seen one fail in about 10,000 hours of flying aircraft with the same sensors. What you do notice while doing the walk around check is that they are fully deflected up if reverse thrust was used on the previous landing. The Max has engines moved far forward of the original position. Perhaps the sensors are being damaged by use of reverse thrust.

Let me offer a couple of thoughts about 737MAX AOA failure rates.

First, we know that the Lion Air accident airplane had an errant AOA sensor signal feeding MCAS for the accident flight and the flight the day earlier. We also know that this sensor had been swapped out just prior to these two flights so it appears to have been reading incorrectly the whole time it was fitted on that airplane. We don’t know yet what caused it to have the bias it exhibited. It is possible that the erroneous operation was the result of improper installation or the associated test/checkout. I don’t think we can yet say with certainty that this sensor had failed. Hopefully time and investigation efforts underway now will shed more light on this.

Second, the data for the Ethiopian accident has not yet been made public so we don’t yet know if errant AOA played a role there. Has anyone learn the history of that sensor on that airplane? Hopefully the data will tell us if it was reading wrong on the accident flight and how it behaved on recent previous flights.

Third, my conservative rough calculation of 737MAX fleet hours to date is half a million. Given that most individual airplane components have failure rates in the neighborhood of 1x10^5 hours one would have expected to see about 10 failed AOA across the MAX fleet by now. (Remember that there are two per airplane). I have not seen or heard anything to suggest a failure rate significantly higher than that.

The bottom line line here is that we don’t yet have clear indication of an abnormally high 737MAX AOA sensor failure rate. If that is an issue let’s see the supporting data.

[EDLB]

Originally Posted by fotoguzzi

(Not a pilot)Will they be forced to make a more "foolproof" 'plane?

Absolutely. Transport category airplane became more complex over time with reducing their fuel consumption. As more complex system automation goes in, as more complex the trouble shooting algorithm become. Within the few minutes left to ground if something goes south after take off there is no realistic time to go through all sorts of manuals. A good part that saved Sullys plane into the Hudson was, that the APU kept running. That kept the alpha floor protection in place. Most likely the APU was running to try restarting the engines but that was an extra which was not in the manuals.

How can you assume that two pilots figure a solution for a complex problem within minutes, while the aircraft designers did not come up with a fast diagnosis scratching their heads for years.

There are ways to produce complex highly reliable systems, but that’s not easy. They are in place in different industries and every “corner cutting” will make at some time the news. Fail safe systems with multiple redundancies and cross checking are possible with a low remaining failure rate. Havaries of those systems come out from either cutting corners in the design like Tesla’s camera only “autopilot”, or deliberately switching off safety systems like Chernobyl.

[FCeng84]

Originally Posted by fotoguzzi

(Not a pilot) THY1951 had a faulty radio altimeter antenna that gave a valid but incorrect value. That single input caused the autothrottle logic to throttle down the engines. Were there any subsequent changes in logic, or did Boeing just reiterate that crew should monitor airspeed?

Now, one duff sensor apparently affects all manner of instruments and systems, and it sounds as if Boeing hope that an indicator light and a conflict warning will be the fix, and an existing trim procedure will work in this new case if the worst happens.

I imagine similar examples exist for weight-on-wheels sensors, the reverse-thrust system, and many more that I cannot conceive.

If I am being fair in the above, will the aviation world allow Boeing to continue this apparent philosophy of simple inputs, simple logic, and an aeroplane arguably simple to fly in manual mode when some part of the above fails? Will they be forced to make a more "foolproof" 'plane?

Airplane system design is driven by a very rigorous process of aligning functional availability and integrity with the associated failure hazards effects. Key questions that receive careful attention are:

1. What are the failure modes of a system?

2. What is the complete set of consequences of each failure mode?

3. What is the hazard level associated with continued operation given the full set of consequences of each failure mode?

4. What is the probability of each failure mode and can it result from a single failure?

Depending on the hazard level from 3 above there are requirements on the probabilities from 4. In addition there is a hard requirement that no single failure regardless of probability can lead to a catastrophic result.

The simplest design that is compliant with the guidance above is usually considered the best solution. Keep in mind that when adding to an existing system the simplest design will be the one that involves the least change to the baseline system - this may not be the same as what would be considered simplest were one starting from scratch.

Where the process described above above leads to single threaded designs and supports continued use of existing single thread designs that approach will continue. To add unnecessary levels of redundancy would drive up cost without providing required benefit.

[Smooth Airperator]

I want to know on what planet is a single point of failure (such as MCAS using input from only one AoA source) acceptable? Where has the concept of redundancy gone to in the year 2019? I would love to read the justification for this.

[FCeng84]

Originally Posted by Smooth Airperator

I want to know on what planet is a single point of failure (such as MCAS using input from only one AoA source) acceptable? Where has the concept of redundancy gone to in the year 2019? I would love to read the justification for this.

A single point failure is acceptable if the consequences are bounded and the failure rate is sufficiently remote. As stated before a single failure with catastrophic results is never acceptable.

[Ian W]

Originally Posted by Smooth Airperator

I want to know on what planet is a single point of failure (such as MCAS using input from only one AoA source) acceptable? Where has the concept of redundancy gone to in the year 2019? I would love to read the justification for this.

If you take the flying aircraft as a system, there was not a single point of failure. Apart from AoA there was the flight crew that were expected to understand and cope with a Stab Trim that was repeatedly trimming down. There was an NNC for Trim Runaway also that would have prevented the accident.

It is obvious that the reliance on the crew (with all the other alarms demanding attention) to be able to switch the Stab Trim cut out switches to off was misplaced.

This raises all sorts of other issues. What happens if there is an intermittent fault caused by say a rat that got into the avionics bay and left an intermittent short, that had the same effect of max nose down trim for a few seconds every so many seconds? The Stab Trim switches are there for not only Stab Trim runaway - but I think that as far as most on this board are concerned they are Stab Trim runaway switches not for anything else - which is a real concern and a training issue. What other systems are only learned in relation to a simple to teach fault response rather than being told this is one of many potential examples?

All automation on the aircraft is built in the expectation that if it all goes to worms the flight crew will sort it out. In Boeing switch everything off (including the Stab Trim) and fly manually; in Airbus drop into alternate and eventually direct law and fly the aircraft. It seems that the limited and highly repetitive sim training and the FOQA studying beancounters are making it difficult for pilots to be that final backstop.

[gums]

Salute!

Some thots for the Boeing MCAS folks that might be lurking here.

As many have posted here, there may be a poor fault tree analysis of the MCAS itself, and then the downstream effects of various MCAS failure modes. Some food for thought....

- Some failure modes at the back of my brain... and there are more that can be combined with other subsystems to confound the pilot ( except for the Yeager crowd that posts here)

1) MCAS doesn't work at all, as with a hard failure of the sfwe module or computer board. If no other sfwe module infected, I would not notice unless I got a high AoA and stick forces got lighter than I was used to in the other half dozen variations of he plane. No warning light, but 99.9% of we pilots would never get to the triggering criteria, or maybe not even notice, right? So no need to confuse we pilots with knowledge of the system or what it is supposed to do. And of course, no warning light that MCAS is FUBAR
.
2) MCAS does not work as designed but the sfwe module and computer is operational ( i realize I am splitting the fault tree, but who said it would be easy?)

a) It does not provide the correct noise down trim at the design AoA and mach conditions, and no warning light because it thinks it is doing O.K. "I'm O.K., Dave, don't bug me!"

b) It does not act at design AoA and mach because the flap switch indicates flaps are extended. Same answer from Hal as in 2, a above.

c) It works when it is not supposed to. This did not happen with Lion, as the AoA was clearly in the MCAS activation criteria.

i) flap switch fails in flaps up position and MCAS triggers due to momentary high AoA upon rotation. Hmmmm... does MCAS see WoW? Does that sucker give ten seconds of trimbefore reset?

ii) AoA momentarily reaches the design value due to turbulence when turning to make an approach fix and you might be about to lower gear/flaps. Does that sucker command ten seconds of trim?

And so on and so forth. Those are just a few of the things some of we old cable and pulley guys think about that have later flown very computerized planes and needed to know everything we could to ensure the safety of our SLF and own skinny butts.

And BTW, after about 200 or 300 hours in the "classic" planes, USAF let me fly 4 more for another few thousand hours that had no direct connection to any control surface with any cables, pushrods or pulleys. So that explains some of my irritation with the folks wanting to "go manual" and asserting all would be well.

Gums ....

[fergusd]

Originally Posted by FCeng84

A single point failure is acceptable if the consequences are bounded and the failure rate is sufficiently remote. As stated before a single failure with catastrophic results is never acceptable.

The mitigation for this issue seems to be that the pilots will realise what is going on and switch the failed system off. This is the oldest trick in the safety case book . . . and regulators in _many_ safety related/critical industries allow the 'human will figure it out' mitigation to be used . . . and in many cases it doesn't because people are tired, bored, stressed, having a bad day, fallible . . .

This is a regulatory failure pure and simple. The safety case would make interesting reading. As would the one for the 'fix' . . .

[Machinbird]

Originally Posted by Dan Winterland

There's something fundamentally wrong here. The AoA sensors on the Max are failing. They are normally very reliable - I've never seen one fail in about 10,000 hours of flying aircraft with the same sensors. What you do notice while doing the walk around check is that they are fully deflected up if reverse thrust was used on the previous landing. The Max has engines moved far forward of the original position. Perhaps the sensors are being damaged by use of reverse thrust.

On the Indonesian accident traces, there were some pretty good indications that the nature of the real problem was some sort of sneak circuit in the port side SBYD box in the vicinity of the analog to digital conversion circuitry. Perhaps the real problem with the AOA indications is some sort of QA problem in the manufacture or installation of the SBYD boxes. If the Ethiopian AOA traces are indicative of a similar problem, then that is the first place I would be looking. The key Indonesian tell tale was noise in the AOA signal while taxiing when compared to the opposite side's indications.
These are very low time aircraft that are having AOA system failures after all.

[b1lanc]

ASRS reports - What else did Boeing not tell pilots

Thanks to James Fallows compilation here, it would seem there are some other items Boeing neglected too inform Max pilots about:

https://www.theatlantic.com/notes/20...37-max/584791/

1) This was the first flight on a Max for both pilots. Unfamiliarity with flight deck displays led to confusion about display annunciations and switch function. The Flight Manual does not address at least one annunciation, or the controls for the display—or if it does, neither pilot could find the explanation. I have spent literally days looking for an explanation, could not find one, and that is why I wrote this report. It shouldn't be this hard to figure out what I'm looking at.

On the First Officer side ND, on the ground only, there is a MAINT annunciation. We both saw it, couldn't find any immediate explanation for it on the ground, and didn't address it until airborne. I researched the FM (Flight Manual) for an explanation, accomplishing a word search of the term MAINT. There are only two references I could find: the overhead MAINT light (a no go item) and the CDS MAINT light (a QRH item). There is no explanation of the ND MAINT annunciation.

We spent the entire hour flight trying to find the meaning of this annunciation and came up empty handed. We determined to check it out once we landed (if the light came on again). Sure enough, after parking, the MAINT annunciation came back on the ND display. We called Maintenance to check out the light. We waited to make an ELB entry, unsure if one was required. Turned out, an ELB entry was not required.

The mechanic explained the light was part of a menu for maintenance use only on the ground.

In addition, there are two selector knobs that are under-explained (i.e., not explained) in the manual, and we were uncertain what their purpose was. One is under the Fuel Flow switch and the other under the MFD/ENG TFR display switch. These knobs don't seem to work in flight. The First Officer offered to hit the SEL function in flight, to test it out, but I thought something irreversible or undesirable might happen (not knowing what we were actually selecting), so we did not try it out in flight. The mechanic later explained SEL on the First Officer side was used on the ground by maintenance to toggle between the maintenance functions. I forgot to ask what my side did, and still don't know.

Finally, in the Captain's preflight procedure in the bulletin, it says, "Selector... C". What selector is this referring to? Is this the same selector under the Fuel Flow switch, (which is shown in the MAX panels on the L position, as if that is the normal position?) This is very poorly explained. I have no idea what switch the preflight is talking about, nor do I understand even now what this switch does.

I think this entire setup needs to be thoroughly explained to pilots. How can a Captain not know what switch is meant during a preflight setup? Poor training and even poorer documentation, that is how.

It is not reassuring when a light cannot be explained or understood by the pilots, even after referencing their flight manuals. It is especially concerning when every other MAINT annunciation means something bad. I envision some delayed departures as conscientious pilots try to resolve the meaning of the MAINT annunciation and which switches are referred to in the setup.

2) I had my first flight on the Max [to] ZZZ1. We found out we were scheduled to fly the aircraft on the way to the airport in the limo. We had a little time [to] review the essentials in the car. Otherwise we would have walked onto the plane cold.

My post flight evaluation is that we lacked the knowledge to operate the aircraft in all weather and aircraft states safely. The instrumentation is completely different - My scan was degraded, slow and labored having had no experience w/ the new ND (Navigation Display) and ADI (Attitude Director Indicator) presentations/format or functions (manipulation between the screens and systems pages were not provided in training materials. If they were, I had no recollection of that material).

We were unable to navigate to systems pages and lacked the knowledge of what systems information was available to us in the different phases of flight. Our weather radar competency was inadequate to safely navigate significant weather on that dark and stormy night. These are just a few issues that were not addressed in our training.

3) The MCAS function becomes active when the airplane Angle of Attack exceeds a threshold based on airspeed and altitude. Stabilizer incremental commands are limited to 2.5 degrees and are provided at a rate of 0.27 degrees per second. The magnitude of the stabilizer input is lower at high Mach number and greater at low Mach numbers. The function is reset once angle of attack falls below the Angle of Attack threshold or if manual stabilizer commands are provided by the flight crew. If the original elevated AOA condition persists, the MCAS function commands another incremental stabilizer nose down command according to current aircraft Mach number at actuation.

This description is not currently in the 737 Flight Manual Part 2, nor the Boeing FCOM, though it will be added to them soon. This communication highlights that an entire system is not described in our Flight Manual. This system is now the subject of an AD.

I think it is unconscionable that a manufacturer, the FAA, and the airlines would have pilots flying an airplane without adequately training, or even providing available resources and sufficient documentation to understand the highly complex systems that differentiate this aircraft from prior models. The fact that this airplane requires such jury rigging to fly is a red flag. Now we know the systems employed are error prone--even if the pilots aren't sure what those systems are, what redundancies are in place, and failure modes.

I am left to wonder: what else don't I know? The Flight Manual is inadequate and almost criminally insufficient. All airlines that operate the MAX must insist that Boeing incorporate ALL systems in their manuals.

[diclemeg]

I read article on MSN that Boeing in a few weeks will release their fix... I cannot link the article here, as I am new... but it doesn't fully seem like the right thing to do... here is the snippet

Among other changes, the revised software would rely on two “angle of attack” sensors, rather than one, to measure the upward or downward angle of the wings and nose in flight. If two sensors send data differing by five degrees or more, MCAS wouldn’t activate at all, according to the officials briefed

Shouldn't there be triple redundancy ? Ok.. Boeing won't pay for that.. next... According to the release, the pilots will have the ability to switch off MCAS now... how about also giving them the option to switch to the second AOA sensor ? If not, seems to me the next incident will be a stall, when the MCAS didn't activate due to one faulty sensor, and the pilots stalling due to the aerodynamic flaw.... I'm interested to hear what a 737 pilot thinks of my words here..... thanks...

[b1lanc]

Originally Posted by GH5

b1lanc post # 356 said " The MCAS function becomes active when the airplane Angle of Attack exceeds a threshold based on airspeed and altitude. Stabilizer incremental commands are limited to 2.5 degrees and are provided at a rate of 0.27 degrees per second."

Thats about 9.3 seconds for 2.5 degrees. How many turns of the trim wheel - probably about 25 ? if as previously posted 5 degrees takes about 50 turns?
25 turns in 10 seconds = about 150 rom !!
How long to electric retrim to ' zero" or where plane" started " re trim/elevator ?
how long if power cut and revert to manual ?

The manual effort required on the trim wheel is not insignficant. I don't see how any crew could recover at low altitude with aggressive manual re-trim if they don't stay on top of the MCAS trim increments as they occur.

[edmundronald]

Originally Posted by fotoguzzi

(Not a pilot) THY1951 had a faulty radio altimeter antenna that gave a valid but incorrect value. That single input caused the autothrottle logic to throttle down the engines. Were there any subsequent changes in logic, or did Boeing just reiterate that crew should monitor airspeed?

Now, one duff sensor apparently affects all manner of instruments and systems, and it sounds as if Boeing hope that an indicator light and a conflict warning will be the fix, and an existing trim procedure will work in this new case if the worst happens.

I imagine similar examples exist for weight-on-wheels sensors, the reverse-thrust system, and many more that I cannot conceive.

If I am being fair in the above, will the aviation world allow Boeing to continue this apparent philosophy of simple inputs, simple logic, and an aeroplane arguably simple to fly in manual mode when some part of the above fails? Will they be forced to make a more "foolproof" 'plane?

Every single NEW system on this type should be considered suspect until verified anew. Existing sensors and controls have been validated by real world testing.

Edmund

[Derfred]

Wow, I’d say those two ASRS reports say more about the pilots than the aircraft.

If you had those concerns prior to getting airborne, why would you get airborne?

[pax2908]

Originally Posted by edmundronald

Every single NEW system on this type should be considered suspect until verified anew. Existing sensors and controls have been validated by real world testing.

Edmund

Existing sensors validated, perhaps you mean "OK as part of existing systems" ? Or you mean "Sensors validated as having a failure rate low enough for them to be used as a single source of information" ? (Is it even possible to qualify a data source/sensor to this purpose ?)

[b1lanc]

Originally Posted by Derfred

Wow, I’d say those two ASRS reports say more about the pilots than the aircraft.

If you had those concerns prior to getting airborne, why would you get airborne?

Corporate/cultural pressure maybe? Question. Does AoA disagree generate a MAINT annunciation and set the light off on ground?
Here's another disturbing entry complements of same source as prior post:

It was day three of six for me and day three with very good FO (First Officer). Well rested, great rapport and above average Crew coordination. Knew we had a MAX. It was my leg, normal Ops Brief, plus I briefed our concerns with the MAX issues, bulletin, MCAS, stab trim cutout response etc. I mentioned I would engage autopilot sooner than usual (I generally hand fly to at least above 10,000 ft.) to remove the possible MCAS threat.

Weather was about 1000 OVC drizzle, temperature dropping and an occasional snow flake. I double checked with an additional personal walkaround just prior to push; a few drops of water on the aircraft but clean aircraft, no deice required. Strong crosswind and I asked Tug Driver to push a little more tail east so as not to have slow/hung start gusts 30+.

Wind and mechanical turbulence was noted. Careful engine warm times, normal flaps 5 takeoff in strong (appeared almost direct) crosswind. Departure was normal. Takeoff and climb in light to moderate turbulence. After flaps 1 to "up" and above clean "MASI up speed" with LNAV engaged I looked at and engaged A Autopilot. As I was returning to my PFD (Primary Flight Display) PM (Pilot Monitoring) called "DESCENDING" followed by almost an immediate: "DONT SINK DONT SINK!"

I immediately disconnected AP (Autopilot) (it WAS engaged as we got full horn etc.) and resumed climb. Now, I would generally assume it was my automation error, i.e., aircraft was trying to acquire a miss-commanded speed/no autothrottles, crossing restriction etc., but frankly neither of us could find an inappropriate setup error (not to say there wasn't one).

With the concerns with the MAX 8 nose down stuff, we both thought it appropriate to bring it to your attention. We discussed issue at length over the course of the return to ZZZ. Best guess from me is airspeed fluctuation due to mechanical shear/frontal passage that overwhelmed automation temporarily or something incorrectly setup in MCP (Mode Control Panel). PM's callout on "descending" was particularly quick and welcome as I was just coming back to my display after looking away. System and procedures coupled with CRM (Resource Management) trapped and mitigated issue.

[GordonR_Cape]

b1lanc

Does AoA disagree generate a MAINT annunciation and set the light off on ground?

AOA is not valid without forward airspeed, so it would require a special setup to test AOA disagree before takeoff.

[silverstrata]

Originally Posted by ImbracableCrunk

They should put longer landing gear on the 737 so the engines can be a bit farther back. And maybe change the nose so it's a bit quieter. Maybe put some cutting-edge 1970's avionics in it. How about a more pleasing brown color for the cockpit?

Another reason for all modern engines being a long way forward, is so that ‘fly-away’ blades do not go straight through the wing. (JT8D blades were much smaller, and the diameter if the engine was smaller too).
You might make a case for the turbine stages being under the wing, but you don’t want the N1 or compressor blades under the wing.

Silver

[b1lanc]

Originally Posted by GordonR_Cape

b1lanc


AOA is not valid without forward airspeed, so it would require a special setup to test AOA disagree before takeoff.

GordonR_Cape
Prior Lion flight had AoA disagree so I guess I'm asking if prior maint events are archived for annunciation during next pre-flight? Also, forward airspeed before rotation?