Maintenance Lapse Identified as Initial Problem Leading to Lion Air Crash
 

Maintenance Lapse Identified as Initial Problem Leading to Lion Air Crash

Crash investigators have concluded preliminarily that improper calibration of an airspeed (AoA?) sensor during maintenance touched off the sequence of events that led to October’s fatal Lion Air jetliner crash in Indonesia, according to people familiar with the details.

The conclusion is subject to further analysis, these people said, but it is the firmest indication so far that a suspected maintenance lapse was the initial misstep that ended with the months-old Boeing Co. 737 MAX aircraft plunging into the Java Sea, killing all 189 people on board.

Two days before Lion Air Flight 610’s early morning takeoff from Jakarta on Oct. 29, according to these people, mechanics installed but failed to properly calibrate a replacement airspeed sensor called an angle-of-attack indicator.

Based on information downloaded from the flight-data recorder, last month’s interim report revealed a constant 20-degree difference between signals from the angle-of attack sensor on the captain’s side—which had been replaced—and those from the co-pilot’s-side sensor.

Until now, the precise cause of the improper signals from the captain’s-side sensor hasn’t been clear.

People closely tracking the probe said that after U.S. air-safety experts re-enacted the tasks of installing, calibrating and verifying operation of the sensor, they deemed current maintenance instructions appropriate.

https://www.wsj.com/articles/mainten...39204?mod=e2tw

I, for one would deem the current procedure NOT appropriate , considering the result!

Now , I am sure they can pull out the old card : It hardly ever happens that someone gets it wrong!
Well that was OK until the Max as the misaligned AoA sensor was just a faulty indicator and did not lead to an uncontrollable aircraft via false trim input ( only needing one crew error to open up the last hole in the cheese.)

I am looking forward to see what was actually done, and it would be nice to know if this has been done before and if an aircraft has gotten airborne with same error.

Not sure why AoA display is an option on the 737 MAX if a critical flight control surface is controlled by a (single) AoA sensor.
Maybe Boeing should update the QRH for BOTH Runaway Stabilizer AND AoA Disagree?
Seems like Boeing rushed this "PATCH" and didn't think this through completely.
In Boeing's defense it took some poor maintenance and sketchy aviating for this issue to manifest.

How many turns of the screw?
 

FWIW seems to be confusion in the above posts between AOA and Airspeed. AFIK the AOA ( Angle of attack ) indicator measures the angle of the plane to the mass of air flowing past the plane- and has little to do with airspeed ( the indicator FUNCTION is independent of airspeed) . Whil it is true that for stable flight, the PROPER angle of attack is a function of airspeed and weight and CG and . . . the measurement device is like a windvane whose angle is RELATIVE to the body-airframe .

Sure sounds like somehow, the AOA indicator was installed in the wrong set of holes ( clocked ) relative to the correct set. " If it doesn't FIT - FORCE IT ' assmbly method :mad:

According to the Preliminary Report, the AoA values from the suspect sensor were 20 degrees adrift.

If you tried to fit the sensor offset by 20 degrees, none of the attachment screws would pick up on the captive nuts, so that sounds highly unlikely.

BluSd, they are saying there is nothing wrong with the existing procedure - IF IT'S FOLLOWED!!!

The best AMM procedure in the world is worthless if it isn't followed.

DR - I'll be very curious to find out how they messed up the installation, but Murphy's Law says that people can be extraordinarily inventive when it comes to messing things up. I remember us getting back parts that had indexing keys to prevent improper installation where the indexing keys had been very professionally machined off... :ugh:

A lot of FBW aircraft have much more flight control surface movements (including completely limiting pilot authority) based on AoA without flight deck indication of the AoA and seem to be flying around safely. Also, it would be very hard to design an aircraft that will be still very safe with "some poor maintenance and sketchy aviating".

Hopefully they find the CVR and we see in final report what really happened.

Looking from another angle (no pun intended) a question for the Engineers/Mechanics out there.... Is the AOA sensor calibrated at the factory BEFORE the operator receives it into his spares stock inventory so as to enable a quick sensor change out on the line. For example, disconnect cannon plug, remove old AOA sensor / Install new AOA sensor, reconnect cannon plug, do test (bite check?) and on your way OR install new AOA sensor and then carry out calibration of said sensor.

Rgds McHale.

The sensor is mounted on the fuselage; so in the case of a sensor that can be mounted on either side, perhaps needs to calibrated to the angle of incidence between local flow at the fuselage location and wing which will be positive or negative. However being 20° off implies a 10° difference between local flow and AoA:confused:

That kind of difference implies a misalignment when mounting.

Are the sensors generic units with various calibration values for different aircraft and sides?

tdracer
I am all with You on that.
I am wondering how many times , if any, an 737 AOA unit has been installed wrong to this extent, and were it was found out, ie on ground or airborne?
If never, then procedure is good.
The fact that a miss-installed AOA on my 737- 800 is not doing much more then making a lot of rattle and making me work hard for some minutes on the QRH as opposed to the MAX that will demand me to take prompt action on that semi- runaway trim , if not A/c becomes uncontrollable as so tragically demonstrated, SHOULD demand of the Maint Procedure even more rigid adherence and double checking!

I wager the procedure was the same, even with the potential for dramatically worse outcome on a different aircraft.
I have seen this on the Dornier 328 Turboprop when they certified the D328 Jet: Same airframe, lots of cut and paste, small details not applicable ( Propheat ciquitbraker being the nobrainer in QRH)

Anyway
I am looking forward to fly the MAX next year.

Yes.You just turn it inside out.

On the 737NG you just have to swap the sensor and thats it job done. It is recommended that you you do the calibration but not mandatory.

The NY Times has some fancy new graphic images and a step by step narrative this morning.
It doesn't seem to add anything to the conversation at hand.
https://www.nytimes.com/interactive/...ml?mtrref=t.co

What Ametek says about it: "• Port and Starboard AOA Transducers Are Interchangeable"

FWIW I found the following online regarding MCAS:
"This is from a maintenance training manual for technicians/mechanics.
Pilots do not see this. Maintenance do a two week difference course, pilots get a handout."

https://cimg4.ibsrv.net/gimg/pprune....fcebabedfa.jpg

What the hell does that mean? Could that have anything to do with the apparent ineffectiveness of the ANU trim commands after control was handed over to the FO?

Separately, with regards to focus on the calibration (or alleged lack thereof) of the replaced left AOA sensor, that aircraft had a history of left-side invalid air data and AOA data problems that preceded the fitting of the replacement sensor. My money has been on a developing problem with the left ADIRU; a Sunwing MAX 8 delivered 6 weeks before PK-LQP had a left ADIRU problem two weeks after the JT610 crash.

Hmmmm- normally the transducer is the device that TRANSforms motion- movement or pressure/vacuum into a electrical signal . But the ' windvane' or aoa ' vane' and mounting brackets would have to be reversible when mounting on opposite sides of aircraft. left side would be O---! versus !---O for right side ( O being transducer mounted on --- axle and ! being vane ( up being forward )

I seem to remember that the position (port or stbd) is indicated by pin programming.

BAengineer,

Many thanks, you are indeed correct.

Upon a bit of further investigation I have been led to believe that the AoA sensors are calibrated at the factory and are interchangeable between L/R and attach directly onto the fuselage with screws and can't be misaligned. The respective sensor vane has a resolver/s attached to it (internally) and sends an electrical signal to the SMYD's (SMYD1 for L(Capt) and SMYD2 for R(F/O)) which alerts the respective SMYD as to what position (L/R) the AoA sensor is in (vane has to move through 180deg when swapped from L to R or R to L positions). The respective SMYD's then recalibrate vane angle and transmit to Capt/FO screens.

There is a ground test function on the SMYD's for AoA sensor test/chk, but as you mentioned NOT mandatory upon installation which I find rather odd. This info pertains to the B738 and I would assume (very dangerous in aviation) that the MAX would be similar if not the same.

Rgds McHale.

I don't know what became of this NPRM but it would seem that a test should be mandatory: https://www.regulations.gov/document...2012-1041-0001

That is only relevant to 737 classics - not the NG or MAX.

I assume that the reason of the check not being mandatory is that there is no calibration adjustment possible on the aircraft - it either passes or it doesn't. If it doesn't then you replace the sensor again. So if you have confidence that the sensor is correctly calibrated from the shop then I can see why checking it again would not be mandatory.

Think you'll find that 737-700/800 = NG, the 737 -7/-8 (without the zeros) = MAX (and -9/-10 also).

I have MM pages for the NG and MCAS isn't there (there is an anti-stall function in NG STS, but very different characteristics to MCAS).

JIMTX in post # 22 said ...Which to this SLF suggests that some sort of detent must be built in to assure exactly 180 rotation. Could it be that the detent is set allow a plus- minus 20 degree movement of AOA VANE ? And thus if not twisted past the detent, a relatively fixed error may be possible ?

Smoking Gun?
 

Peter Lemme's Blog on the JT610 accident contains a lot of good information on the 737NG AOA systems which I am going to draw on. AOA Failure Modes
In this blog, he basically shows that any interference with the signal coming directly from the AOA sensor can be expected to create an error that varies with the indicated angle. By overlaying the left and right AOA signals, he shows that the 22 degree error between the sensors does not vary significantly until the aircraft is on the ground. Therefore we should really look elsewhere in the aircraft for the problem. Due to cross wind effects on the ground, you can understand that there might be some variation while on the ground, but there is one other difference visible in the trace while the aircraft is riding over the bumps in the taxiways, the electrical noise level!
The trace from the left AOA forms a fat line with numerous spikes and the trace from the right AOA is smooth. This is the smoking gun that gives significant clues as to what may have happened to cause the problem.
https://cimg8.ibsrv.net/gimg/pprune....4c4865f0b3.png

In the course of a rather long and varied life, I have had occasion to troubleshoot misbehaving electrical circuits that carry signals, and when I see a circuit responding to vibration, I suspect either a "sneak circuit" or an intermittent circuit. In the case of JT610, other data suggests we are dealing with a sneak circuit, but when you get to the circuit board level where data is manipulated, it could also be an intermittent condition. Typically what you do to troubleshoot these problems is to put your meter/scope into the energized circuit and shake things until you see a response. In this case, we see the response, but we do not know just what got shaken. Instead, we have to infer what might be the problem by looking at the system and where it gets its signals and do a bit of mental exercise.

https://cimg2.ibsrv.net/gimg/pprune....d757305251.png
In the top left corner of the Stall Warning System-Stall Management Yaw Damper diagram, we see the signals coming in directly from the AOA resolver.
They do not go through any box to get there. If there were some sort of cross connection in those lines we would see data that varies differently from what we are observing (constant error at all angles while inflight).
This would indicate that the problem is downstream from the wires running to the SMYD. Since Boeing went to a high level of attention to ensure that there is a high level of commonality with prior 737 aircraft, and an ability to interchange between right and left AOA sensors, we should assume that it would be very difficult to mis-install one of these sensors (relatively fool proof). So lets focus our attention down stream on the SMYD box.
I do not have the schematic on this item, so I'm left to improvise by learning what I can about the topic of converting the Sine and Cosine signals inside the SMYD box. One thing I learned is that the signals likely receive conditioning by various circuit elements (amplification and filtering of noise) before being combined into a signal representing an angle. Background-signal conditioning-resolver to digital converter
There is therefore an area inside the SMYD box that can foul up these angle signals. And what does the SMYD box do? Among other things, it activates the stick shaker!

Now what else do we know about this aircraft? Why it was nearly new! You might even say, it was still on its "shakedown" period. Supposing a loose nut, a blob of solder, or a wire fragment was floating around inside the SMYD box and eventually found the place where it could do the most damage.
Someone at Boeing who knows this system will have to look at the contents of SMYD box to have a chance at finding what failures might behave the way we have seen in the JT610 AOA data, I don't have enough info.

I think the AOA probe change is likely a red herring that can easily lead this accident investigation astray. The core problem is that the system was changed in the MAX, and with it, certain systems became critical. It is true that maintenance did not fix the problem, but given the procedures they were given, would they ever have found the problem without resorting to shotgun type trouble shooting? Does the troubleshooting ever lead to a changeout of the SMYD box?

Salute!

Thanks, 'bird, I am almost "all in" withya on this line of inquiry/reasoning. I.e. not necessarily a poorly installed or faulty sensor, but something in that "box". And recall we had a plane going down in that area a few years back due to a bad solder joint on a card.

I also note the noisy data that was on the flight data recorder which you pointed out over a month ago. Almost like there is a bad connection or poor solder joint or ....... I assume the FDR got its AoA plot downstream of the "box" and not directly from the actual vane, but your diagram doesn't show where else AoA data goes from the sensor. I also doubt that the FDR decodes the synchro/resolver analog signals, huh?

Gums sends...

Gums,
Just thinking like a systems engineer, without concrete knowledge of the 738 systems, but I think you will find that the AOA data is digitized inside the SMYD box and sent through a data bus to the FDR and ADIRU and wherever else needed in the aircraft.

When the first NATO E-3A aircraft were delivered a couple of them had recurring AOA problems, left and right indicators consistently had the same disparity. After much head scratching it was found that during manufacture, the template for drilling the holes for securing the mounting rings for the interchangeable transducers had not be turned over when going from the left to the right of the nose, with the results we found. The men from Boeing came with their template, drilled the holes in the right place and all was good.

But this was 35 years ago.

CONSO,

The AoA sensor vane is aerodynamically shaped and can swing through 360deg without a problem (on the ground) and you often see the vanes on a parked aircraft drooping vertically or with a good breeze blowing, 180deg in reverse and anywhere in between. This is NOT a problem and is merely the nature of the beast. The SMYD computers ignore this because the aircraft is in "ground mode" signalled through squat switches, no engines running, park brake on, airspeed <30kts etc, etc, etc.

When the aircraft is starting to accelerate down the runway, the airflow around the nose of the aircraft will reposition the vanes to the required position. I don't know the exact requirements/signals for the SMYD's to start paying attention to AoA vane position/angle (transmitted by internal resolvers/synchro's) but it could be when (don't quote me) BOTH engines are started and >80% N1, park brake off, all cabin doors closed, airspeed >30kts etc, etc, etc. Hope this helps.

Rgds McHale.

Thanks - theat seems to leave ( per the 4 previous posts ) some sort of electrical-issue since if installed with wrong tgemplate- or force fit, the problem would have been happening from day one.

But IMO that does NOT excuse Boeing from allowing a single point failure (AOA system ) from being able to fubar a major flight control system- with NO mention of the MCAS.

thus the ' genie unzipping and urinating on the pillars of science ' comment by Ernie Gann "

Thanks - theat seems to leave ( per the 4 previous posts ) some sort of electrical-issue since if installed with wrong tgemplate- or force fit, the problem would have been happening from day one.

But IMO that does NOT excuse Boeing from allowing a single point failure (AOA system ) from being able to fubar a major flight control system- with NO mention of the MCAS.

thus the ' genie unzipping and urinating on the pillars of science ' comment by Ernie Gann "

The problem at first was an intermittent AoA signal. That meant no airspeed and altitude for the captain sometimes because AoA is included in the calculation.
After the change of the AoA sensor in Denpasar the signal was not intermittent but offset by 20 degrees on the previous and the accident flight.

So it could still very well be either that the sensor itself was faulty and not properly checked at the factory or the installation was done "creatively".

Salute!

Thanks, Cpt McHale. Guess those puppies move about a lot, although the right vane seems to be more highly dampened.

I had noticed a change in the left AoA a minute or two before T/O roll and figured it was some kinda configuration change. Upon looking at the heading trace, the vane prolly changed when the jet "jinked" on the taxiway or ramp when heading to the rwy. Then the thing apparently moves again as the plane lines up. From there we see a near constant bias from the other AoA vane. And so...

@ 'bird and others with lots more maintenance experience that I ------ If we assume that the FDR does not get the raw analog signals from the AoA vanes that we see on the data, but from a "box" or module in a "box", then I wonder if there is a dip switch change or "plug" that must be inserted/removed when the AoA vane is replaced. This would tell the "box" using the raw analog signals to bias the digital data according to "right" AoA or "left" AoA mounting.. In other words, the "universal" vanes can be placed on either side without having different mounting hole spacing. I like that idea for $$$, standardization of parts, and so forth. But it is a procedure that must be followed and then verified
.
Jez wondering...

Gums

B737 MAX AOA sensors have an endstop at 100° each swing direction.

According to the CONSO/Lemme diagram there are PINS on the SMYD which I assume could be used to "orient" the device.
Lemme also appears to pursuing another line of thought about this discrepancy in recent twitter posts.

Peter Lemme‏ @Satcom_Guru
A bent vane or mass imbalance are the likely culprits. Mass imbalance might be harder to notice. Mass imbalance would reflect something broken, which might correlate to loss of damping. AoA vane supplier should have failure mode and effects or other analysis to confirm. #JT610

Salute Wiede
I agree with you about the decoding in the SMYD or two of them, if left vane to one and right to another. From there, looks like AoA value goes to the two ADIRU and FCC components. Since the pilot control had the shaker and from what others here have presented, I have to go with two of the SYMD's or a single one maintains the data from left for left ADIRU and right for the co-pilot side.

Unlike my trusty Viper, only one AoA can cause serious problems in the 737. Our system had the basic two conical AoA probes and then a serious looking hemispherical proble that used 5 holes to produce sideslip, AoA, static and dynamic pressures for use. We used a middle value prorocol from the two mechanical cones and one pneumatic sensor.

Maybe FC eng can find a better "flow chart" or "block diagram" like we old farts used back in FORTRAN IV and sliderule days, heh heh.

Gums sends....