Go Back  PPRuNe Forums > Flight Deck Forums > Tech Log
Reload this Page >

Maintenance Lapse Identified as Initial Problem Leading to Lion Air Crash

Tech Log The very best in practical technical discussion on the web

Maintenance Lapse Identified as Initial Problem Leading to Lion Air Crash

Old 28th Dec 2018, 18:27
  #41 (permalink)  
swh

Eidolon
 
Join Date: May 2001
Location: Some hole
Posts: 2,171
Received 24 Likes on 13 Posts
Originally Posted by BAengineer
I assume that the reason of the check not being mandatory is that there is no calibration adjustment possible on the aircraft - it either passes or it doesn't..
Exactly, the article is a pile of horse manure. No calibration is done, it comes calibrated from the shop and the screw holes line up where it goes.

if you read the full article and not just what is above the source of this information is not the Indonesian investigations, NTSB, or Boeing, they all declined to comment about this.

it comes from joe public who is “following the investigation”. They got their $1000 of click bait out of it.
swh is offline  
Old 28th Dec 2018, 21:58
  #42 (permalink)  
 
Join Date: Mar 2014
Location: WA STATE
Age: 78
Posts: 0
Likes: 0
Received 0 Likes on 0 Posts
A long detailed discussion of AOA sensors by Boeing- note that 737 is not mentioned- except in the document dont know how old this is but pdf document shows date of year 2000

Aero 12 - Angle of Attack


and at bottom of online text emphasis added

SUMMARYAOA is a long-standing subject that is broadly known but one for which the details are not broadly understood. While AOA is a very useful and important parameter in some instances, it is not useful and is potentially misleading in others.
  • The relationship between AOA and airplane lift and performance is complex, depending on many factors, such as airplane configuration, Mach number, thrust, and CG.
  • AOA information is most important when approaching stall.
  • AOA is not accurate enough to be used to optimize cruise performance. Mach number is the critical parameter.
  • AOA information currently is displayed on Boeing flight decks. The information is used to drive the PLI and speed tape displays.
  • An independent AOA indicator is being offered as an option for the 737, 767-400, and 777 airplanes. The AOA indicator can be used to assist with unreliable airspeed indications as a result of blocked pitot or static ports and may provide additional situation and configuration awareness to the flight crew.

Last edited by CONSO; 28th Dec 2018 at 22:10. Reason: added year of publication of pdf
CONSO is offline  
Old 29th Dec 2018, 12:34
  #43 (permalink)  
 
Join Date: Jan 2008
Location: Reading, UK
Posts: 15,786
Received 196 Likes on 90 Posts
Originally Posted by wiedehopf
The problem at first was an intermittent AoA signal. That meant no airspeed and altitude for the captain sometimes because AoA is included in the calculation.
After the change of the AoA sensor in Denpasar the signal was not intermittent but offset by 20 degrees on the previous and the accident flight.
Are you sure about that ?

I haven't seen any FDR traces for the two flights where problems were reported that occurred prior to the AoA sensor replacement. Nor AFAIK has there been any report on the condition of the removed sensor.

IMHO it's entirely possible that neither sensor was faulty and both were correctly installed, i.e. the fault was farther downstream and replacing the sensor made no difference.
DaveReidUK is online now  
Old 29th Dec 2018, 12:50
  #44 (permalink)  
 
Join Date: Jul 2014
Location: Germany
Posts: 344
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by DaveReidUK
Are you sure about that ?

I haven't seen any FDR traces for the two flights where problems were reported that occurred prior to the AoA sensor replacement. Nor AFAIK has there been any report on the condition of the removed sensor.

IMHO it's entirely possible that neither sensor was faulty and both were correctly installed, i.e. the fault was farther downstream and replacing the sensor made no difference.
No i'm not sure. I'm reading between the lines.
On the preliminary report on pages 7, 8 and 9 the maintenance log is described.
(Preliminary Report: http://jacdec.de/database/2018-10-29...y%20Report.pdf)

Before changing the AoA sensor the main symptom is:
Speed and Altitude Flag show on Captain Primary Flight Display (no speed and altitude indication)
Only after changing the AoA sensor the IAS disagree behaviour started:
IAS and ALT Disagree shown after take off
Also i'm quite certain if the wrong AoA with stick shaker activation would have been present prior to the Denpasar-Jakarta leg it would have been mentioned in the report.

Could also be maintenance screwed up the SMYD1 somehow i don't know. But it seems certain the really dangerous fault was introduced with that maintenance option.

Of course the report could have left out that the other flights had an AoA disagree as well but somehow i doubt that. Also yes the maintenance log could be wrong, but no speed indication vs airspeed disagree indication, i trust them to get at least that much right. Especially because there were multiple occasions of that problem.
wiedehopf is offline  
Old 29th Dec 2018, 15:13
  #45 (permalink)  
 
Join Date: Jun 2009
Location: florida
Age: 81
Posts: 1,609
Received 52 Likes on 15 Posts
Salute!

One thing I would have hated in that left seat would have been the stoopid shaker remaining activated no matter what we did.

The trace we see from the previous flight clearly shows the shaker active until some "event" upon landing ( no flow chart, so I guess it was WoW), and it was clearly not operating as part of the MCAS implementation. The shaker remained active regardless of flap position, time after takroff and turning off the electrical pitch trim system ( those guys flew using the manual trim wheel, and I guess the captain let the FO do most stuff and just "tolerated" the shaker when he used his control). So now we see that the shaker is driven independently of MCAS. This had to be confusing, even if you knew all about MCAS. Especially if you had turned of the stab trim to get rid of the annoying nose down trim that Hal was commanding over and over and over and "went manual" ( feel the Force, Luke)

The maintenance log shows the wrenchbenders trying to cure the airspeed gripes, and one of the computer codes pointed to the AoA. Had they left that sucker alone we would not be here on this forum, huh? And why didn't the crew gripe about the incessant shaker?

Something bad happened from the AoA sensor along the path to the MCAS and the shaker. A good flow chart/block diagram would really help here, but I am not sure the current generation believes in those things anymore. ANother process was our "murder boards" where we crackerjack pilot/engineers briefed our system or proposal using said charts/diagrams, and then explaining each and every interface and results from failures all along the various paths Hal or the lowly carbon-based lifeforms could take.

Gums sends...
gums is online now  
Old 29th Dec 2018, 20:22
  #46 (permalink)  
 
Join Date: Jan 2008
Location: uk
Posts: 857
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by DaveReidUK
Are you sure about that ?

I haven't seen any FDR traces for the two flights where problems were reported that occurred prior to the AoA sensor replacement. Nor AFAIK has there been any report on the condition of the removed sensor.
No FDR but Maint log in prelim report shows the following AOA errors before sensor replacement:

AOA SIGNAL FAIL (34-21023)
34-21107 (AIR DATA SIGNAL INVALID)
34-21123 (AOA SIGNAL OUT OF RANGE)

We also know the problems were intermittent because I think it's already been shown in these threads that there was at least one uneventful sector in between.

IMHO it's entirely possible that neither sensor was faulty and both were correctly installed, i.e. the fault was farther downstream and replacing the sensor made no difference.
Agree - I think far more likely one fault elsewhere than two faulty sensors. However, replacing the sensor did make a difference: an intermittent signal corruption / loss changed into a constant signal corruption...
infrequentflyer789 is offline  
Old 30th Dec 2018, 02:57
  #47 (permalink)  
 
Join Date: Jul 2009
Location: Not far from a big Lake
Age: 81
Posts: 1,454
Likes: 0
Received 0 Likes on 0 Posts
Thank you weidehopf for pointing out the earlier flight maintenance history. It points out an area for investigation.
Originally Posted by gums
The maintenance log shows the wrenchbenders trying to cure the airspeed gripes, and one of the computer codes pointed to the AoA. Had they left that sucker alone we would not be here on this forum, huh? And why didn't the crew gripe about the incessant shaker?
IMO the AOA maintenance had nothing to do with the stick shaker problem, but something else done at the same time probably did trigger the AOA signal problem. I agree that the prior crew needed to do a better job explaining the seriousness of the problem to maintenance.
I also have to fault Boeing for failing to tell pilots about their new MCAS system which handicapped the prior crew (and probably maintenance as well) in explaining what their problem was. The aircraft should never have been released for a revenue flight without a test flight by an experienced crew.
The hardest problems to diagnose are intermittent ones. Computers are able to aid the diagnosis of problems by recording what the symptoms were, but system complexity often prevents actual diagnosis, and only suggests a path toward correction.
Lets dig into the maintenance history a bit:





Comments to follow. I'm, looking at the conductive FOD scenario causing erratic behavior.
Machinbird is offline  
Old 31st Dec 2018, 06:00
  #48 (permalink)  
 
Join Date: Jul 2009
Location: Not far from a big Lake
Age: 81
Posts: 1,454
Likes: 0
Received 0 Likes on 0 Posts
Maintenance Decryption-Part 1

On October 26th on a flight from Tianjin Binhai China to Manado Indonesia, the Captains PFD had speed and altitude flags which caused speed and altitude not to be displayed and caused the maintenance light to illuminate after landing.Maintenance used the Onboard Mainteance Function (OMF) to identify a code 27-31000. Maintenance used the Interactive Fault Isolation Manual for that code and did a self test on the SMYD1 box which showed no errors. As a result, they erased the maintenance message, and cleared the aircraft for flight.Between October 26th and 27th the aircraft flew from Manado to Denpasar, apparently without a gripe.

Then, on October 27th, on a flight from Denpasar to Manado, the Captains Speed and altitude flag problem reoccurred together with Speed Trim fail and Mach Trim fail warning lights. This time, there was a message, “Stall Warning System Left” and the OMF spit out a code 27-32-00-810 thru 816.
A self test on SMYD1 failed.While trouble shooting the Speed Trim Fail and Mach Trim fail warnings, there were another set of IFIM codes indicating ADR invalid and AOA signal fail.Corrective action was to reset circuit breakers on ADIRU L DC and AC, and ADIRU L (self-test?) carried out and performed system test on SMYD1-result pass.

A few questions for the maintenance folks here:
  1. Does Reset a circuit breaker mean to push a tripped breaker back in or does it mean to cycle it, or does it mean either of those possibilities?
  2. There is a secondary correction of “Reconnect and clean electrical plug of data module” Is this data module located inside SMYD1?
  3. Is this the device that digitizes the AOA data?
I’m looking for actions that would potentially relocate conductive FOD inside SMYD or its internal / associated components.
If you have intimate knowledge of these systems, feel free to look ahead in the maintenance actions previously posted above to see if you recognize something that would move something associated with processing the AOA signal.
I strongly suspect that something else was moved around at the same time the AOA probe was replaced. That electrical disturbance (noise) seen on the Left AOA signal should not be glossed over. You can even see some extra spikes in the Left AOA data during the accident flight that are not matched by the Right AOA signal.
Machinbird is offline  
Old 31st Dec 2018, 09:43
  #49 (permalink)  
 
Join Date: Jul 2014
Location: Germany
Posts: 344
Likes: 0
Received 0 Likes on 0 Posts
Identifying the fluctuation in the AoA signal as electrical noise does not make sense to me.

Being only present on the ground it is much more likely it is some form of imbalance situation in the AoA vane.
Normally you want the AoA vane combined with the internal rotating parts of the sensor to be balanced so it doesn't react too much to bumps in the runway or turbulence in the air.
So i find a mechanical problem with the AoA vane like a broken off part combined with a bend much more likely.

Also no one has explained so far why electrical noise would never be visible during flight.
wiedehopf is offline  
Old 31st Dec 2018, 11:14
  #50 (permalink)  
 
Join Date: Feb 2013
Location: 60 north
Age: 59
Posts: 17
Likes: 0
Received 0 Likes on 0 Posts
Gums
With regards to tolerating the false stick shaker I would time permitting pull the CB and kill it!
Smart: maybe not?
BluSdUp is offline  
Old 1st Jan 2019, 21:04
  #51 (permalink)  
 
Join Date: Jul 2009
Location: Not far from a big Lake
Age: 81
Posts: 1,454
Likes: 0
Received 0 Likes on 0 Posts
Identifying the fluctuation in the AoA signal as electrical noise does not make sense to me.

Being only present on the ground it is much more likely it is some form of imbalance situation in the AoA vane.
Normally you want the AoA vane combined with the internal rotating parts of the sensor to be balanced so it doesn't react too much to bumps in the runway or turbulence in the air.
So i find a mechanical problem with the AoA vane like a broken off part combined with a bend much more likely.

Also no one has explained so far why electrical noise would never be visible during flight.
Wiedehopf, I gather that you have not had to troubleshoot intermittent electrical problems before-fair enough.
Consider an out of balance AOA sensor as a potential cause. This would create a constant torque on the sensor in the increasing AOA direction (assuming vane heavy). As we go faster in the air, the restoring force is increasing and the error would decrease, but this is not what we are observing-we are seeing a constant error between sides.
With regard to a bent vane causing the error, we would be assuming gross incompetence on the part of a number of people, both maintainers and whomever in the aircrew preflighted the aircraft for the last two flights. A bent vane with a 20 degree offset should be as obvious as the nose on your face.
Why would we be seeing Left AOA noise on the ground but not in the air? Consider the difference in vibration environments. The SMYD boxes are mounted in the nose avionics bay, correct? The front end of the airplane experiences much more jiggling on the ground than does the area of the aircraft closer to the CG and main gear. (Watch any Youtube video of a transport crew making al landing, and there should be no doubt in your mind when the aircraft has touched down just from the bouncing around the crew gets when they touch down.)
Finally, consider the intermittency of the problem. Different presentations (or lack of presentation in 1 case) of the problem over the course of 5 flights. During maintenance, the problem would often fail to present itself, and only leave its calling cards. This level of complexity shouts intermittent electrical problem, loud and clear.
Machinbird is offline  
Old 1st Jan 2019, 23:15
  #52 (permalink)  
 
Join Date: Jul 2014
Location: Germany
Posts: 344
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Machinbird
Finally, consider the intermittency of the problem. Different presentations (or lack of presentation in 1 case) of the problem over the course of 5 flights. During maintenance, the problem would often fail to present itself, and only leave its calling cards. This level of complexity shouts intermittent electrical problem, loud and clear.
While i agree that an intermittent electrical problem may well have been present, the coincidental 20 degree offset connected to the replacement of the sensor is quite stunning.
So the intermittent electrical fault might have nothing to do with the offset at all.
Also a constant exact offset of 20 degrees does not fit at all with an intermittent problem.

Your supposed electrical noise could be wind as well that is coming from one side while taxiing. Maybe it is gusty and therefore producing the "noise" you see.
The temporal resolution of the FDR readout is far too bad to conclude that that is electrical noise.

The imbalance in the sensor could be such that it is apparent when taxiing over rough taxiways creating the "noise" but is irrelevant as soon as the airspeed picks up on the takeoff roll. Aerodynamic forces far outweigh any inbalance at those speeds.
Such imbalance could be created by violently attaching the sensor. Or bending/breaking the vane.

And yes i find gross incompetence and negligence far more likely than an electrical error producing a 20 degree offset.
But i will entertain the idea gladly and ask you how an electrical problem could introduce such a fault?

The SMYD has two inputs for the AoA sensor, labeled sinus and cosinus.

This for me points to a resolver.
https://en.wikipedia.org/wiki/Resolver_(electrical)

Now i really have no idea how an intermittent or basically any electrical problem would introduce the 20 degree offset.
Do you have an idea?
(I'm happy to be corrected on the points made, it's about the puzzle less about being right in my opinion)
wiedehopf is offline  
Old 1st Jan 2019, 23:39
  #53 (permalink)  
 
Join Date: Mar 2014
Location: WA STATE
Age: 78
Posts: 0
Likes: 0
Received 0 Likes on 0 Posts
RE WEIDEHOPF - is it possible to mount the resolver to the vane incorrectly, eg a 20 degree rotation due to mis- clocking ? or is there some sort of indexing-? neet a detailed assembly diagram or such for the AOA sensor- resolver ... is there 9 equally spaced screws mounting resolver to vane ?
CONSO is offline  
Old 2nd Jan 2019, 00:02
  #54 (permalink)  
 
Join Date: Jul 2014
Location: Germany
Posts: 344
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by CONSO
RE WEIDEHOPF - is it possible to mount the resolver to the vane incorrectly, eg a 20 degree rotation due to mis- clocking ? or is there some sort of indexing-? neet a detailed assembly diagram or such for the AOA sensor- resolver ... is there 9 equally spaced screws mounting resolver to vane ?
I don't have detailed information on the sensor. But i would presume the resolver is normally not detached from the vane by line mechanics, instead my guess would be that the unit is only opened by the manufacturer.

Maybe it has been said before but i'm curious, is the AoA sensor used on the MAX the same as on the 737 NG?
wiedehopf is offline  
Old 2nd Jan 2019, 02:10
  #55 (permalink)  
 
Join Date: Jul 2009
Location: Not far from a big Lake
Age: 81
Posts: 1,454
Likes: 0
Received 0 Likes on 0 Posts
Wiedehopf, we are badly handicapped by not having access to schematics of the SMYD box, its associated systems, and the Interactive Fault Isolation Manual to help better understand the maintenance codes. All we have going for us is a bit of theory. Some activity like Boeing that holds the actual data package for the aircraft, or the manufacturer of the SMYD boxes are the only ones likely to have sufficient data to figure this out in detail.

If we look at the Sine and Cosine signals from the AOA resolver, these are analog signals and have to be put together to make a piece of angle data. At some point, apparently in close proximity to the SMYD box, these signals are digitized and sent by the aircraft data bus throughout the aircraft.
There are two ways to put these signals together that I can immediately think of.
  1. Digitize the two signals independently, and then digitally convert the result to an angle, or
  2. Combine both Sine and Cosine signals to an analog angle signal, and then digitize that.
There are purpose built circuits for digitizing resolver data, but they may be newer in design than the currently utilized SMYD method.

In the analog state, any voltage applied to the combined angle analog signal will create an error that has the potential to change the indicated angle. Once the data is fully digitized, the potential for inducing errors is much reduced.
Electronic circuits are very dense, and any conductive FOD has the potential to lodge among the circuit traces (unless potted) and create sneak circuits.

If Boeing can nail down where in the system this proposed FOD probably landed, they could likely duplicate the error by inserting a BB onto the circuit board and rolling it around-
There is also a second potential mode of failure to intermittently lose an internal reference voltage due to a failed component or improperly soldered connection.
Remember, this was a brand new aircraft and it could have experienced an "infantile" failure of a circuit component.
At our level, we are not going to be able to positively solve this question. but we can look for indications in the maintenance record that indicate other maintenance actions could have disturbed SMYD components simultaneous with the changeout of the AOA probe.
The difficulty with this is proper understanding of technical abbreviations, current maintenance word usage, and IFIM codes/workpackages.

Personal background: I first flew using AOA over 50 years ago, and have had plentiful opportunity to observe AOA vane behavior in an aircraft carrier flight deck environment, so I think I understand the subject of cross winds and jet blast impingement fairly well. I have also worked in aircraft maintenance in a number of capacities. There is still a fair bit for this old dog to learn though.
Machinbird is offline  
Old 2nd Jan 2019, 02:28
  #56 (permalink)  
 
Join Date: Jul 2009
Location: Not far from a big Lake
Age: 81
Posts: 1,454
Likes: 0
Received 0 Likes on 0 Posts
is there 9 equally spaced screws mounting resolver to vane ?
Conso, either find a picture of the nose of a NG/MAX 737 and count the screw holes or go to the airport and make your own picture. I doubt if it has more than 6.
They can limit the interchangeability of probes by playing with the screw hole spacing, the mounting plate size, the electrical connectors, etc. There is an indexing pin involved as well, and that can control proper installation too.
With 9 holes you you would have a 40 degree error. That doesn't fit the scenario.
Machinbird is offline  
Old 2nd Jan 2019, 04:00
  #57 (permalink)  
 
Join Date: Mar 2014
Location: WA STATE
Age: 78
Posts: 0
Likes: 0
Received 0 Likes on 0 Posts
Smile

Originally Posted by Machinbird
Conso, either find a picture of the nose of a NG/MAX 737 and count the screw holes or go to the airport and make your own picture. I doubt if it has more than 6.
They can limit the interchangeability of probes by playing with the screw hole spacing, the mounting plate size, the electrical connectors, etc. There is an indexing pin involved as well, and that can control proper installation too.
With 9 holes you you would have a 40 degree error. That doesn't fit the scenario.
reason im picked 9 was due to 180 degree change right to left - and i was talking mounting the electric transducer internal to the vane and not the external mount. still cannot find details of AOA sensor which was **PROBABLY** made by Rockwell- Collins
CONSO is offline  
Old 2nd Jan 2019, 05:20
  #58 (permalink)  
 
Join Date: Apr 2004
Location: near EDDF
Posts: 775
Likes: 0
Received 0 Likes on 0 Posts

1 AOA
2 Alignment Pins
8 Screws
IFixPlanes is offline  
Old 2nd Jan 2019, 08:18
  #59 (permalink)  
 
Join Date: Jan 2008
Location: Reading, UK
Posts: 15,786
Received 196 Likes on 90 Posts
Originally Posted by infrequentflyer789
We also know the problems were intermittent because I think it's already been shown in these threads that there was at least one uneventful sector in between.
Not according to the KNKT, who stated that problems had been encountered on four consecutive sectors.
DaveReidUK is online now  
Old 2nd Jan 2019, 13:14
  #60 (permalink)  
 
Join Date: Jan 2008
Location: uk
Posts: 857
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by DaveReidUK
Not according to the KNKT, who stated that problems had been encountered on four consecutive sectors.
KNKT also shows the four sectors from the maint log (in the prelim report and posted in full further up the thread):

26 October 2018 :: Tianjin Binhai to Manado
27 October 2018 :: Denpasar to Manado
27 October 2018 :: Manado to Denpasar
28 October 2018 :: Denpasar to Jakarta

Methinks that Occam's razor comes down on the side of "four consecutive" being a translation error, as opposed to teleportation of a 737 from Manado to Denpasar - I could be wrong though...
infrequentflyer789 is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service

Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.