Of modern airliner certification

[atakacs]

This is an honest question: under what regulation / system was the the 787 MAX certified ? What are the actual regulations ? Does an FAA approval necessarily extend for the whole world ?
I understand in this specific case we have an extension of the original 737, thus not requiring a full review. Still, how far can you go with that approach ?
Of obvious particular interest: is it actually possible / legal to certify any autonomous flight control system (however benign, and MCAS isn't) that would get input from a single sensor ?

[gearlever]

Delegated to Boeing (https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/)

The FAA, citing lack of funding and resources, has over the years delegated increasing authority to Boeing to take on more of the work of certifying the safety of its own airplanes.Early on in certification of the 737 MAX, the FAA safety engineering team divided up the technical assessments that would be delegated to Boeing versus those they considered more critical and would be retained within the FAA.But several FAA technical experts said in interviews that as certification proceeded, managers prodded them to speed the process. Development of the MAX was lagging nine months behind the rival Airbus A320neo. Time was of the essence for Boeing.A former FAA safety engineer who was directly involved in certifying the MAX said that halfway through the certification process, “we were asked by management to re-evaluate what would be delegated. Management thought we had retained too much at the FAA.”“There was constant pressure to re-evaluate our initial decisions,” the former engineer said. “And even after we had reassessed it … there was continued discussion by management about delegating even more items down to the Boeing Company.” Even the work that was retained, such as reviewing technical documents provided by Boeing, was sometimes curtailed.“There wasn’t a complete and proper review of the documents,” the former engineer added. “Review was rushed to reach certain certification dates.”

[DaveReidUK]

Does an FAA approval necessarily extend for the whole world ?

No, but FAA disapproval does, so to speak.

FAA type certification is accepted by many countries, but notable exceptions (other authorities that issue their own TCs) include Brazil and, of course, EASA.

That said, as it's a US aircraft the FAA has prime responsibility for continuing airworthiness for the type, so if the FAA suspends or withdraws the TC then the other authorities will inevitably follow suit.

[GlobalNav]

This is an honest question: under what regulation / system was the the 787 MAX certified ? What are the actual regulations ? Does an FAA approval necessarily extend for the whole world ?
I understand in this specific case we have an extension of the original 737, thus not requiring a full review. Still, how far can you go with that approach ?
Of obvious particular interest: is it actually possible / legal to certify any autonomous flight control system (however benign, and MCAS isn't) that would get input from a single sensor ?
Tranport airplanes, actually the “type design” of transport airplanes, are certified by the FAA under 14 CFR Part 25. Other countries have similar, if not identical certification requirements. Other countries may choose to validate the FAA certification and certify the system under its own rules.

It is significant to note that, over time, these rules are amended, presumably to an increased level of safety and/or to address newer technology not originally addressed. For a brand new model, like the 787, the latest amendment at the time of application is applied. For the 737 (or 747) it gets complicated to establish the certification basis, which according to certain criteria uses various amendment levels for different subsystems depending (in a simplified description) on the amendment levels used in earlier certification for those systems. New materials, new designs and technologies dictate the most current amendment. (This is a simplified explanation to an increasingly complex process.)

One rule that can be complicated and controversial to comply with is “System Safety”, 25.1309. A key concept of this rule, for the systems that it applies to, is that the level of safety (probability of a failure event) must be commensurate with the level of hazard associated with it. A catastrophic failure event is one that may result in loss of an airplane and most occupants. It’s probability must be “Extremely Improbable” and numerically is on the order of one such event per 10E-9 flight hours, which practically means it would not happen (but...).

To design a system to keep the probability of a particular failure event to that level is most expensive and accomplished through strategies like redundancy, dissimilar paths, and software design assurance level A.

Regarding your particular question about autonomous flight control systems, the FAA would probably apply a combination of existing rules at the latest amendment level, such as 25.1329, and to the degree that the system is considered “new and novel”, that is, not covered adequately by existing rules, one or more “special conditions”, which are new rules written specifically for that certification. Particular modes and functions that have a high degree of control authority and potentially catastrophic failure modes, a single sensor input could hardly comply with the requirements. There are many variables affecting the design strategies needed to assure the required level of safety. For example, are there reliable means to detect an invalid sensor signal and prevent its use by the system.

Too complicated a question for a short answer and one that would take tons of technical meetings and debates before an agreement would be reached.

[atakacs]

(...)
Too complicated a question for a short answer and one that would take tons of technical meetings and debates before an agreement would be reached.
Thanks - very interesting and informative post (at least to me). So bottom line there is no easy answer, especially with the case at hand (737 MAX).

I'd be really interested - and I'm certainly not alone - in the certification process for the MCAS. I really hope for all involved that it was not "self-certified "... Is this process publically documented?

[tdracer]

Thanks - very interesting an informative answer (at least to me). So bottom line there is no easy answer, especially with the case at hand (737 MAX).

I'd be really interested - and I'm certainly not alone - in the certification process for the MCAS. I really hope for all involved that it was not "self-certified "... Is this process publically documented?

Much of what an airframer provides during the certification is considered proprietary and/or commercially sensitive. So while the process is documented and available via the freedom of information act, the details and documentation typically are not. In fact much of the documentation submitted to the authority is either returned to the airframer or destroyed once the certification is finished.
When making a significant change to an existing aircraft model, there is something called the Changed Product Rule (CPR) - which was negotiated and 'harmonized' about 20 years ago based on input from Boeing, Airbus, FAA, EASA/JAA, and a host of others (harmonized means that the rules are identical between the FAA and EASA). It has specific rules regarding what would require an Amended Type Cert (ATC) vs. the normal ongoing changes (e.g. new interiors, updated flight control or FADEC software, etc.). Normal on-going changes use the cert basis as defined in the TC - an Amended TC opens up the cert basis. The details can get messy, but in general portions of the aircraft that are 'unchanged' can use their existing cert basis, while new or changed systems need to step up to the latest regulations.
I would assume that since MCAS was a new system for the MAX, it needed to be certified to the latest regulations. My educated guess is that MCAS wasn't given the hazard classification it really deserved (i.e. it was defined as Major or Hazardous, instead of Catastrophic) and hence wasn't given the level of attention it deserved.

BTW, it's worth noting that the push to "Delegated Compliance" wasn't Boeing's idea - it was dictated by the FAA in Washington DC and is being applied to all airframers and major suppliers in the US, not just Boeing. My understanding is that EASA has something similar with Airbus and the other airframers/suppliers in Europe/UK.
I was a DER - which became "Authorized Representative" or AR once Boeing became delegated - and almost to a person the DER/ARs hated it. It made the DER/AR job much, much harder.

[megan]

From https://www.avweb.com/eletter/archives/101/4288-full.html?ET=avweb:e4288:244160a:&st=email#232426MCAS Certification Flawed: Report - Russ Niles

The Maneuvering Characteristics Augmentation (MCAS) system at the center of investigations into two fatal crashes of the Boeing 737 MAX 8 was misunderstood and mischaracterized in a flawed certification process as Boeing and the FAA rushed to bring the new jet to market, a Seattle Times investigation published Sunday alleges. https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/?utm_source=marketingcloud&utm_medium=email&utm_campaign=BNA_031719130226+Crucial+flaws+in+Boeing+737+MA X+safety+analysis_3_17_2019&utm_term=Active+subscriber&fbclid=IwAR0heI7SPRsoWF0DnSKthNZ9opvMePUi1DQpRJ1K6bTC2XjUQIS 5JFZ49SM

Citing named and unnamed sources, the Times’ Dominic Gates says the final certification of the system, which was intended to give pilots a control feel on the aerodynamically different MAX similar to that of previous iterations of the 737, not only gave “unlimited authority” to the stabilizer for nose-down trim, it literally fought the pilots’ attempts to correct the condition possibly to the point where they were physically unable to fight the stabilizer down force any longer.

“It had full authority to move the stabilizer the full amount,” Peter Lemme, former Boeing flight controls engineer, told the Times. “There was no need for that. Nobody should have agreed to giving it unlimited authority.

”The Times story said the profound ability of the system to take over a key flight control action should have resulted in close scrutiny in the certification process.

But the original specifications of the system called for MCAS to limit its ability to move the horizontal stabilizer .6 degrees at a time. By the time deliveries began, it could pitch the stabilizer 2.5 degrees, about half its total travel, in one movement, the result of flight testing tweaks aimed at finessing the flight control feel.

The system would also pivot the stabilizer that much repeatedly as long as data inputs indicated the aircraft was about to stall, regardless of the pilots’ strenuous efforts to overpower the system. In the October Lion Air crash, which killed 189 people, the flight data recorder counted the captain countering the system 21 times with the first officer taking over for few tries before the captain’s final futile efforts to arrest a 500-MPH dive. The data indicated the nose-down yoke forces peaked at a little more than 100 pounds.

The newspaper’s investigation said that engineers involved in the safety assessment of MCAS were not aware the system could move the tail five times more than the original specs called for. The certification documents should have been amended to reflect the final configuration but they apparently were not, according to the Times report. If they had been, the seriousness of a potential failure of the system would have required it to receive data from at least two sources.

MCAS gets data from only one of two angle of attack indicators on the MAX and the flight data recorder on the Lion Air airplane showed the AOA feeding MCAS was malfunctioning. “A hazardous failure mode depending on a single sensor, I don’t think passes muster,” said Lemme.

The newspaper is reporting that Boeing’s software fix will wire MCAS to both AOAs and only allow the system to move the tail feathers once, instead of repeatedly battling manual control inputs. It will also require additional pilot training and operating manual changes, both of which were called for by pilots unions following the Lion Air crash.

Boeing’s position, endorsed by the FAA, has been that because MCAS is only supposed to trigger in extreme circumstances—high angles of attack and accelerated stalls—that additional pilot training was not necessary. The company has also said that it assumed that based on their existing training on earlier models pilots would recognize the erroneous nose-down commands and hit cutoff switches that would disable the system. This is a standard runaway trim scenario for all aircraft.

“The assumptions in here are incorrect. The human factors were not properly evaluated,” the Times quoted an unnamed FAA safety engineer as saying.

The story also suggests that due to budget cuts the FAA’s certification managers were under increasing pressure to delegate more and more of the safety assessments to Boeing itself. The unprecedented levels of self-certification in the MAX were compounded by the urgency to get the airplane into service because of competitive pressure from Airbus’s new A320neo series. “There wasn’t a complete and proper review of the documents,” the former FAA engineer is quoted as saying. “Review was rushed to reach certain certification dates.

[hans brinker]

So a follow up question. As the 737 is an older design, would the MAX still only be held to the rules in effect in 1967?

[tdracer]

So a follow up question. As the 737 is an older design, would the MAX still only be held to the rules in effect in 1967?

You didn't read my post - at least the part related to Change Product Rule.
CPR basically says unchanged aspects can retain their original cert basis, while new or changed aspects need to step up to the latest regulations (and amendment levels).

I suspect, aside from some structural stuff, precious little of the MAX is still certified to the 1967 rules.

[hans brinker]

You didn't read my post - at least the part related to Change Product Rule.
CPR basically says unchanged aspects can retain their original cert basis, while new or changed aspects need to step up to the latest regulations (and amendment levels).

I suspect, aside from some structural stuff, precious little of the MAX is still certified to the 1967 rules.
I did, but not very well. Thanks for clarifying.

[atakacs]

Very informative posts.

Time to throw in the EASA. Did they actually certify the 737 MAX or did they simply accept the FAA aka Boeing review as gospel ?

[wrmiles]

One other issue regarding safety/failure compliance with 25.1309 is aircrew awareness and mitigation of the failure. This is allowed, however the aircrew must be warned by either a dedicated warning means or inherent flying characteristics, and corrective action must not require exceptional flying skill. There is room for some subjectivity and differences in interpretation here.

[tdracer]

Very informative posts.

Time to throw in the EASA. Did they actually certify the 737 MAX or did they actually took the FAA aka Boeing review as gospel ?

Starting with (IIRC) the 777, all new or major Boeing derivatives have been jointly certified by both the FAA and the JAA/EASA. What EASA typically does is pick out various 'areas of interest' - usually (but not always) areas where the FARs and JAR/CS differ. This starts out with a series of meetings between Boeing and EASA where they agree to a list of 'significant regulatory differences' (i.e. the regulations where showing compliance to the FARs won't necessarily show compliance with the corresponding CS) and this is documented by EASA in a "CRI" (Certification Review Item). Every CS in this CRI needs to be certified with EASA - not the FAA - although the process of how that certification occurs is very similar to how it's done with the FAA - and yes EASA will delegate some compliance activities to either the FAA or Boeing. Areas of specific interests with result in additional CRI's issued by EASA (very similar to the Issue Paper process with the FAA). Like Issue Papers, all CRIs must be agreed to and closed prior to Type Cert.
As a general rule, the certification of on-going changes/improvements after Type Cert are delegated to the FAA (Boeing still needs to show compliance with the applicable CS, but the FAA certifies the change without direct EASA involvement). Of course, there are exceptions...

[groundbum]

The FAAs mission, copied from their website

"Our continuing mission is to provide the safest, most efficient aerospace system in the world."

I would suggest there would be a lot of tension between SAFEST and MOST EFFICIENT.

I also thought the FAA had as part of their remit to promote commercial aviation but can't find a reference.

G

[LeadSled]

The FAA and Boeing have a bit of history when it comes certification review.

https://soundcloud.com/aerosociety-podcast/audio-the-dp-davies-interview-comets?in=aerosociety-podcast/sets/an-interview-with

One of a log series of RAeS interviews with D P Davies. The B707 section is worth a listen.


Folks,
In my opinion, based on hard experience, is that the modifications demanded for G- (UK) certification of the B707 introduced significant hazards to operating this aircraft.
They included, but were not limited to: Changes to the T/O stab settings for takeoff, modifications to the spoiler control system, and initially, limiting the landing flap settings..
In my opinion, Davies was a person of limited flying experience but very strong prejudices --- particularly involving any aircraft built on the western side of the Atlantic
My experience extends to air/ground photography of test flight wrecks on the ground, resulting from stall testing to the D.P.D. prescription --- a modern jet should handle like a Tiger Moth. ---- and having an ARB testing approval (not Experimental) with his autograph affixed.
Tootle pip!!

[Zeffy]

FAA and Industry Guide to Certification (https://www.faa.gov/aircraft/air_cert/design_approvals/media/CPI_guide.pdf)

This version of the Guide also recognizes that there are significant variations in the compliance maturity level of companies in terms of processes and capabilities. Likewise, FAA oversight offices may vary in how they conduct their oversight responsibilities. Therefore, the level of effort in achieving the objectives of the Guide expectations will be different from one Applicant or FAA office to another.

[GlobalNav]

FAA and Industry Guide to Certification (https://www.faa.gov/aircraft/air_cert/design_approvals/media/CPI_guide.pdf)

Chapter 5 may be of particular interest.

[Mansfield]

The FAAs mission, copied from their website

I also thought the FAA had as part of their remit to promote commercial aviation but can't find a reference.

G

The FAA's Dual Mandate was eliminated by Title IV, Section 401 of the Federal Aviation reauthorization Act of 1996...

[Mansfield]

It is interesting to compare the Dominic Gates article from the Seattle Times with the Qantas 72 report. If you're not familiar, QF72 was an A330 that experienced multiple pitch upsets at cruise back in 2008. Several injuries and a successful diversion. The problem was traced to random data spikes from the No. 1 ADIRU. It's a great report; only one I know of that discusses system safety assessments in detail.

That said, consider this: In the QF72 report, the ATSB said: "The A330/A340 FCPC algorithm for processing AOA data was redesigned after a problem was found with the initial algorithm during flight testing that was conducted before the aircraft type was certified. The redesign unintentionally introduced the design limitation in the algorithm, and the fault-tolerant features of the system were not able to fully mitigate the problem. The design limitation was not identified during the redesign activities. Although the SSA identified the relevant failure condition (incorrect, high AOA data leading to a pitch-down command), it did not identify the scenario that led to this condition on the 7 October 2008 flight. The results of the SSA and other design evaluation activities can be summarised as shown in Figure 54."

In yesterday's article, Gates said: "The original Boeing document provided to the FAA included a description specifying a limit to how much the system could move the horizontal tail - a limit of 0.6 degrees, out of a physical maximum of just less than 5 degrees of nose-down movement. That limit was later increased after flight tests showed that a more powerful movement of the tail was required to avert a high-speed stall, when the plane is in danger of losing lift and spiraling down."

The QF72 report also discusses the complete absence of human factors considerations in the design room, as opposed to the cockpit. Airbus missed the alteration to the safety assessment created by the change in software during flight test. It strikes me that perhaps Boeing did the same…also during flight test, and possibly for the same reasons. Food for thought.

[zzuf]

So a follow up question. As the 737 is an older design, would the MAX still only be held to the rules in effect in 1967?

Not as heavily "grandfathered" as some seem to believe.
The changes in certification basis, and FAR25 amendment status for compliance of most 737 models is here:
https://rgl.faa.gov/Regulatory_and_Guidance_Library/rgMakeModel.nsf/0/179cdacd213801658625832a006b2e37/$FILE/A16WE_Rev_64.pdf

[lomapaseo]

Grandfathering coupled with millions of hours of experience goes a long way in demonstrating compliance with today's standards.

It doesn't mean that you do not have to comply with lessons learned in the basis of subsequent rule changes