VnV2178B

As we seem to have drifted back to FADEC software and its construction may I add something which will probably scare those of you who are already concerned:

The Trent FADECs run the same software, developed by the same team in both lanes of the FADEC, these are also hardware identical.

However, the design and construction, code, test* etc. are so different from the commercial world that I do have confidence in the correctness of function. There are so many cross-checks - sanity checking is a good term - that I doubt that any of the scenarios painted so far could happen.

But, as I was on that team, I would say that wouldn't I???

Last time I posted on this thead I got a dire warning, so unfortunately I can't add much more.

Safe flying

VnV...

*check my moniker.

pacplyer

Finally,

The truth is fleshed out. Thank you immensely VnV2178B. That was what I was hoping would emerge by posting so many inflammatory posts.

A true pillar of software honesty has emerged ladies and gentlemen. And I will not soon forget it.

Please, please,

The direction of my inquiry, dozy, and others, is not to implicate aerospace software vs. the pilot input as some kind of culprit. My intent only, is to increase aviation safety; as now I find myself in the back, with knowledge and experience in commanding both Boeing and Airbus products (and a rudimentary degree in Computer Science.)

That I hate automation and aviation software? Nothing could be further from the truth. Software flying is never going to go away and I know it. That is not what I am arguing for. The days of John Wayne are over and I know it (although those days were better for professional pilots.)

What is important now, is that we allow flight crews to stay proficient at John Wayne skills for the day that the software hit's an endless loop and gives up.

I really hope that you my friends in programing can discern the subtle difference that I am advocating. Obviously, I have failed to convey that intent.

Fraternally,

pacific plyer

(the above is all and only: just my opinions only.)

DozyWannabe

OK, but comparing it with the A320 incidents is now no longer a valid position, as the A320 software *did not* let go.

VnV, I'd *love* to know the sanity checks you guys put this code through, because I'd still be very concerned there was no fallback or cross-check in day-to-day operation.

VnV2178B

DW,

I would post, but, as I noted above, I got a dire warning last time about revealing stuff not already in the public domain.

However, as a guide, the inputs are all duplicated and cross-checked, there are performance models of the engine and the FMV and the expected outputs are compared with the actual outputs, there are reversion modes should there be a loss of a speed, temperature or pressure input (the GE scenario of freezing probes is unlikely to fool the R-R FADEC), the FADEC hardware is continually checked for correect function etc, etc.

The whole lot is validated to be what is wanted and verified for function, starting with the lowest assembler and finishing with a full-up systems rig test.

I have faith in the product, I have flown on 777s since the incident without qualms!

VnV...

infrequentflyer789

This is (roughly) the main hypothesis behind NVP (n-version programming). Knight & Leveson famously claimed to have disproved this experimentally http://sunnyday.mit.edu/critics.pdf. Specifically, they claimed that errors were correlated between independently developed versions of the software. This might be counterintuitive, but it wouldn't be the first counterintuitive result in the history of science.

Also worth noting that Boeing dropped NVP when developing the 777 flight control software. They engaged a single contractor for triplex development (three separate teams with "chinese walls" between), but changed to one team part way through development, apparently because the teams were asking such similar questions about the spec it was felt independence was compromised anyway ("common culture" perhaps...).

All of which is not relevant to the FADEC software which is a different beast and may well have used a different (but still safety-critical) development methodology.


With regards to this incident, based on the information published so far all the software appears to have functioned correctly, which means we are looking at a different cause (although I take the point that there could have been something going on between sampling intervals or a sensor failure).

DozyWannabe

Hmm - I suspect the 777 processes were still under wraps when I was at Uni then, as NVP was still considered as a very good thing. Knight and Leveson's criticisms were mentioned, but very much in the context of the jury being out.

bsieker

DozyWannabe, Pacplyer, VnV, others, ...

This is about the B777's FBW system (more specifically, the PFCs).

The approach of having three separate coding teams, isolated from each other, was initially attempted, but eventually rejected. Iin his paper "Design Considerations in Boeing 777 Fly-By-Wire Computers" Y. C. (Bob) Yeh wrote:

So this sounds like diversity is theoretically a good idea, but hasn't been proven to be beneficial in practice.

Coding diversity will not eliminate the most common form of errors, which are requirements errors

I know that the A320's most important flight control computers, the ELACs, each contain one Motorola 68000 and one Intel 80186 processor, which run the same algorithms, but I do not know if their software was developed by isolated teams. There are 2 redundant ELACs, and if they both fail, there are 2 SECs, which also provide pitch and roll control, albeit in a degraded mode (alternate or direct law.)

I do not know about hardware/software redundancy within each FADEC channel, neither for Trent nor for CFM56.


Bernd

PJ2

Thanks for the intervention and returning the thread, Bernd. The Habsheim item comes up anytime Airbus automation and software is discussed - sorry for the thread-drift!

VnV2178B

Bernd,

I agree that requirement definition is a difficult task to get right. My experiences of systems engineering is that they speak a sufficiently different dialect for there to be a considerable margin for misinterpretation.

Having spent some time with Airbus I found their approach refreshing in as much as we had meetings at which all the stakeholders were supposed to agree the wording, implementation and testing of every requirement. This meant that problems of interpretation could be caught early in the process. I hope they still do this as I, for one, found it useful.
I would expect a badly worded requirement to be subject to the same problems from every diverse team that encountered it as most implementers would have come through the same education process.

VnV...

DozyWannabe

VnV - I was told that the stakeholder meeting was still a big factor in AI's development process in 2001 - no idea about now, but I can't think of a reason they'd abandon it.

VnV2178B

DW,

my Airbus experience was later than 2001: so it was still being used 2003/4ish.

Perhaps some current AUK/AI person could enlighten us, and perhaps a Boeing person could do the same on their elicitation process.

VnV...

tanimbar

VnV2178B,
I like asking daft questions every now and then, even when I don't expect an answer, but can you say whether the R-R FADEC software/hardware has been investigated in relation to the BA038 incident?

The spirit of this enquiry is: If you don't ask, you don't get.

Regards, Tanimbar

VnV2178B

Tanimbar,

I actually don't know as I am not involved anymore.

I assume (dangerous!) that R-R has been looking into all aspects of the systems, including the FADEC software.

From what I have read and memory I would not point the finger at the software myself as all the reports state that it functioned as designed, demanding more fuel when the reduced flow was detected.

I won't post more here - I too think this discussion is not news and should be tech log, I only wanted to clarify the software process.


VnV...

Old Engineer

In the comments section of The Register May 13, a Heathrow mechanic for 777s said large transports landing from long flights at altitude in cold conditions were showing up with booster pumps with frozen intakes and that check valves held open with solid ice were also seen. He states he personally changed some of these pumps. I took booster pump to mean the LP pumps in the wing tanks. This may be a reliable report, because it seemed to fit with the physical test program AAIB stated in their May 12 supplemental report (3 pages). Here is the link:

From The Register, London, publ 5-13-2008, URL = Heathrow 777 crash: Siberian cold to blame? | The Register

His comments are easy to find among the few on the website, at the AAIB story.

My own thoughts are that something like closing the throttles to flight idle at some reasonably high altitude in the landing regime would increase the LP pump discharge pressure. In turn, this would decrease pressure at the impeller area of a centrifugal pump, as this pump is. I know this is very counter-intuitive (so pump engineers tend to be specialists). Under the right adverse conditions, a decrease in pressure within the fluid fuel column could cause ice crystals to precipitate out of solution or entrainment, where before the water content may have been causing no problem.

OE

Green-dot

At low altitude, even if the pumps would have encountered such conditions as you describe, suction feed bypass valves in the engine feed system would have opened to feed the engines. Those are check valves and not sensitive to the conditions the pump impellers are subjected to according to your explanation. No ice crystals would precipitate out of solution at those suction feed bypass valves with the fuel quality being within specs as tested by the AAIB. A scenario whereby both the boost pumps and the suction feed bypass valves would have been blocked by ice therefore seems very remote. This could perhaps have occurred with a much higher water content in the fuel (way outside the required specifications) than was actually sampled from the fuel in G-YMMM's tanks.

Just my thoughts, something other than ice in the fuel must have (partially) restricted fuel to the engines to less than required. It may have been a "manifold" of circumstances (holes in the swiss cheese) interacting in the same slice of time that made it so. Something not recorded.


Regards,
Green-dot

Old Engineer

Thanks for reminding me of the alternate fuel route. I agree with what you say in regard to the suction feed bypass valves.

I don't think that this (alleged, but assuming it did occur to some A/C) LP pump icing, particularly accumulation of ice in the area of the LP discharge check valve, is at all likely without some severe restriction of fuel flow rate somewhere along the normal path of fuel flow. Otherwise, such ice crystals as might form in the low-pressure region of the pump would not have time to grow beyond very small size and quantity before being swept into the higher pressure region of the discharge, which would stop their growth.

So yes, it well may be a swiss-cheese situation. It's certainly possible that ice, if any, could have been just an effect of a more primary chain of events.

OE

777fly

Green Dot:

I repeat my theory here, since you mention the suction bypass system. Boeing acknowledge that gas or air can be trapped in the suction line and cause flame out or thrust reduction under suction conditions.. If some unusual circumstance ( excess gas in the fuel trapped under pressure in the line or LP pump icing reducing manifold pressure) caused the suction line NRVs to open, enough gas could be introduced, almost simultaneously, into both sides of the fuel supply manifold and starve both engines of fuel. There would be no evidence of that after the event.

Green-dot

777fly: I agree, a scenario such as you describe could be amongst the plausible possibilities if both left and right engine feed systems had low manifold pressure due to ice contaminated boost pumps (all 4 pumps in the main tanks) and enough vapor trapped near the suction feed bypass valves. Or, if not vapor, leaking connections in both left and right engine feed manifolds, which would reduce suction feed effectiveness with boost pumps failing to deliver. On the other hand, if boost pump pressure dropped due to ice in the pumps, the crew would have been alerted (pressure lights in the pump switches on the overhead panel and the master caution light (including aural alert) would most likely have been presented). No such alerts have been reported by the AAIB. So even if there was no evidence of it after the event, there should have been evidence of it just prior to- and while the event took place, in the form of alerts mentioned above. Green-dot

CuriositySnr

Has there been an accident report or are they still investigating? Can anyone fill me?

Re-Heat

Still investigating - see www.aaib.dft.gov.uk