bsieker

Well, that is the $500m question, is it not?

Could it perhaps be the reason why there has been no further word from the AAIB/Boeing/Rolls Royce? They haven't figured it out yet.

FADEC is a concept, a "philosophy", if you will. It stands for Full Authority Digital Engine Control, and means a computer controls the fuel metering valve (the "throttle"), gives the desired thrust, and protects the engine. There is no way to manually override the fuel valve position. EEC is the actual computer that does the work. Sometimes the computer as such is also referred to as "The FADEC".

Someone in the know please corerct me but I am pretty certain that the 747-400 also has FADEC-controlled engines, as all current airliners have. Hydromechanical control got out of fashion a long time ago. Concorde was the first airliner with full-authority electronic engine control, although it was not digital.

Old school or not, it wouldn't hurt to check the facts. Btw, cables can break, and need constant re-adjustment due to lengthening, and on multi-engine aircraft all enginess are never adjusted equally leading to thrust-lever staggering in normal operation to get them all to the same power output.

Right. Never let the facts get in the way of a good rant.

Why do you feel entitled to make comments about stuff you obviously have no clue about?

And just so you know: The AAIB stated that the Fuel Metering Valve (the "throttle", that ultimately controls the amount of fuel going to the spray nozzles) was fully open. Just as it would have been with a mechanically controlled engine.

Good luck finding a passenger aircraft in service with a modern western carrier that has non-FADEC engine control.

Contrary to popular belief, fly-by-wire has nothing to do with engine control. Even most non-FBW airliners have FADEC these days.


Bernd

Rightbase

As I recall, the AAIB did report - or imply - a problem with the fuel. It was not getting to the fire.

The 'throttles' were open, the engines were sucking, the pumps in the tanks were pumping, but the noise stayed quiet....

on both sides.....

not quite simultaneously.

trickii

There is a silence in the report about software causes or other transitory causes.Presumably this has been covered and dismissed?

pacplyer

bsieker,

Who said anything about fly-by-wire? The subject of my post was thrust lever control; nothing else. What I am talking about is a well known FADEC phenomenon called rollback. You don't seem to be familiar with it or else you wouldn't swallow the fuel valve open preliminary investigation comment so gullibly.

Whether you know it or not in FADEC, code algorithms are constantly checking engine parameters and "voting" when, or if, to schedule twin spool acceleration. Since they won't allow overboost, and are concerned about thermodynamic engine life they sometimes reject a flight deck command in favor of a slower acceleration schedule. (Most of the time this is a good thing.) Sometimes they reject everything altogether and command idle logic instead (which can't happen with old 747-100's and 200's with steel cable to hydomechanical control.) Thus my position that the old design was safer from a "fail safe" consideration at spool up. A thrust lever cable breaking is extremely rare. In over twenty years of flying them I've never met anybody who's had it happen. But I have met pilots who have had unexplained rollbacks using FADEC.

bsieker, A snapshot (data point) of the sun in the sky does not mean it is daylight 24 hours a day. Fuel valve positions can be momentary recorded in the middle of transitory problems; so this report of everything being fine with the fuel valve means nothing to me.

Here's what one of the so-called "experts" who writes in aviation rags has to say:

BA038 crash landing caused by a software glitch? « David Cenciotti’s weblog - the most visited Italian Aviation Blog

bsieker said:
Seperately, who cares about staggered throttles? It doesn't hurt your flying ability to be a knob or so out of rig. But it does hurt your flying ability to have microsoft windows type crap software between you and the spray nozzles. No way to override it in time. Bad Bad design, imho. But that's how were doing it now.....

BTW, I was just being melodramatic to underscore a point. If I only got on non-FADEC controlled aircraft, of course I'd never get anywhere.

Regards

Carnage Matey!

In X million cycles how many times has this happened?

Did not an Evergreen 747 divert into LHR in the last year or two with unresponsive engines?


Your quote regarding from the press says:

Well there's an enormous assumption if ever there was one!

pacplyer

Carnage,

Agree. That's why I said "so-called expert" (he calls himself that in his "about me" section on his site.)


Found this little conversation about BA038 on a maintenance forum:
Heathrow B777 accident - Aircraft Maintenance Forums

I don't think it's one in a million odds anymore. I think dozens of fadec issues are well documented throughout the industry regardless of make. We were aware of rollbacks on both P&W and GE A310's when I flew that airplane back in the 90's. The solution was nearly always a new software "load."

There must be a FADEC code standard that is industry-wide prior to a powerplant being certified on a particular model. Where's Machaca? He's good at finding a needle in a haystack....

pac

pacplyer

JMHO's here.

I know the GE is a different engine, but both RR fadecs and GE fadec software versions go through the same flakey FAA appoval process.

Air Safety Week:
Lots more here at the source; worth reading:

Air Safety Week :: The Unthrustworthy 777

pac

Ptkay

Software debugging is tedious and difficult process.

I am a programmer myself and have encountered several cases
of debugging mysteries, that took me weeks to solve.

My uncle worked many years for Litton, GE and NASA
as safety and quality control specialist.
(he is now retired, but still called in from time to time as a consultant)
He worked on Polaris, Delta but also Space Shuttle projects.

I asked him, is it true, that the core programme to run Space Shuttle
is still 128 kB big (no error - kilobytes).

His answer was affirmative.

He confirmed, that they checked by modelling and theoretical
calculations, that it is impossible to completely debug
(test the programme reaction to every and each input data combination)
by programme codes bigger than 128 kB.

The killer in this case is, that no redundancy can help in case when
on every of 5 computers sits the same software with the same bug.

Hence Space Shuttle files with a software smaller than that of
your modern washing machine.

I don't know what is the size of the RR FADEC system, but if it's
bigger than that of Space Shuttle, it will never be properly debugged.

bsieker

pacplayer,

no offence intended, maybe my words were a bit harsh.

With cooler head:

I assumed you were talking about fly-by-wire since you specifically mentioned Airbus and B777, which are the most predominant and widely-used fly-by-wire airliners. Apologies.

No, that's not really how it works. There are feedback-control-loops, and additional safety checks, but when, e. g. requesting TOGA thrust in flight, which almost always indicates some sort of emergency, spool up is as fast as possible while still avoiding stalls and flameouts. Engine life protection does not usually take precedence over thrust demand. Case in point: engine vibration is measured and can be displayed in the cockpit. But this, although it may seriously damage the engine, is only advisory. Full thrust even on an excessively vibrating engine is available (if mechanically possible, but physics will decide that, not the FADEC).

(And just to be nitpicking again, the Rolls-Royce Trents are triple-spool, not twin-spool.)

Pilots with fully mechanical control (which has been very rare for a long time, even before FADEC) cannot always, especially in a high-workload situation such as a go-around, do the optimal spool-up.

Voting, if at all, only happens between two or more identical (or diverse, depending on the development model) circuits, one of which alone is sufficient to control the engine. Often there is no voting, but one computer (often called a "channel") internally checks its computations with a separate processor, and if both disagree, the entire channel declares itself failed, and the fallback channel takes over.

They do not allow anything beyond TOGA thrust. But continued TOGA thrust, while not recommended, is possible, even if it significantly shortens engine life. I don't know about the 777, but on the A320, TOGA thrust is allowed for a maximum of 5 minutes (or 10 minutes with one engine out). But this limit is not enforced by the FADEC.
Yes, but the reason for that is not only increased engine life, but also avoiding stalls and flameouts.

No.

Even if the readings from the thrust levers fail completely (or are inconsistent, i. e. both redundant sensors read differently), FADECs do not automatically command idle thrust. A lot of throught has gone into the logic what thrust to set in case of such failures. Depending on the flight phase, either idle, MCT or the last commanded thrust is set.


Of course it can happen. The cable could just break. Rare, I know, but so are EEC/FADEC malfunctions.

Yes, I am aware that there may be issues with the digital electronics involved. You are mistaken if you think I have blind faith in them.

However, I assume FMV position is recorded once every second, or at worst every 4 seconds. It seems extremely unlikely that it was oscillating at .25 Hz, and always at the point of sampling for the DFDR, was fully open, and almost closed in between. There is still the remote chance that the recording as such was faulty. Since we have no access to the EEC software, we cannot say.

As to the relative numbers of engine control cables breaking, as opposed to FADEC problems: Just consider the sheer number of FADEC-engined aircraft today, and the dwindling numbers or airframes, and low number of flight hours, for mechanically controlled engines.

Even if both were equally safe, we would expect to see more problems with the electronic variant than the mechanical ones.

On the other hand, modern digitally controlled FADEC and autothrust systems offer increased stall protection, e. g. Airbus' "ALPHA-Floor-Protection".

(my emphasis)

I know you're trying to make a point, but the development process for business software, and for safety-critical software for embedded systems cannot be compared. These are totally different things.

Not a single thing you may think you know about "Microsoft Windows" type software is applicable to embedded systems development.

I say again, this does not mean that I believe there are no problems with FADECs; I know that there are. But there are methods known in the industry to make extremely reliable software. Not all of those techniques seem to be applied with sufficient rigour all the time.

Try searching for "SCADE", "Lustre" and "SPARK" to get an idea of what embedded software development is like. Seriously no Windows-Type crap there.

----

Ptkay,

While it is true that very high software reliability cannot be shown by testing (i. e. subjecting the software to varying input combinations and checking its output), there are methods (see above) to develop very-high reliability software.

The limit is not the code size as such, but the inherent discontinuity of digital computers, i. e. even the smallest possible input variations can create a completely different output.

One name for development methods that do not rely on testing to ensure proper functioning is "Correct by Construction". Formal mathematical methods are applied to ensure that the software does what it is supposed to do.


Bernd

TechnoFreak

Ladies and Gents,

Over the last few months I have read most of this thread, perhaps I should get out more.

Some contributors seem to ignore the facts in order to support wild conspiracy theories.

My understanding of the state of the investigation is as follows:

The fuel valves were fully open but very little fuel was being delivers to the engines.
The High Pressure fuel pumps showed evidence of cavitation damage on the input side.

When cavitation occurs, the pump will not deliver the required output pressure and therefore not enough fuel will be delivered to engines.

Cavitation occurs when either the fuel is less dense that it should be or there is not enough of it reaching the pumps. In this case it seems that the latter is most likely.

Why is the investigation taking so long?

All they need to do is to put the 777 wings, tanks and fuel system in to a big fridge and run a temperature simulation of the entire flight. Then repeat this for an infinite number of different combinations of fuel contaminants. Can’t see the problem myself, (sarcasm).

For what is worth, I think that this accident was caused by debris/contaminants in the fuel together with the abnormal low temperatures experienced during the flight. Due to the complexity of modelling the infinite number of variables, we may never get a final answer.

bsieker

TechnoFreak,

Yes.

One minor correction: The cavitation damage was on the outlet side of the HP pump. Which is to be expected, as, although low pressure in the inlet makes the bubbles form, the damage occurs when the cavities implode violently after the pressure has risen again.

Right, no problem there, just test all the possible combinations in a big fridge. Could be done in one afternoon.


Bernd

Leodis737

Quote:

Due to the complexity of modelling the infinite number of variables, we may never get a final answer.

Nah, they WILL get there.

The fuel: they have big samples of it - can test as much as they need to.
The environment: they have air temp data from the whole BA38 flight and previous flights if necessary.
The fuel system: they have the real one to test with, plus lots of copies.

They will fly test flights if necessary (remember the ATR72 icing investigation), and they will keep plugging away until they get there. They won't give up on one this important.

pacplyer

Bernd,

No offense taken. My bombastic happy hour posts were submitted for entertainment as much as anything to stimulate thought.

Yes they are triple-spooled. (Old lingo hang up.)

Bernies post:
(my emphasis)

You keep talking about what fadec is designed to do from the factory. That's not the context of the conversation I was attempting to delve into. A rollback is a well-documented software created anomaly, as I understand it, in which, for whatever reason the fadec has either miscalculated the thrust solution (thinks it's at high power and stays at idle) or has had a dual failure of both channels and is presumably trying to reboot itself at idle (since both channels are susceptible to the same software bug.)

Do you recall the Air Florida accident? That was a thrust miscalculation by the crew since the pt2 was frozen and the pt7 continued to input data to the epr system resulting in a falsely high EPR reading. The captain thought they had set max power when reality it was actually nothing close to the target.

I dare say, in some rare cases, the FADEC is not much smarter when freezing conditions are present and fadec signal sensor probes get moisture freezing on them. (If lessons learned from the GE rollbacks prove to be instructive or simular in the RR trent engs.)

Bernies post:
Fadec gets it's sensor data and responds up to 70 times a second. The code is too complex and it behaves differently every time when it's working right as to make that realm unknowable. Recreation simulations are not always revealing even for design engineers or accident investigators. This is why I argue for a direct cable back up of some sort.

Throw in moisture to the FADEC sensors and who knows what weird power solutions have been calculated? And what are you are really reading with that DFDR anyway? Probably not actual valve position, but rather a software command to an actuator thinking it has called for rated power. (just speculation on my part.)

Do we know for certain that the DFDR records the actual metering valve position? If so, the Next question is how does it record that? Prox sensor? rheostat? actuator drive position? Fuel flow?

I'm not ready to buy into a correct fcu fuel metering valve "positioned open" just yet. I'll agree that it was logged as being commanded open once every four seconds. I want to hear more from engineers and programers first.

[The microsoft slam was just intended as over-the-top sarcasm; but thank God we don't fly around on windows.... We'd crash twice a day!]

The above is, as all my post are: only my opinion only.

pac

Wikipedia reference:

bsieker

Yes, that's what I was trying to say with my remarks that high-reliability in software cannot be shown by testing. The argument can be made for mechanical backup, but also for better software development methods (Correct by Construction, instead of "Validation by Testing", which is proven to be insufficient.)

Good software will have a lot of sanity-checks, i. e. will check its input parameters for consistency. Very simple checks are correspondance between EPR vs. N1/N2 (and even N3 in this case), or commanded/actual FMV position vs. actual fuel flow, or fuel flow and N1 vs. EGT, ...

There is another indication that the reduced fuel flow was not the consequence of the FMV not opening, but was present before, and also that the actual FMV position is measured independently of the EEC's commands:

The EEC recorded in its NVRAM that reduced fuel flow had been detected. Probably this would only be recorded if it was unexpectedly low. In consequence, the FMV was commanded to open more and more, up to fully open. (cf. AAIB SB 2008-03, p. 2):

No, we cannot. And these are fair questions that we all have asked. To highlight the difficulty, the A320, with which I am more familiar, has 8 thrust lever angle sensors for each thrust lever. Two for the FADEC (high precision, contactless angle resolvers), and a pair of potentiometers for each of the three SECs for spoiler/autobrake conditions. I do not know which of these is used for the DFDR data. I assume it's one of the resolvers, but I just don't know. Similar question will be asked (by the AAIB, among others) for the B777's Fuel Metering Valve position.

I seem to recall that the actual position is measured and recorded independently of the EEC (FADEC), and not just the commanded position, since the FMV is such a crucial piece.

The developers who know for certain probably won't be allowed to tell us.

Yes, I didn't think you'd mean that literally.

The really ugly part of this accident really is that every conceivable scenario is, a priori, "extremely unlikely", so if anyone can figure it out, it's the guys with all the data (AAIB, NTSB, RR, Boeing, QinetiQ, ...).


Bernd

flash8

What do you think should happen in the spilt second that the PIC decides max power is needed? A computer vote? A cyber committee? A democracy?


Actually the n-version engineered software does vote if memory serves me right. A fault that resides through n-versions is somewhat unusual unless it directly relates to the software specification. I recall reading the Z notational definition of the 744 FADEC Software at University. Most impressed and find it hard to believe that the Software is actually at fault.

GearDown&Locked

Bsieker
...until the air to fuel mix parameters were out of range for a correct and effective burn I'd say.

Q1: how exactly is the fuel flow measured? mass? capacitance? delta P?
Q2: would a density difference for this type of special winter fuel influence the real fuel flow metering?

GD&L

TopBunk

A BA engineering source known to me (being vague deliberately) today has reported that the current gossip is that a valve in the fuel pump is currently a major suspect. It would appear that the fuel flow around it causes turbulence and that they (CAA/RR/FAA) have been able to reproduce icing and fuel flow restrictions under test.

The fuel pumps in the 777 are not engine specific and the architecture differs from that on other aircraft, so whilst the authorities may never be able to say with certainty what caused the crash, the current thinking is the probability is fuel and fuel pump related.

If so, then there could be some redesign of the fuel pump valves in the near future.

glad rag

If confirmed this is excellent news indeed!!

stadedelafougere

@ TopBunk:
What pump would be incriminated? The feeding pump in the fuel tanks or the high pressure pump in the engine?

barit1

Per TopBunk:
I infer from this that the LP pumps (aircraft side) are implicated, and not the HP pumps (which are certainly engine-specific).

However, it may also be that the specific R-R HP pump is somehow more susceptible, compared to Brand X or Brand Y, to whatever LP-side shortfall may exist. Time will time I hope.