PPRuNe Forums

PPRuNe Forums (https://www.pprune.org/)
-   Tech Log (https://www.pprune.org/tech-log-15/)
-   -   New Software Issues Found on the MAX (https://www.pprune.org/tech-log/628965-new-software-issues-found-max.html)

Ian W 21st January 2020 12:37


Originally Posted by Turb (Post 10667942)
I feel sorry for the code-writers who are working on this. I mean the ones actually doing the job, trying to understand and solve the problems of the system they need to change, while the rest of the company and its suppliers, thousands and thousands of people mostly with families and mortgages, just stand around twiddling their thumbs and praying that today is the day the damn thing actually works. Just think what that must feel like for the coders, knowing that all those thousands of other people are kind of peering over their shoulders and willing them to stop messing about and just fix this, right now. And I bet the team of coders is tiny. It has to be tiny. The job can't be done by a huge team, it's not that sort of job. Throwing extra resources at it would be nonsense - in fact it would be counter-productive. If I'm right there's this tiny group of very clever people coming in to work each day, to spend another day struggling to dig the whole Boeing company out of the hole it's in, none of them making the sort of money the Boeing board does, and none of them responsible for the design errors which caused the problem in the first place. So I feel sorry for them.

In some respects its worse than that. Each time they 'fix' the code they then have to test the fix and regression test the system to try to break what they have just built. It's a difficult task telling the team that the system failed one of the final sequence of regression tests.

clearedtocross 21st January 2020 13:25


Originally Posted by n5296s (Post 10667863)
Rubbish. You can do anything in C++ that you can do in C, and a great deal safer. I speak from much experience. People often confuse C++ with more "automagic" languages like Java and Ruby. C++ has no garbage collection, and if you want to manage memory allocation yourself, including doing everything statically, it's easy. In 2020 doing any major new code in C is a sign of insanity. Of course if you already have a legacy of 10 million lines of C code, like my erstwhile employer, you're a bit stuck with it.

Ok, rubbish if you like. But nobody suggested the FCC should be written on a 286 in C in 2020. That would be truly insane. It's questionable if the FCC should have been developped from scratch on a new platform and new soft and comms resources, but here we are in the old hamsterweel territory regarding re-do or update.

The last thing I would like to do is trigger off a debate in this forum about what's the best program language for this and that. However, I would bite my tongue off without the remark that if you do not use all the fancy ++ stuff of C++ then you end up rather close to ... you guessed it.

But thank you very much, Fly Airpt, for the link, very interesting read and the results are no surprise to me.

Luc Lion 21st January 2020 15:10

I believe that the B737 FCC software is written in ADA.
I am not 100% certain of it, but I know that the FCC of the Classic serie (300, 400, 500) was written in ADA, that the FMS of nowadays B737 (NG and MAX) is in ADA, that Collins Aerospace (the FCC subcontractor) works mostly in ADA, and 99% of the B777 ​​​​​code is written in ADA.

Note: I believe that earlier versions of the FCC were written in FORTRAN.

FrequentSLF 21st January 2020 16:00

I would assume that the main issues are the interrupt handling, working with the interrupts on the 286 has been always a pain.
What surprises me is that they did not setup a fully functional rig, they have 400 737 parked, and could have used one as a lab rig for the development of the SW, is that something difficult to do?

MechEngr 21st January 2020 16:13


Originally Posted by FrequentSLF (Post 10668109)
I would assume that the main issues are the interrupt handling, working with the interrupts on the 286 has been always a pain.
What surprises me is that they did not setup a fully functional rig, they have 400 737 parked, and could have used one as a lab rig for the development of the SW, is that something difficult to do?

That would only handle the first 5 seconds or so of the software operation. As soon as any mechanical system was required you need a plane that is ready to be flown, as in fueled and generators and engines running. Otherwise it is stuck like a non-running car turned to the Run position, but not started, and is just an idiot light check. I think the risk of a ready to fly with engines-running plane in a parking lot is more of a problem.

n5296s 21st January 2020 16:58


setup a fully functional rig
Yep. Heck, when I wrote software to run my garden railway, I wrote a simulator for it that emulated everything the actual hardware would do. Nobody was likely to get killed, maybe the cat would have got a nasty surprise, but it just makes everything a great deal easier.

Of course you CAN still be surprised by something the simulator didn't do accurately (just like a flight sim) but you reduce the chances and you pay special attention to the places where that might make a difference.

Luc Lion 21st January 2020 16:59

FrequentSLF please see here for interrupt support in ADA.
https://www.adaic.org/resources/add_...ml/RM-C-3.html

OldnGrounded 21st January 2020 17:07


Originally Posted by MechEngr (Post 10668119)
That would only handle the first 5 seconds or so of the software operation. As soon as any mechanical system was required you need a plane that is ready to be flown, as in fueled and generators and engines running. Otherwise it is stuck like a non-running car turned to the Run position, but not started, and is just an idiot light check. I think the risk of a ready to fly with engines-running plane in a parking lot is more of a problem.

I seriously doubt that Boeing would find it difficult to arrange to test on a ready-to-fly MAX in a safe and convenient location.

MarcK 21st January 2020 17:23

The 286 was introduced 2/1982, and notified as end-of-life (last ship) 3/1999.

fergusd 21st January 2020 17:54


Originally Posted by FrequentSLF (Post 10668109)
I would assume that the main issues are the interrupt handling, working with the interrupts on the 286 has been always a pain.
What surprises me is that they did not setup a fully functional rig, they have 400 737 parked, and could have used one as a lab rig for the development of the SW, is that something difficult to do?

What surprises me is that they don't have a full simulator, which simulates the aircraft to the control system (not the same as the simulator the fly-boys sit in and play). In this day and age it is criminally inept not to do so. 30 years ago we fully simulated trains to the train control systems to ensure that when the control system hit the tracks in a train it worked, including everythng from signalling systems to wheel slip on wet rails, same goes on today in the automotive industry, down to the simulation of the chassis/road/surface/tyre, etc, etc. I have never delivered a control system capable of killing people which was not substantially tested on a full capability simulator whch could inject any fault or situation possible to the software to ensure it dealt with it, however impossible to believe.

A commercial aircraft is not any more complex than a modern fly by wire train . . . probably not even an order of magnitude more complex than a modern car . . .

Surely . . Boeing don't do this on a real plane . . . like people used to do in the stone age . . .

Edited to add : testing as described is only a small part of the testing required, factor the cost of testing being >2-4x the cost of writing the software in the first place . . . and that is conservative . . . which answers the java programmers question of why it's taken so long to get round to testing it on an aircraft . . .

Ian W 21st January 2020 18:03


Originally Posted by MarcK (Post 10668167)
The 286 was introduced 2/1982, and notified as end-of-life (last ship) 3/1999.

Possibly as a commercial PC chip but there are manufacturers making 80286 boards as aviation uses a lot of them. The reason is that in the late 1980's the safety people in aviation were really taken with the idea of 'formal proof' of all the processing in avionics. I can remember briefings back then on how this would remove errors - totally disregarding the fact that hardware manufacturing errors and even software coding errors were even then getting rarer and could be handled. The worst kind of error was as in MCAS a perfectly implemented poor design. Nevertheless, it became a requirement that the hardware had to be formally proved. Then along came multi-core chips with intelligent prefetch and with preemption and multiprocessing... and the 'formal proof' rapidly became an extremely hard Np problem. So 80286's continue to be used as they can be formally proved. And this in itself is a problem as they are 'beasts of very little brain' especially when trying to handle a multiplicity of interrupts.
Is Ancient Silicon The Root Of Boeing’s Problems?
It is about time that the entire area is revisited as the handheld game player the 4 year old has in seat 28G is many times more powerful than the FCCs. All smart phones are way way above the capabilities of the CPU in the FCC. Like communications the time has come to abandon some of these old 'safety' ideas and move to a more commercial approach as is used in other areas. Size, Weight and Power are not an issue these days.

Ian W 21st January 2020 18:12


Originally Posted by fergusd (Post 10668184)
What surprises me is that they don't have a full simulator, which simulates the aircraft to the control system (not the same as the simulator the fly-boys sit in and play). In this day and age it is criminally inept not to do so. 30 years ago we fully simulated trains to the train control systems to ensure that when the control system hit the tracks in a train it worked, including everythng from signalling systems to wheel slip on wet rails, same goes on today in the automotive industry, down to the simulation of the chassis/road/surface/tyre, etc, etc. I have never delivered a control system capable of killing people which was not substantially tested on a full capability simulator whch could inject any fault or situation possible to the software to ensure it dealt with it, however impossible to believe.

A commercial aircraft is not any more complex than a modern fly by wire train . . . probably not even an order of magnitude more complex than a modern car . . .

Surely . . Boeing don't do this on a real plane . . . like people used to do in the stone age . . .

Edited to add : testing as described is only a small part of the testing required, factor the cost of testing being >2-4x the cost of writing the software in the first place . . . and that is conservative . . . which answers the java programmers question of why it's taken so long to get round to testing it on an aircraft . . .

Like this system you mean?

MechEngr 21st January 2020 18:14


Originally Posted by OldnGrounded (Post 10668147)
I seriously doubt that Boeing would find it difficult to arrange to test on a ready-to-fly MAX in a safe and convenient location.

Which is what they did. But it isn't something that seems reasonably arranged for the multiple compilations each day that might be done while working with simulator testing.

MarcK 21st January 2020 18:27


Originally Posted by Ian W (Post 10668196)

Where the simulation computers are 1990's vintage?

Imagegear 21st January 2020 18:31


So 80286's continue to be used as they can be formally proved
….and therein I suspect lies the problem, being "formally proved" may not necessarily mean "Failsafe". I would like to think that error reporting and recovery rapidly returns the system (hardware and software) to a state in which the integrity of the platform is not compromised.

What of "formally proved" now. How much fix testing and regression testing is required to regain the "formally proved" status. It is not sufficient to prove just the hardware and software since a layer of microcode exists as the sandwich filling within the ALU and PROM/ low order RAM portion of the 286 which may not be tolerating some minor timing discrepancy on either side of the interface to cause the failure. Is the 286 perhaps being driven to it's critical limits?

IG

MechEngr 21st January 2020 18:37


Originally Posted by MarcK (Post 10668167)
The 286 was introduced 2/1982, and notified as end-of-life (last ship) 3/1999.

That would be the Intel 80286, I expect. However Intel would certainly license the core to embedded systems developers for production at rates and for markets in which Intel could no longer afford to maintain fabrication. For example: Renasas 80C286 went EOL in 2017. Would not be surprised if someone bought 5000 of them for the shelf.

I would expect the next generation processor to be an FPGA rather than a microprocessor. I see some links exploring ADA on FPGAs, but it seems to be programming the FPGA to act like a microprocessor.

fergusd 21st January 2020 19:44


Originally Posted by Ian W (Post 10668196)

Nope . . .

Fd

crankyanker 21st January 2020 21:48


Originally Posted by FrequentSLF (Post 10668109)
I would assume that the main issues are the interrupt handling, working with the interrupts on the 286 has been always a pain.

In what sense? A lot of what made the 286 unpopular was its poor backwards compatibility with DOS which was designed for real mode, that ought not be an issue on a system that's hopefully not trying to run DOS.


Originally Posted by MechEngr (Post 10668220)
That would be the Intel 80286, I expect. However Intel would certainly license the core to embedded systems developers for production at rates and for markets in which Intel could no longer afford to maintain fabrication. For example: Renasas 80C286 went EOL in 2017. Would not be surprised if someone bought 5000 of them for the shelf.

Boeing almost certainly uses a radiation hardened processor (MIL-STD-883) and not a standard desktop CPU.


Originally Posted by MechEngr (Post 10668220)
I would expect the next generation processor to be an FPGA rather than a microprocessor. I see some links exploring ADA on FPGAs, but it seems to be programming the FPGA to act like a microprocessor.


I'd be surprised here too. These days you can get haardened RISC and PPC based processors that would provide a ton more power and flexibility than a 286-based design. There's also probably a push to use lower cost off-the-shelf components.

clearedtocross 22nd January 2020 08:42

Now the poor software guys working on the max get more time. The first move of B's new management in the right direction to get the max flying again. I hope they clear up some other issues as well until summer (2020 !).
And thank you all for your valuable contributions and explanations! Has been very enlightening and is good to see that pprune also hosts know-how complementary to stick and rudder.

RomeoTangoFoxtrotMike 22nd January 2020 09:30


Originally Posted by Ian W (Post 10667519)
The Boeing engineers did not have the luxury of starting from a clean sheet and "getting it right by design". They had stiffware that has been operational for years and had to modify the code so that the FCCs operated in a different way without changing anything that was not essential to change for the task at hand and without breaking any current functions. Maintenance programming especially of embedded code and modifying the code so it does things differently without affecting anything else is nothing like simple code writing. It is possible that some very basic timing issue made the live aircraft slightly different to the avionics test bench. This is the reason regression tests are run when the new code is ported to and implemented in the aircraft - and the tests found an issue - that is what the tests are for.

If it helps, think of it like this...



All times are GMT. The time now is 06:16.


Copyright © 2026 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.