Go Back  PPRuNe Forums > Flight Deck Forums > Tech Log
Reload this Page >

New Software Issues Found on the MAX

Tech Log The very best in practical technical discussion on the web

New Software Issues Found on the MAX

Old 21st Jan 2020, 12:37
  #41 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,346
Originally Posted by Turb View Post
I feel sorry for the code-writers who are working on this. I mean the ones actually doing the job, trying to understand and solve the problems of the system they need to change, while the rest of the company and its suppliers, thousands and thousands of people mostly with families and mortgages, just stand around twiddling their thumbs and praying that today is the day the damn thing actually works. Just think what that must feel like for the coders, knowing that all those thousands of other people are kind of peering over their shoulders and willing them to stop messing about and just fix this, right now. And I bet the team of coders is tiny. It has to be tiny. The job can't be done by a huge team, it's not that sort of job. Throwing extra resources at it would be nonsense - in fact it would be counter-productive. If I'm right there's this tiny group of very clever people coming in to work each day, to spend another day struggling to dig the whole Boeing company out of the hole it's in, none of them making the sort of money the Boeing board does, and none of them responsible for the design errors which caused the problem in the first place. So I feel sorry for them.
In some respects its worse than that. Each time they 'fix' the code they then have to test the fix and regression test the system to try to break what they have just built. It's a difficult task telling the team that the system failed one of the final sequence of regression tests.
Ian W is offline  
Old 21st Jan 2020, 13:25
  #42 (permalink)  
 
Join Date: Jul 2007
Location: Switzerland
Age: 74
Posts: 96
Originally Posted by n5296s View Post
Rubbish. You can do anything in C++ that you can do in C, and a great deal safer. I speak from much experience. People often confuse C++ with more "automagic" languages like Java and Ruby. C++ has no garbage collection, and if you want to manage memory allocation yourself, including doing everything statically, it's easy. In 2020 doing any major new code in C is a sign of insanity. Of course if you already have a legacy of 10 million lines of C code, like my erstwhile employer, you're a bit stuck with it.
Ok, rubbish if you like. But nobody suggested the FCC should be written on a 286 in C in 2020. That would be truly insane. It's questionable if the FCC should have been developped from scratch on a new platform and new soft and comms resources, but here we are in the old hamsterweel territory regarding re-do or update.

The last thing I would like to do is trigger off a debate in this forum about what's the best program language for this and that. However, I would bite my tongue off without the remark that if you do not use all the fancy ++ stuff of C++ then you end up rather close to ... you guessed it.

But thank you very much, Fly Airpt, for the link, very interesting read and the results are no surprise to me.
clearedtocross is offline  
Old 21st Jan 2020, 15:10
  #43 (permalink)  
 
Join Date: Mar 2006
Location: Vance, Belgium
Age: 57
Posts: 181
I believe that the B737 FCC software is written in ADA.
I am not 100% certain of it, but I know that the FCC of the Classic serie (300, 400, 500) was written in ADA, that the FMS of nowadays B737 (NG and MAX) is in ADA, that Collins Aerospace (the FCC subcontractor) works mostly in ADA, and 99% of the B777 ​​​​​code is written in ADA.

Note: I believe that earlier versions of the FCC were written in FORTRAN.

Last edited by Luc Lion; 21st Jan 2020 at 19:28.
Luc Lion is offline  
Old 21st Jan 2020, 16:00
  #44 (permalink)  
 
Join Date: Aug 2007
Location: Alabama
Age: 54
Posts: 366
I would assume that the main issues are the interrupt handling, working with the interrupts on the 286 has been always a pain.
What surprises me is that they did not setup a fully functional rig, they have 400 737 parked, and could have used one as a lab rig for the development of the SW, is that something difficult to do?
FrequentSLF is offline  
Old 21st Jan 2020, 16:13
  #45 (permalink)  
 
Join Date: Oct 2019
Location: USA
Posts: 133
Originally Posted by FrequentSLF View Post
I would assume that the main issues are the interrupt handling, working with the interrupts on the 286 has been always a pain.
What surprises me is that they did not setup a fully functional rig, they have 400 737 parked, and could have used one as a lab rig for the development of the SW, is that something difficult to do?
That would only handle the first 5 seconds or so of the software operation. As soon as any mechanical system was required you need a plane that is ready to be flown, as in fueled and generators and engines running. Otherwise it is stuck like a non-running car turned to the Run position, but not started, and is just an idiot light check. I think the risk of a ready to fly with engines-running plane in a parking lot is more of a problem.
MechEngr is offline  
Old 21st Jan 2020, 16:58
  #46 (permalink)  
 
Join Date: Jun 2003
Location: PUDBY
Posts: 723
setup a fully functional rig
Yep. Heck, when I wrote software to run my garden railway, I wrote a simulator for it that emulated everything the actual hardware would do. Nobody was likely to get killed, maybe the cat would have got a nasty surprise, but it just makes everything a great deal easier.

Of course you CAN still be surprised by something the simulator didn't do accurately (just like a flight sim) but you reduce the chances and you pay special attention to the places where that might make a difference.
n5296s is offline  
Old 21st Jan 2020, 16:59
  #47 (permalink)  
 
Join Date: Mar 2006
Location: Vance, Belgium
Age: 57
Posts: 181
FrequentSLF please see here for interrupt support in ADA.
https://www.adaic.org/resources/add_...ml/RM-C-3.html
Luc Lion is offline  
Old 21st Jan 2020, 17:07
  #48 (permalink)  
 
Join Date: Apr 2015
Location: Under the radar, over the rainbow
Posts: 707
Originally Posted by MechEngr View Post
That would only handle the first 5 seconds or so of the software operation. As soon as any mechanical system was required you need a plane that is ready to be flown, as in fueled and generators and engines running. Otherwise it is stuck like a non-running car turned to the Run position, but not started, and is just an idiot light check. I think the risk of a ready to fly with engines-running plane in a parking lot is more of a problem.
I seriously doubt that Boeing would find it difficult to arrange to test on a ready-to-fly MAX in a safe and convenient location.
OldnGrounded is offline  
Old 21st Jan 2020, 17:23
  #49 (permalink)  
 
Join Date: Oct 2004
Location: California
Posts: 283
The 286 was introduced 2/1982, and notified as end-of-life (last ship) 3/1999.
MarcK is online now  
Old 21st Jan 2020, 17:54
  #50 (permalink)  
 
Join Date: Jan 2008
Location: Wintermute
Posts: 50
Originally Posted by FrequentSLF View Post
I would assume that the main issues are the interrupt handling, working with the interrupts on the 286 has been always a pain.
What surprises me is that they did not setup a fully functional rig, they have 400 737 parked, and could have used one as a lab rig for the development of the SW, is that something difficult to do?
What surprises me is that they don't have a full simulator, which simulates the aircraft to the control system (not the same as the simulator the fly-boys sit in and play). In this day and age it is criminally inept not to do so. 30 years ago we fully simulated trains to the train control systems to ensure that when the control system hit the tracks in a train it worked, including everythng from signalling systems to wheel slip on wet rails, same goes on today in the automotive industry, down to the simulation of the chassis/road/surface/tyre, etc, etc. I have never delivered a control system capable of killing people which was not substantially tested on a full capability simulator whch could inject any fault or situation possible to the software to ensure it dealt with it, however impossible to believe.

A commercial aircraft is not any more complex than a modern fly by wire train . . . probably not even an order of magnitude more complex than a modern car . . .

Surely . . Boeing don't do this on a real plane . . . like people used to do in the stone age . . .

Edited to add : testing as described is only a small part of the testing required, factor the cost of testing being >2-4x the cost of writing the software in the first place . . . and that is conservative . . . which answers the java programmers question of why it's taken so long to get round to testing it on an aircraft . . .
fergusd is offline  
Old 21st Jan 2020, 18:03
  #51 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,346
Originally Posted by MarcK View Post
The 286 was introduced 2/1982, and notified as end-of-life (last ship) 3/1999.
Possibly as a commercial PC chip but there are manufacturers making 80286 boards as aviation uses a lot of them. The reason is that in the late 1980's the safety people in aviation were really taken with the idea of 'formal proof' of all the processing in avionics. I can remember briefings back then on how this would remove errors - totally disregarding the fact that hardware manufacturing errors and even software coding errors were even then getting rarer and could be handled. The worst kind of error was as in MCAS a perfectly implemented poor design. Nevertheless, it became a requirement that the hardware had to be formally proved. Then along came multi-core chips with intelligent prefetch and with preemption and multiprocessing... and the 'formal proof' rapidly became an extremely hard Np problem. So 80286's continue to be used as they can be formally proved. And this in itself is a problem as they are 'beasts of very little brain' especially when trying to handle a multiplicity of interrupts.
Is Ancient Silicon The Root Of Boeing’s Problems?
It is about time that the entire area is revisited as the handheld game player the 4 year old has in seat 28G is many times more powerful than the FCCs. All smart phones are way way above the capabilities of the CPU in the FCC. Like communications the time has come to abandon some of these old 'safety' ideas and move to a more commercial approach as is used in other areas. Size, Weight and Power are not an issue these days.
Ian W is offline  
Old 21st Jan 2020, 18:12
  #52 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,346
Originally Posted by fergusd View Post
What surprises me is that they don't have a full simulator, which simulates the aircraft to the control system (not the same as the simulator the fly-boys sit in and play). In this day and age it is criminally inept not to do so. 30 years ago we fully simulated trains to the train control systems to ensure that when the control system hit the tracks in a train it worked, including everythng from signalling systems to wheel slip on wet rails, same goes on today in the automotive industry, down to the simulation of the chassis/road/surface/tyre, etc, etc. I have never delivered a control system capable of killing people which was not substantially tested on a full capability simulator whch could inject any fault or situation possible to the software to ensure it dealt with it, however impossible to believe.

A commercial aircraft is not any more complex than a modern fly by wire train . . . probably not even an order of magnitude more complex than a modern car . . .

Surely . . Boeing don't do this on a real plane . . . like people used to do in the stone age . . .

Edited to add : testing as described is only a small part of the testing required, factor the cost of testing being >2-4x the cost of writing the software in the first place . . . and that is conservative . . . which answers the java programmers question of why it's taken so long to get round to testing it on an aircraft . . .
Like this system you mean?
Ian W is offline  
Old 21st Jan 2020, 18:14
  #53 (permalink)  
 
Join Date: Oct 2019
Location: USA
Posts: 133
Originally Posted by OldnGrounded View Post
I seriously doubt that Boeing would find it difficult to arrange to test on a ready-to-fly MAX in a safe and convenient location.
Which is what they did. But it isn't something that seems reasonably arranged for the multiple compilations each day that might be done while working with simulator testing.
MechEngr is offline  
Old 21st Jan 2020, 18:27
  #54 (permalink)  
 
Join Date: Oct 2004
Location: California
Posts: 283
Originally Posted by Ian W View Post
Where the simulation computers are 1990's vintage?
MarcK is online now  
Old 21st Jan 2020, 18:31
  #55 (permalink)  
 
Join Date: Nov 2018
Location: back out to Grasse
Posts: 171
So 80286's continue to be used as they can be formally proved
….and therein I suspect lies the problem, being "formally proved" may not necessarily mean "Failsafe". I would like to think that error reporting and recovery rapidly returns the system (hardware and software) to a state in which the integrity of the platform is not compromised.

What of "formally proved" now. How much fix testing and regression testing is required to regain the "formally proved" status. It is not sufficient to prove just the hardware and software since a layer of microcode exists as the sandwich filling within the ALU and PROM/ low order RAM portion of the 286 which may not be tolerating some minor timing discrepancy on either side of the interface to cause the failure. Is the 286 perhaps being driven to it's critical limits?

IG
Imagegear is online now  
Old 21st Jan 2020, 18:37
  #56 (permalink)  
 
Join Date: Oct 2019
Location: USA
Posts: 133
Originally Posted by MarcK View Post
The 286 was introduced 2/1982, and notified as end-of-life (last ship) 3/1999.
That would be the Intel 80286, I expect. However Intel would certainly license the core to embedded systems developers for production at rates and for markets in which Intel could no longer afford to maintain fabrication. For example: Renasas 80C286 went EOL in 2017. Would not be surprised if someone bought 5000 of them for the shelf.

I would expect the next generation processor to be an FPGA rather than a microprocessor. I see some links exploring ADA on FPGAs, but it seems to be programming the FPGA to act like a microprocessor.
MechEngr is offline  
Old 21st Jan 2020, 19:44
  #57 (permalink)  
 
Join Date: Jan 2008
Location: Wintermute
Posts: 50
Originally Posted by Ian W View Post
Nope . . .

Fd
fergusd is offline  
Old 21st Jan 2020, 21:48
  #58 (permalink)  
 
Join Date: Nov 2019
Location: Earth
Posts: 7
Originally Posted by FrequentSLF View Post
I would assume that the main issues are the interrupt handling, working with the interrupts on the 286 has been always a pain.
In what sense? A lot of what made the 286 unpopular was its poor backwards compatibility with DOS which was designed for real mode, that ought not be an issue on a system that's hopefully not trying to run DOS.

Originally Posted by MechEngr View Post
That would be the Intel 80286, I expect. However Intel would certainly license the core to embedded systems developers for production at rates and for markets in which Intel could no longer afford to maintain fabrication. For example: Renasas 80C286 went EOL in 2017. Would not be surprised if someone bought 5000 of them for the shelf.
Boeing almost certainly uses a radiation hardened processor (MIL-STD-883) and not a standard desktop CPU.

Originally Posted by MechEngr View Post
I would expect the next generation processor to be an FPGA rather than a microprocessor. I see some links exploring ADA on FPGAs, but it seems to be programming the FPGA to act like a microprocessor.

I'd be surprised here too. These days you can get haardened RISC and PPC based processors that would provide a ton more power and flexibility than a 286-based design. There's also probably a push to use lower cost off-the-shelf components.
crankyanker is offline  
Old 22nd Jan 2020, 08:42
  #59 (permalink)  
 
Join Date: Jul 2007
Location: Switzerland
Age: 74
Posts: 96
Now the poor software guys working on the max get more time. The first move of B's new management in the right direction to get the max flying again. I hope they clear up some other issues as well until summer (2020 !).
And thank you all for your valuable contributions and explanations! Has been very enlightening and is good to see that PPRuNe also hosts know-how complementary to stick and rudder.
clearedtocross is offline  
Old 22nd Jan 2020, 09:30
  #60 (permalink)  
 
Join Date: Mar 2002
Location: London, UK
Posts: 432
Originally Posted by Ian W View Post
The Boeing engineers did not have the luxury of starting from a clean sheet and "getting it right by design". They had stiffware that has been operational for years and had to modify the code so that the FCCs operated in a different way without changing anything that was not essential to change for the task at hand and without breaking any current functions. Maintenance programming especially of embedded code and modifying the code so it does things differently without affecting anything else is nothing like simple code writing. It is possible that some very basic timing issue made the live aircraft slightly different to the avionics test bench. This is the reason regression tests are run when the new code is ported to and implemented in the aircraft - and the tests found an issue - that is what the tests are for.
If it helps, think of it like this...

RomeoTangoFoxtrotMike is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service - Do Not Sell My Personal Information

Copyright © 2018 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.