View Full Version : Boeing 787s must be turned off and on every 51 days
kiwi grey
4th Apr 2020, 04:55
According to https://www.theregister.co.uk/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ "The US Federal Aviation Administration has ordered Boeing 787 operators to switch their aircraft off and on every 51 days to prevent what it called 'several potentially catastrophic failure scenarios' – including the crashing of onboard network switches."
The FAA directive is here https://ad.easa.europa.eu/ad/US-2020-06-14, and comes into effect on Tuesday 7th April 2020
DaveReidUK
4th Apr 2020, 06:30
Interesting.
Given that some 787s haven't flown for nearly a year now as a consequence of the Trent 1000 problems, one wonders why this issue appears only to have surfaced now ?
dcoded
4th Apr 2020, 06:44
According to https://www.theregister.co.uk/2020/04/02/boeing_787_power_cycle_51_days_stale_data/ "The US Federal Aviation Administration has ordered Boeing 787 operators to switch their aircraft off and on every 51 days to prevent what it called 'several potentially catastrophic failure scenarios' – including the crashing of onboard network switches."
The FAA directive is here https://ad.easa.europa.eu/ad/US-2020-06-14, and comes into effect on Tuesday 7th April 2020
Wasn't a similar issue discussed a couple of years ago already? That the AC needs to remain completely powerless. Or was it on another fleet, A350? :confused:
DaveReidUK
4th Apr 2020, 07:58
Wasn't a similar issue discussed a couple of years ago already? That the AC needs to remain completely powerless. Or was it on another fleet, A350? :confused:
EASA AD 2017-0129 concerned unmodded A350 aircraft where various avionic systems stopped talking to each other after 149 hours of continuous power-up.
There was also a previous 787 AD in 2015 which related to a loss of AC power after a counter overflowed.
mach2.6
4th Apr 2020, 15:18
OK, Hal, I have control of the aircraft. Hal? Hal...??! Hal.......??????
Luc Lion
4th Apr 2020, 15:42
I would bet that this issue is caused by an incorrect garbage collection of disused packets in CCS.
I wonder is this is another bug from Wind River Systems (the company selling VxWorks) or if the bug only touches the customisation that Honeywell made for Boeing.
jimjim1
4th Apr 2020, 15:54
It was pointed out in Another Place that 51 days is quite close to 2^32 milliseconds.
50d 17h 02m 47s
Luc Lion
4th Apr 2020, 16:08
It looks as though the guy who computed that used a standard round rather than a round down for the day count calculation.
The correct value is:
2^32 ms = 49d 17h 2m 47s 296ms
Does anyone know if the software was outsourced or did Honeywell have complete control over this?
jimjim1
4th Apr 2020, 16:49
It looks as though the guy who computed that
It was me :-) I just used Excel and trusted some number on intertubes for the number of secs in a day. I won't be flying on one so was a bit slapdash.
Edited - AH! Penny dropped. I accepted the whole number of days but of course Excel rounded it up.
DaveReidUK
4th Apr 2020, 18:20
It looks as though the guy who computed that used a standard round rather than a round down for the day count calculation.
The correct value is:
2^32 ms = 49d 17h 2m 47s 296ms
A common tick counter period in real-time systems is 1.024 ms (don't ask me why!). That works out at 50.9 days.
RatherBeFlying
4th Apr 2020, 18:33
Way back when, the odd student co-op student would want to see how long it took to overflow a 32 bit register by bumping it by one. The time would of course double adding unsigned.
At about 5 microseconds per Add and Branch on a '70s mainframe times 2^32 you could be looking at some 20,000 seconds which will be a few hours.
fergusd
4th Apr 2020, 18:54
A common tick counter period in real-time systems is 1.024 ms (don't ask me why!). That works out at 50.9 days.
It really isn't . . .
Fd
Rember this?The new US stealth fighter, the F-22 Raptor (http://www.f-16.net/modules/Gallery2/gallery2/main.php?g2_view=core.DownloadItem&g2_itemId=208072&g2_serialNumber=2), was deployed for the first time to Asia earlier this month. On Feb. 11, twelve Raptors flying from Hawaii to Japan were forced to turn back when a software glitch crashed all of the F-22s' on-board computers as they crossed the international date line.
pilotmike
4th Apr 2020, 19:44
A common tick counter period in real-time systems is 1.024 ms (don't ask me why!).
It is because 16MHz / 2^14 gives a period of 1.024ms, ie. 976.5625Hz
jetfour
4th Apr 2020, 20:05
in the real world, CTRL-ALT-DEL would sort it. Or maybe, a hard restart!
a software glitch crashed all of the F-22s' on-board computers as they crossed the international date line.
That was a well known one in commercial systems some years ago that they could not handle the clock going backwards; we had to hard shut down overnight in the autumn when winter time came in.
Another one worked fine for a couple of years but could not handle the next leap year that came along for any payments initiated before February 29th that were due afterwards.
One accounting system I was involved with started to lose large numbers of transactions. Eventually traced to an internal counter that went up to 9999, but that was proving insufficient so it was increased - everywhere but in one place, so while everything else moved on from 9999 to 10001, this went to 0001 and started overwriting the initial items.
Global Aviator
4th Apr 2020, 23:56
Power down... power up...
I would say that most pilots who have flown the 320 series for any period of time would hav seen or been asked my maintenance to do the power down.
In 10k hours on the baby bus, 3 times comes to mind, 2 reset the issue and the 3rd was never going to work!
Ahhh computers, software...
Now where is that old Cessna and the 6 pack?
FullWings
5th Apr 2020, 00:19
Just wait until 19th Jan 2038 (or when systems start referencing that date or beyond)...
RickNRoll
5th Apr 2020, 00:36
in the real world, CTRL-ALT-DEL would sort it. Or maybe, a hard restart!
Have you tried turning it off and on again. Oh, wait....
Ndicho Moja
5th Apr 2020, 03:30
I would have thought a good time to perform a power cycle would be when the nav data bases are updated, usually every twenty-eight days.
DaveReidUK
5th Apr 2020, 06:23
It really isn't . . .
Fine. In that case, feel free to tell us how the 51 days is really derived.
Use more than three words, if you need to. :O
Rimmon
5th Apr 2020, 19:27
Fine. In that case, feel free to tell us how the 51 days is really derived.
Use more than three words, if you need to. :O
Because maybe it has nothing do with a fixed period of time? Maybe there is a cache or a temporary log etc. that gets updated regularly and a some new data is attached. After 51 days the storage that is used for this activity is full and the system crashes. Just one example.
fergusd
5th Apr 2020, 21:03
Fine. In that case, feel free to tell us how the 51 days is really derived.
Use more than three words, if you need to. :O
35 years of experience in real time control systems software in safety critical industries says you're talking sh1te.
That's more than 3 words . . .
Back you your area of speciality, if you have one . . .
Check Airman
5th Apr 2020, 22:02
It was pointed out in Another Place that 51 days is quite close to 2^32 milliseconds.
50d 17h 02m 47s
For those of us not so inclined, what's the significance of 2^32?
KRviator
5th Apr 2020, 22:08
Have you tried turning it off and on again. Oh, wait....Good to see it isn't just us. Our modern locomotives have dozens of computers running everything from the diesel engine & traction power systems to crew displays, signalling and ATP. I've lost count of the number of BOBO (Battery off, wait a bit, battery on) reboots I've had to do to try to get them talking to each other again!
DaveReidUK
5th Apr 2020, 22:16
35 years of experience in real time control systems software in safety critical industries says you're talking sh1te.
That's more than 3 words . . .
Back you your area of speciality, if you have one . . .
You could have just left it at "I can't explain", then three words would indeed have sufficed.
DaveReidUK
5th Apr 2020, 22:32
For those of us not so inclined, what's the significance of 2^32?
Counters, and microprocessor registers in general, typically have a capacity that corresponds to some multiple of 8 bits.
For example a 16-bit counter isn't capable of counting beyond 65,535 (one less than 2^16). Using one for an application that requires counting to values higher than that will produce unpredictable results.
In the case of the 787 issue, whatever the maximum value that the counter can accommodate is (2^32 is being suggested, equivalent to 4,294,967,295 clock ticks), that value is reached after approximately 51 days.
HighWind
6th Apr 2020, 06:57
A common tick counter period in real-time systems is 1.024 ms (don't ask me why!). That works out at 50.9 days
It is because 16MHz / 2^14 gives a period of 1.024ms, ie. 976.5625Hz
This is just one example..There is no common implementation.. although OS system ticks in the order of 1 to 10 ms. is common. (For big OS'es)
Real-time systems interfacing directly with hardware are often have much faster system tick and OS preemptive interrupts.
And almost all CPU's today have a clock faster than 16MHz. (i.e. prescaler values need to be larger than 2^14 to divide to 1ms. )
For PC hardware google High Precision Event Timer and cpu Time_Stamp_Counter
Luc Lion
6th Apr 2020, 10:19
A common tick counter period in real-time systems is 1.024 ms (don't ask me why!). That works out at 50.9 days.
Thanks Dave.
I forgot that most modern CPUs are clocked to give a reliable 1 microsecond interval.
That means that an interrupt coded to click every millisecond is actually triggered every 2^10 microseconds or 1.024 millisecond.
That is the reason.
Jumbo Jockey
6th Apr 2020, 10:20
The Genesis Manoeuvre.... Turn it off, turn it on again :)
The Genesis Manoeuvre.... Turn it off, turn it on again :)
Not applicable as that would mean doing the reset 13 times in 8 days, which is not the case.
Check Airman
7th Apr 2020, 05:49
Counters, and microprocessor registers in general, typically have a capacity that corresponds to some multiple of 8 bits.
For example a 16-bit counter isn't capable of counting beyond 65,535 (one less than 2^16). Using one for an application that requires counting to values higher than that will produce unpredictable results.
In the case of the 787 issue, whatever the maximum value that the counter can accommodate is (2^32 is being suggested, equivalent to 4,294,967,295 clock ticks), that value is reached after approximately 51 days.
Appreciate the explanation. Am I to assume that there's some sort of clock somewhere on the plane that was only designed to count up to 2^32, and so will need to be reset before it gets to that value, so it doesn't run out of fingers to count on?
DaveReidUK
7th Apr 2020, 07:18
Appreciate the explanation. Am I to assume that there's some sort of clock somewhere on the plane that was only designed to count up to 2^32, and so will need to be reset before it gets to that value, so it doesn't run out of fingers to count on?
Exactly.
Of course the reset happens automatically every time the aircraft is powered down. It sounds like it never occurred to Boeing or the FAA that you could have a scenario where an aircraft was continuously powered-up for 51 days or more, and I'm struggling too to imagine that actually happening.