PPRuNe Forums

PPRuNe Forums (https://www.pprune.org/)
-   Rumours & News (https://www.pprune.org/rumours-news-13/)
-   -   U.K. NATS Systems Failure (https://www.pprune.org/rumours-news/654461-u-k-nats-systems-failure.html)

Kiltrash 29th Aug 2023 14:32

I once had to deal with a major retail park where all the stores were reporting a similar problem at the same time. Their card transactions were not going through . I got called out at silly o'clock to go there and have a look as the techy boys off site could not see the problem....
it was a wireless link to a 3G mast that someone built a building near and blocked the signal...
Solution was a Fibre Cable run of about 3 miles... So we dug two different routes in case someone else descided to put a digger through one...and yes some one did 😊

Ninthace 29th Aug 2023 15:02


Originally Posted by DaveReidUK (Post 11493584)
Neither do I - it's inconceivable.

Why inconceivable? Do none of the explanations of how it might happen, offered by other posters, not persuade you?
Do you have another idea that we might discuss?

pax britanica 29th Aug 2023 15:22

Multiple fibre routes to mission critical buildings are prety much the norm for many years now, even if the building owner doesnt spec it the comms provider is likely to insist. Sometimes there are more than two routes but they do need to be celalry seperated internally and externally.

CBSITCB gave a very eleoquant and concise descritpiton of non hrdware fail modes and software is useallya more difficult fix than hardware. However in a really mission critical situation then a huge amount of effort has to go into software resilience-think Airbus- and for that reason it would seem close to nonsensicle that misentered flight plans could cause such chaos.
I have worked with telecoms network managemnt and switching systems that had very high levels of resilience national or regional level of the telecoms network including in todays world intenet hardware reallycannot be allowed to fail. . To be fair they do not have the functional complexity of a system like NATs has msotly just switch paths around and maanging digital data flows. We did have one susyem that used a comparator function where two processors controlling one system constantly compared operating states. Ifa failure mode occured this would determien which system had in someway been corrupted and switched it out leaving the unchanged processor in charge.. However we did have a short outage due to the comparator failing so nothings completley foolproof . Personally i think the Daily Mail and Express sabotaged it to stop people travelling to Europe or vice versa

grizzled 29th Aug 2023 16:07


Originally Posted by Ninthace (Post 11493620)
Why inconceivable? Do none of the expla nations of how it might happen, offered by other posters, not persuade you?
Do you have another idea that we might discuss?

DRUK said he doesn't believe the reason provided by NATS. I too find it inconceivable in this instance. See ATC Watcher's post #52 for an explanation of why NATS statement is at least highly suspect. Many of us who have significant experience in this realm (as does ATC Watcher) would call "BS" on NATS in this case.

eglnyt 29th Aug 2023 16:20

If we assume the problem was with NAS then the explanation of an "unusual" flight plan as the initiator is not inconceivable, that system has a 40 year history of similar events. Read the lengthy contributions above from those with an understanding of the system to see why. What that doesn't explain is why the controls & processes which have controlled that risk for the last 20 years didn't yesterday.

c52 29th Aug 2023 16:30

I worked on a system at a large airport that crashed every time the QNH reached something like 1037.5. A rare enough event and I am full of admiration for whoever it was who spotted the link between all the failures. I would have said it was a system that had no need to know about QNH.

grizzled 29th Aug 2023 16:31


Originally Posted by eglnyt (Post 11493672)
that system has a 40 year history of similar events. .

Hint as to the root cause of this event: If your statement is correct then the root cause of the system failure is NOT a specific "French airline's flight plan".

CBSITCB 29th Aug 2023 16:35


Originally Posted by NWSRG (Post 11493457)
Don't believe the incorrect flight plan malarky...seems an incredulous reason for a whole system failure. And if it was cyber, would they actually admit it?

Originally Posted by DaveReidUK (Post 11493584)
Neither do I - it's inconceivable.


Saving this post just in case I need to refer to it later...;)

fitliker 29th Aug 2023 16:38

Did you turn it off and on again ?
As per the IT Crowd best practices advice

eglnyt 29th Aug 2023 16:50


Originally Posted by grizzled (Post 11493680)
Hint as to the root cause of this event: If your statement is correct then the root cause of the system failure is NOT a specific "French airline's flight plan".

Correct. I have no idea yet whether it was an issue with NAS or a French flight plan. But if that was the case it is a, probably unusual, flight plan that has been through multiple layers of validation that happens to expose an issue in the FDP system. The flight plan is the initiator and, using some methods, a cause but not the real issue.

pax britanica 29th Aug 2023 17:10

Government,immediate response 'Its not a cyber attack'

NATS somewhat later response 'We dont know what caused it yet'

Perhaps HMG jumping the gun , ie nothing to do with us .

Can out of pocket pax sue NATS ?? Are memebers of the 'Airline Group' owning 42% of NATS liable even if the airlines are not ;. after all they are a significant part of the the management

eglnyt 29th Aug 2023 17:42


Can out of pocket pax sue NATS ?? Are memebers of the 'Airline Group' owning 42% of NATS liable even if the airlines are not ;. after all they are a significant part of the the management
I'm no lawyer but there is no contract between NATS and the pax so what would be the basis of that action? There is a contract between the airline and the pax but what pax can claim is limited by legislation. There is a contract between NATS and the Airline but as the charging regime provides for penalties it would be unlikely that the airlines could sue NATS.

This is all quite deliberate. NATS was privatised just after the Hatfield rail crash where penalties for service standards had produced an adverse effect on safety. The Government of the day wanted to avoid any financial imperative that would have any effect on day to day operational decisions affecting safety. Hence yesterday the flow regulation could be imposed purely with regard to safety.

Note that the Airline Group is a separate entity from the Airlines who hold shares in it, indeed not all shareholders in the Airline Group are airlines. The Airline Group is a PLC limiting the liability of its shareholders and in turn NATS is a collection of Limited Companies limiting the liability of its shareholders.

Private jet 29th Aug 2023 17:55

As I recall, Eurocontrol had then/has now, their system totally backed up, duplicated in two different control centres. As others have said, deteriorating infrastructure in the UK (or rather lack of it from the get go), but who pays to improve it & why should they pay???

Equivocal 29th Aug 2023 18:08


This is all quite deliberate. NATS was privatised just after the Hatfield rail crash where penalties for service standards had produced an adverse effect on safety. The Government of the day wanted to avoid any financial imperative that would have any effect on day to day operational decisions affecting safety. Hence yesterday the flow regulation could be imposed purely with regard to safety.
This takes me back…..I had a peripheral involvement with the original NATS licence and I can recall lots of debate about closure of airspace - principally that the civil service/government people never wanted UK airspace to be closed (unless they said so, of course). Lots of people involved in day to day running things said that there were very rare, but valid reasons that this might happen. But applying a 0 aircraft per hour flow rate was fine. Is this what they call spin?

Link Kilo 29th Aug 2023 18:13

Some interesting discussion on this forum: https://forums.theregister.com/forum...hts_disrupted/

Longtimer 29th Aug 2023 18:15

UK air travel disruption may last for days, says British transport minister Mark Harper

https://www.firstpost.com/world/uk-air-travel-disruption-may-last-for-days-says-british-transport-minister-mark-harper-13053662.html

eglnyt 29th Aug 2023 18:17


Originally Posted by Private jet (Post 11493733)
As I recall, Eurocontrol had then/has now, their system totally backed up, duplicated in two different control centres. As others have said, deteriorating infrastructure in the UK (or rather lack of it from the get go), but who pays to improve it & why should they pay???

Would that be the system that failed in 2018 when they thought it had switched to the other centre but hadn't, and because the phones didn't switch over as expected, nobody could ring them to tell them?

NATS has a lot of redundancy and duplication and is currently investing in even more but please read the great explanations by others above as to why it might not always help.

Private jet 29th Aug 2023 18:29


Originally Posted by eglnyt (Post 11493750)
Would that be the system that failed in 2018 when they thought it had switched to the other centre but hadn't, and because the phones didn't switch over as expected, nobody could ring them to tell them?

NATS has a lot of redundancy and duplication and is currently investing in even more but please read the great explanations by others above as to why it might not always help.

Thankyou, & with respect, point taken!

DaveReidUK 29th Aug 2023 19:23


Originally Posted by Link Kilo (Post 11493745)
Some interesting discussion on this forum: https://forums.theregister.com/forum...hts_disrupted/

Hmmm. That's 10 minutes of my life I'll never get back, and I'm none the wiser. :O

ATC Watcher 29th Aug 2023 20:02

As far as I know at this time , no-one really know what caused the system to crash , the root cause I mean , not yet but likely by tomorrow or in the next coming days we'll know. From my experience in the last 40 years or so the Flight plan processing systems (FDPs) are generally crashing following a system update. one line of programming is wrong ,and when an external factor comes in . it causes the issue. This could happen at any time , generally when the system is peaking . Typically system updates are done at night, tested for a few hours , then if OK put on line in the morning ,At least that how we do it in most centers. Done it for years in my own center. If it crashed later in the day we just reverted to the previous level which is on stand by on the back up computers, whole thing takes no more that minutes of an hour max to be back to normal. When if takes half a day or more then something is wrong on your processes or your system architecture. Could also simply be the result of cost cutting measures, like not replacing back up computers, or outsourcings maintenance and code writing ,to far away countries with cheaper labor , etc.. I am not saying that this was the case here in NATS, but I have seen this happening in other places recently .

Finally FDP system failures are not a unique UK/NATS issue, Geneva had a major failure some time ago, , Brussels, a couple of years back etc. even Roma had one also yesterday.at the same time as London , so we feared a wider cyber attack. But so fat it looks like the 2 were not connected. But if the investigation later shows they were, then we really are in the sh*t .


All times are GMT. The time now is 07:51.


Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.