Go Back  PPRuNe Forums > Flight Deck Forums > Rumours & News
Reload this Page >

U.K. NATS Systems Failure

Rumours & News Reporting Points that may affect our jobs or lives as professional pilots. Also, items that may be of interest to professional pilots.

U.K. NATS Systems Failure

Old 29th Aug 2023, 14:32
  #81 (permalink)  
 
Join Date: Sep 2009
Location: UK
Posts: 11,837
Likes: 0
Received 31 Likes on 26 Posts
I once had to deal with a major retail park where all the stores were reporting a similar problem at the same time. Their card transactions were not going through . I got called out at silly o'clock to go there and have a look as the techy boys off site could not see the problem....
it was a wireless link to a 3G mast that someone built a building near and blocked the signal...
Solution was a Fibre Cable run of about 3 miles... So we dug two different routes in case someone else descided to put a digger through one...and yes some one did 😊
Kiltrash is offline  
Old 29th Aug 2023, 15:02
  #82 (permalink)  
 
Join Date: Jan 2008
Location: Glorious Devon
Posts: 2,687
Received 852 Likes on 495 Posts
Originally Posted by DaveReidUK
Neither do I - it's inconceivable.
Why inconceivable? Do none of the explanations of how it might happen, offered by other posters, not persuade you?
Do you have another idea that we might discuss?
Ninthace is offline  
Old 29th Aug 2023, 15:22
  #83 (permalink)  
 
Join Date: Aug 2001
Location: se england
Posts: 1,579
Likes: 0
Received 48 Likes on 21 Posts
Multiple fibre routes to mission critical buildings are prety much the norm for many years now, even if the building owner doesnt spec it the comms provider is likely to insist. Sometimes there are more than two routes but they do need to be celalry seperated internally and externally.

CBSITCB gave a very eleoquant and concise descritpiton of non hrdware fail modes and software is useallya more difficult fix than hardware. However in a really mission critical situation then a huge amount of effort has to go into software resilience-think Airbus- and for that reason it would seem close to nonsensicle that misentered flight plans could cause such chaos.
I have worked with telecoms network managemnt and switching systems that had very high levels of resilience national or regional level of the telecoms network including in todays world intenet hardware reallycannot be allowed to fail. . To be fair they do not have the functional complexity of a system like NATs has msotly just switch paths around and maanging digital data flows. We did have one susyem that used a comparator function where two processors controlling one system constantly compared operating states. Ifa failure mode occured this would determien which system had in someway been corrupted and switched it out leaving the unchanged processor in charge.. However we did have a short outage due to the comparator failing so nothings completley foolproof . Personally i think the Daily Mail and Express sabotaged it to stop people travelling to Europe or vice versa
pax britanica is offline  
Old 29th Aug 2023, 16:07
  #84 (permalink)  
 
Join Date: Dec 2007
Location: Itinerant
Posts: 828
Received 77 Likes on 13 Posts
Originally Posted by Ninthace
Why inconceivable? Do none of the expla nations of how it might happen, offered by other posters, not persuade you?
Do you have another idea that we might discuss?
DRUK said he doesn't believe the reason provided by NATS. I too find it inconceivable in this instance. See ATC Watcher's post #52 for an explanation of why NATS statement is at least highly suspect. Many of us who have significant experience in this realm (as does ATC Watcher) would call "BS" on NATS in this case.
grizzled is offline  
Old 29th Aug 2023, 16:20
  #85 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 478
Likes: 0
Received 0 Likes on 0 Posts
If we assume the problem was with NAS then the explanation of an "unusual" flight plan as the initiator is not inconceivable, that system has a 40 year history of similar events. Read the lengthy contributions above from those with an understanding of the system to see why. What that doesn't explain is why the controls & processes which have controlled that risk for the last 20 years didn't yesterday.
eglnyt is offline  
Old 29th Aug 2023, 16:30
  #86 (permalink)  
c52
 
Join Date: Sep 2007
Location: Surrey
Posts: 2,262
Received 0 Likes on 0 Posts
I worked on a system at a large airport that crashed every time the QNH reached something like 1037.5. A rare enough event and I am full of admiration for whoever it was who spotted the link between all the failures. I would have said it was a system that had no need to know about QNH.
c52 is offline  
Old 29th Aug 2023, 16:31
  #87 (permalink)  
 
Join Date: Dec 2007
Location: Itinerant
Posts: 828
Received 77 Likes on 13 Posts
Originally Posted by eglnyt
that system has a 40 year history of similar events. .
Hint as to the root cause of this event: If your statement is correct then the root cause of the system failure is NOT a specific "French airline's flight plan".
grizzled is offline  
Old 29th Aug 2023, 16:35
  #88 (permalink)  
 
Join Date: Mar 2016
Location: Location: Location
Posts: 59
Received 0 Likes on 0 Posts
Originally Posted by NWSRG
Don't believe the incorrect flight plan malarky...seems an incredulous reason for a whole system failure. And if it was cyber, would they actually admit it?
Originally Posted by DaveReidUK
Neither do I - it's inconceivable.
Saving this post just in case I need to refer to it later...
CBSITCB is offline  
Old 29th Aug 2023, 16:38
  #89 (permalink)  
 
Join Date: Apr 2008
Location: cowtown
Posts: 898
Received 60 Likes on 43 Posts
Did you turn it off and on again ?
As per the IT Crowd best practices advice
fitliker is offline  
Old 29th Aug 2023, 16:50
  #90 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 478
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by grizzled
Hint as to the root cause of this event: If your statement is correct then the root cause of the system failure is NOT a specific "French airline's flight plan".
Correct. I have no idea yet whether it was an issue with NAS or a French flight plan. But if that was the case it is a, probably unusual, flight plan that has been through multiple layers of validation that happens to expose an issue in the FDP system. The flight plan is the initiator and, using some methods, a cause but not the real issue.
eglnyt is offline  
Old 29th Aug 2023, 17:10
  #91 (permalink)  
 
Join Date: Aug 2001
Location: se england
Posts: 1,579
Likes: 0
Received 48 Likes on 21 Posts
Government,immediate response 'Its not a cyber attack'

NATS somewhat later response 'We dont know what caused it yet'

Perhaps HMG jumping the gun , ie nothing to do with us .

Can out of pocket pax sue NATS ?? Are memebers of the 'Airline Group' owning 42% of NATS liable even if the airlines are not ;. after all they are a significant part of the the management
pax britanica is offline  
Old 29th Aug 2023, 17:42
  #92 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 478
Likes: 0
Received 0 Likes on 0 Posts
Can out of pocket pax sue NATS ?? Are memebers of the 'Airline Group' owning 42% of NATS liable even if the airlines are not ;. after all they are a significant part of the the management
I'm no lawyer but there is no contract between NATS and the pax so what would be the basis of that action? There is a contract between the airline and the pax but what pax can claim is limited by legislation. There is a contract between NATS and the Airline but as the charging regime provides for penalties it would be unlikely that the airlines could sue NATS.

This is all quite deliberate. NATS was privatised just after the Hatfield rail crash where penalties for service standards had produced an adverse effect on safety. The Government of the day wanted to avoid any financial imperative that would have any effect on day to day operational decisions affecting safety. Hence yesterday the flow regulation could be imposed purely with regard to safety.

Note that the Airline Group is a separate entity from the Airlines who hold shares in it, indeed not all shareholders in the Airline Group are airlines. The Airline Group is a PLC limiting the liability of its shareholders and in turn NATS is a collection of Limited Companies limiting the liability of its shareholders.
eglnyt is offline  
Old 29th Aug 2023, 17:55
  #93 (permalink)  
 
Join Date: Nov 2001
Location: Delta of Venus
Posts: 2,383
Likes: 0
Received 2 Likes on 1 Post
As I recall, Eurocontrol had then/has now, their system totally backed up, duplicated in two different control centres. As others have said, deteriorating infrastructure in the UK (or rather lack of it from the get go), but who pays to improve it & why should they pay???
Private jet is offline  
Old 29th Aug 2023, 18:08
  #94 (permalink)  
 
Join Date: Dec 2020
Location: Home
Posts: 116
Received 28 Likes on 6 Posts
This is all quite deliberate. NATS was privatised just after the Hatfield rail crash where penalties for service standards had produced an adverse effect on safety. The Government of the day wanted to avoid any financial imperative that would have any effect on day to day operational decisions affecting safety. Hence yesterday the flow regulation could be imposed purely with regard to safety.
This takes me back…..I had a peripheral involvement with the original NATS licence and I can recall lots of debate about closure of airspace - principally that the civil service/government people never wanted UK airspace to be closed (unless they said so, of course). Lots of people involved in day to day running things said that there were very rare, but valid reasons that this might happen. But applying a 0 aircraft per hour flow rate was fine. Is this what they call spin?
Equivocal is offline  
Old 29th Aug 2023, 18:13
  #95 (permalink)  
 
Join Date: Dec 2007
Location: uk
Posts: 99
Likes: 0
Received 0 Likes on 0 Posts
Some interesting discussion on this forum: https://forums.theregister.com/forum...hts_disrupted/
Link Kilo is offline  
Old 29th Aug 2023, 18:15
  #96 (permalink)  
 
Join Date: Nov 2000
Location: Canada
Posts: 603
Likes: 0
Received 0 Likes on 0 Posts

UK air travel disruption may last for days, says British transport minister Mark Harper

https://www.firstpost.com/world/uk-air-travel-disruption-may-last-for-days-says-british-transport-minister-mark-harper-13053662.html
Longtimer is offline  
Old 29th Aug 2023, 18:17
  #97 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 478
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Private jet
As I recall, Eurocontrol had then/has now, their system totally backed up, duplicated in two different control centres. As others have said, deteriorating infrastructure in the UK (or rather lack of it from the get go), but who pays to improve it & why should they pay???
Would that be the system that failed in 2018 when they thought it had switched to the other centre but hadn't, and because the phones didn't switch over as expected, nobody could ring them to tell them?

NATS has a lot of redundancy and duplication and is currently investing in even more but please read the great explanations by others above as to why it might not always help.
eglnyt is offline  
Old 29th Aug 2023, 18:29
  #98 (permalink)  
 
Join Date: Nov 2001
Location: Delta of Venus
Posts: 2,383
Likes: 0
Received 2 Likes on 1 Post
Originally Posted by eglnyt
Would that be the system that failed in 2018 when they thought it had switched to the other centre but hadn't, and because the phones didn't switch over as expected, nobody could ring them to tell them?

NATS has a lot of redundancy and duplication and is currently investing in even more but please read the great explanations by others above as to why it might not always help.
Thankyou, & with respect, point taken!
Private jet is offline  
Old 29th Aug 2023, 19:23
  #99 (permalink)  
 
Join Date: Jan 2008
Location: Reading, UK
Posts: 15,816
Received 201 Likes on 93 Posts
Originally Posted by Link Kilo
Some interesting discussion on this forum: https://forums.theregister.com/forum...hts_disrupted/
Hmmm. That's 10 minutes of my life I'll never get back, and I'm none the wiser.
DaveReidUK is offline  
Old 29th Aug 2023, 20:02
  #100 (permalink)  
Pegase Driver
 
Join Date: May 1997
Location: Europe
Age: 74
Posts: 3,682
Likes: 0
Received 0 Likes on 0 Posts
As far as I know at this time , no-one really know what caused the system to crash , the root cause I mean , not yet but likely by tomorrow or in the next coming days we'll know. From my experience in the last 40 years or so the Flight plan processing systems (FDPs) are generally crashing following a system update. one line of programming is wrong ,and when an external factor comes in . it causes the issue. This could happen at any time , generally when the system is peaking . Typically system updates are done at night, tested for a few hours , then if OK put on line in the morning ,At least that how we do it in most centers. Done it for years in my own center. If it crashed later in the day we just reverted to the previous level which is on stand by on the back up computers, whole thing takes no more that minutes of an hour max to be back to normal. When if takes half a day or more then something is wrong on your processes or your system architecture. Could also simply be the result of cost cutting measures, like not replacing back up computers, or outsourcings maintenance and code writing ,to far away countries with cheaper labor , etc.. I am not saying that this was the case here in NATS, but I have seen this happening in other places recently .

Finally FDP system failures are not a unique UK/NATS issue, Geneva had a major failure some time ago, , Brussels, a couple of years back etc. even Roma had one also yesterday.at the same time as London , so we feared a wider cyber attack. But so fat it looks like the 2 were not connected. But if the investigation later shows they were, then we really are in the sh*t .
ATC Watcher is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service

Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.