Go Back  PPRuNe Forums > Flight Deck Forums > Rumours & News
Reload this Page >

U.K. NATS Systems Failure

Rumours & News Reporting Points that may affect our jobs or lives as professional pilots. Also, items that may be of interest to professional pilots.

U.K. NATS Systems Failure

Old 4th Sep 2023, 22:02
  #241 (permalink)  
 
Join Date: Nov 2018
Location: UK
Posts: 81
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by eglnyt
You won't find me defending Executive pay but we may not be talking about the same person.
The CEO renumeration did dip when income dipped, remember most of financial year 2019/2020 wasn't affected and the critical summer period was the best ever. It has rebounded as traffic did, to my mind rather better than NATS staff in general who actually gave back their 2019/20 payrise when the pandemic hit.
We're talking about the (current) CEO, and his 7-figure bonus. This is supposed to be 'performance related', but that's a joke when your business drops by 80% (I know they're now covered over the pay outs)
Neo380 is offline  
Old 4th Sep 2023, 22:18
  #242 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 470
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Neo380
Why sack an entire cohort, some of whom were only 2 weeks away from qualifying?

You put a fantastic burnish on everything NATS does, with real detail (hence 'Corporate Comms'!), but this was clearly the wrong thing to do.
Heading way off topic now. Not my decision to defend but those were very strange times. We now know how the future of aviation panned out but at the time pre vacs it didn't look anywhere near as positive. The furlough scheme offered a reprieve to many NATS jobs but it was quite clear that was going to run out long before income returned. NATS prioritised keeping it's validated controllers. Others were not so lucky. A large proportion of technical staff were on short term contracts through complex multi company contracts and they were let go straight away. NATS had long since outsourced many ancillary functions to organisations who weren't so good to their staff. A fair few NATS were made redundant although the terms were pretty good so they faired pretty well.
eglnyt is offline  
Old 4th Sep 2023, 22:23
  #243 (permalink)  
 
Join Date: Nov 2018
Location: UK
Posts: 81
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by eglnyt
Heading way off topic now. Not my decision to defend but those were very strange times. We now know how the future of aviation panned out but at the time pre vacs it didn't look anywhere near as positive. The furlough scheme offered a reprieve to many NATS jobs but it was quite clear that was going to run out long before income returned. NATS prioritised keeping it's validated controllers. Others were not so lucky. A large proportion of technical staff were on short term contracts through complex multi company contracts and they were let go straight away. NATS had long since outsourced many ancillary functions to organisations who weren't so good to their staff. A fair few NATS were made redundant although the terms were pretty good so they faired pretty well.
Yes and no, voluntary redundancy was taken by all (bar one) volunteers. The £1.5m debt financing prevented the culling of any non-volunteers - and NATS is already well back into 'feast and famine' of controllers.

The subjects may change a little, but the theme throughout is identical, NATS is taking no blame at all for a catastrophic systems failure, that even you accept was inevitable - I suppose complexity will fool the tabloids.
Neo380 is offline  
Old 4th Sep 2023, 22:36
  #244 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 470
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Neo380
Yes and no, voluntary redundancy was taken by all (bar one) volunteers. The £1.5m debt financing prevented the culling of any non-volunteers - and NATS is already well back into 'feast and famine' of controllers.
As I said those made redundant faired quite well but voluntary is an interesting term. NATS had served notice that it was going to unilaterally withdraw the relatively good redundancy scheme. In mid 2020 it looked quite possible that income would not recover and redundancies would be required further down the track possibly on far worse terms. Many jumped because of that. Others realised that if a lot of people jumped they would be left in an under resourced organisation that wouldn't be a nice place to be.
eglnyt is offline  
Old 4th Sep 2023, 22:44
  #245 (permalink)  
 
Join Date: Nov 2018
Location: UK
Posts: 81
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by eglnyt
As I said those made redundant faired quite well but voluntary is an interesting term. NATS had served notice that it was going to unilaterally withdraw the relatively good redundancy scheme. In mid 2020 it looked quite possible that income would not recover and redundancies would be required further down the track possibly on far worse terms. Many jumped because of that. Others realised that if a lot of people jumped they would be left in an under resourced organisation that wouldn't be a nice place to be.
Hahahahaha - under resourced?! And voluntary means all (bar one) volunteers. No compulsory redundancies were made, despite the spin at the time (and clearly now), sorry, but this is turning into fantasy cover up.
Neo380 is offline  
Old 4th Sep 2023, 22:46
  #246 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 470
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Neo380

The subjects may change a little, but the theme throughout is identical, NATS is taking no blame at all for a catastrophic systems failure, that even you accept was inevitable - I suppose complexity will fool the tabloids.
We don't know who is taking what blame. The report was supposed to be with CAA today, the Secretary of State said it would be published later in the week. Until that runs its course we have no idea what happened or how much blame is to be apportioned.
Every software fault is inevitable we just don't know why this one played out with the impact it did. I expect every system to fail sooner or later, I wouldn't expect the failure of a system in the Flight Data thread to take so long to fix or have that impact so I'm waiting for the report to see why it did.
eglnyt is offline  
Old 5th Sep 2023, 01:09
  #247 (permalink)  
 
Join Date: Jan 2008
Location: Glorious Devon
Posts: 2,575
Received 504 Likes on 278 Posts
When you think that they started with a system that had been running successfully for many years and that the fault occurred on a Bank Holiday weekend, and take into account all the steps they would have to work through just to isolate the faulty code, identify the cause, work out a fix and then test it, in many ways, I am surprised how quickly they did fix it.
Ninthace is offline  
Old 5th Sep 2023, 05:24
  #248 (permalink)  
 
Join Date: Dec 2006
Location: Whanganui, NZ
Posts: 276
Received 5 Likes on 4 Posts
Originally Posted by Abrahn
Parliament can help in quite a lots of ways.
Firstly it can compel NATS management to tell the truth in public, which would otherwise be sadly lacking.
Secondly HMG holds a 49% share, and parliament could require that stake be exercised to improve things.​​​​

They could also commission their own report. And could come up with another number for the cost, one that might be less susceptible to manipulation by vested interests (or at least manipulated by different vested interests).
Or they could legislate to compulsorily acquire say 10% of each of the other shareholders' shares, thus leaving HMG as a 54.1% majority shareholder. Then require the Board (or a new Board if the current Directors seem reluctant) to immediately and comprehensively address the underlying issues.

Originally Posted by Abrahn
Or they could modify the terms of the licence to require whatever they see fit.

They might even look at a company that makes huge profits and then comes begging for public subsidy and think, you know what, maybe they should reinvest some of that profit rather than beg from the public purse. Or they could propose a tax on NATS profit to subsidise the investment if NATS won't do it themselves.
​​​​
kiwi grey is offline  
Old 5th Sep 2023, 10:35
  #249 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 470
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Abrahn
NATS Holdings profit before tax:

2013 £215m
2014 £240m
2015 £252m
2016 £137m
2017 £136m
2018 £132m
2019 £ 98m
2020 £ 25m
2021 £-38m
2022 £ 8m
2023 £148m
As I've previously stated the whole World of NATS Finance remains a mystery to me but shouldn't you use the profit after tax as a meaningful measure of what might be available to invest? In fact isn't the amount paid out in dividends the only real indicator of cash lost to investment as any other "profit" is effectively retained within the business?
eglnyt is offline  
Old 5th Sep 2023, 10:47
  #250 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 470
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by kiwi grey
Or they could legislate to compulsorily acquire say 10% of each of the other shareholders' shares, thus leaving HMG as a 54.1% majority shareholder. Then require the Board (or a new Board if the current Directors seem reluctant) to immediately and comprehensively address the underlying issues.
I'm not sure that's a valid way forward. There are rules about acquiring shareholdings in public companies. If you introduce legislation that steamrollers through those rules, even for a limited scope, you are setting a dangerous precedent. For a country totally reliant on other people's investment any hint that it might go all "Socialist" would be very problematic.

It actually probably doesn't need to do that. It holds more shares than anybody else and the 5% in the Employee Sharetrust is effectively out of play so it already has the power to control the company by asking its appointed directors to intervene but again that would send shockwaves through the markets. Still we currently have a Government that takes decisions based on "focus groups" regardless of the effect on the economy so who knows.

Last edited by eglnyt; 5th Sep 2023 at 10:54. Reason: changed a word for clarity
eglnyt is offline  
Old 5th Sep 2023, 16:28
  #251 (permalink)  
 
Join Date: Nov 2018
Location: UK
Posts: 81
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by eglnyt
I'm not sure that's a valid way forward. There are rules about acquiring shareholdings in public companies. If you introduce legislation that steamrollers through those rules, even for a limited scope, you are setting a dangerous precedent. For a country totally reliant on other people's investment any hint that it might go all "Socialist" would be very problematic.

It actually probably doesn't need to do that. It holds more shares than anybody else and the 5% in the Employee Sharetrust is effectively out of play so it already has the power to control the company by asking its appointed directors to intervene but again that would send shockwaves through the markets. Still we currently have a Government that takes decisions based on "focus groups" regardless of the effect on the economy so who knows.
The problem is, the government, as the 'largest' shareholder, doesn't understand what the real issues are, and everything is getting smothered is a positive PR campaign about how well NATS is doing at everything.

Anyone who properly knows the organisation knows that this isn't true (at least at a management level, not to take away from the great work the ATCOs and engineers are doing generally). The PPP - which strangely seems to be morphing into a PLC - doesn't help in gaining transparency and accountability. The main aim seems to be to keep the status quo, and the monopoly going as long as possible, or at least 'until I retire'...

The sector is about to undergo massive change with a 10x increase in aircraft (mostly uncrewed) in the next decade, and the wide scale digitisation of services. This sits poorly with a (largely) analogue organisation.

(At the risk of another 250 messages!) no one should still be running 1960s technology, especially not in a mission critical environment, and especially if you haven't provisioned for any fall backs (ignore the 'expense, complexity' etc deflections). And despite all of the posturing about 'airspace modernisation' it's the organisation that hasn't done it for sixty years that is claiming it's the only organisation that can do it now. Really, there are other approved providers of these services out there? We could go on, but there's plainly a significant management malaise in NATS - there has been for a long time - and just 'spinning the story' won't resolve the core issues.

Maybe privatisation would...?
Neo380 is offline  
Old 6th Sep 2023, 07:57
  #252 (permalink)  
Ecce Homo! Loquitur...
Thread Starter
 
Join Date: Jul 2000
Location: Peripatetic
Posts: 16,912
Received 1,320 Likes on 601 Posts
https://publicapps.caa.co.uk/modalap...etail&id=12321

CAP2582: NERL Major Incident Preliminary Report

https://publicapps.caa.co.uk/docs/33...y%20Report.pdf
ORAC is offline  
Old 6th Sep 2023, 08:04
  #253 (permalink)  
 
Join Date: Sep 2006
Location: Out of a bag
Posts: 654
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by ORAC
The report, submitted earlier this week to the Civil Aviation Authority, identified an issue with a flight plan processing sub-system called Flight Plan Reception Suite Automated – Replacement (FPRSA-R).

A small but important part of NATS’ overall air traffic control technical infrastructure, it was found to have encountered an extremely rare set of circumstances presented by a flight plan that included two identically named, but separate waypoint markers outside of UK airspace.

This led to a ‘critical exception’ whereby both the primary system and its backup entered a fail-safe mode. The report details how, in these circumstances, the system could not reject the flight plan without a clear understanding of what possible impact it may have had. Nor could it be allowed through and risk presenting air traffic controllers with incorrect safety critical information. On the day, the time taken to recover was driven by the need to identify the problem and the specific data, isolate and remove it in a controlled way, and then test it to ensure it could be returned safely into operation.

This scenario had never been encountered before, with the system having previously processed more than 15 million flight plans over the 5 years it has been in service. Steps have been taken to ensure the incident cannot be repeated.
Flying Wild is offline  
Old 6th Sep 2023, 08:18
  #254 (permalink)  
aox
 
Join Date: Mar 2015
Location: UK
Posts: 227
Received 0 Likes on 0 Posts
https://www.bbc.co.uk/news/business-66723586

Martin Rolfe, chief executive of Nats, said that the system did "what it was designed to do, i.e. fail safely when it receives data that it can't process".

He described it as "a one in 15 million flight plan that we received", meaning the engineers took a few hours to work out a situation they were not familiar with.
aox is offline  
Old 6th Sep 2023, 08:18
  #255 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 470
Likes: 0
Received 0 Likes on 0 Posts
So it wasn't the ageing Flight Data Processing system. It was relatively new, well compared to the systems around it, so lack of investment wasn't the issue.
it's a system used all over the place. There are no COTS products in ATC but this one is as close as it gets.
It is the sacrificial lamb in the NATS architecture, it processes and sanitises data before it gets to the more critical systems. It does have main and fallback running the same software.
eglnyt is offline  
Old 6th Sep 2023, 08:43
  #256 (permalink)  
 
Join Date: Mar 2016
Location: Location: Location
Posts: 57
Received 0 Likes on 0 Posts
Originally Posted by eglnyt
it's a system used all over the place. There are no COTS products in ATC but this one is as close as it gets.
The report states the particular process that failed was unique to NATS. No other FPRS has to convert flight plans to NAS format - only the UK has NAS.

Good ‘ole NAS - tried and tested technology. Not your fault.
CBSITCB is offline  
Old 6th Sep 2023, 08:53
  #257 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 470
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by CBSITCB
The report states the particular process that failed was unique to NATS. No other FPRS has to convert flight plans to NAS format - only the UK has NAS.

Good ‘ole NAS - tried and tested technology. Not your fault.
If it failed in the bit that does the conversion then technically you can still blame NAS because of the need to have that function. But it appears to have failed whilst parsing the route which I suspect may be one of their more standard software modules.as lots of the variants will need to do that. That said IFPS which is a relative must do some of that in order to know when to send the plan to each state but it didn't fail.

Last edited by eglnyt; 6th Sep 2023 at 08:53. Reason: typo
eglnyt is offline  
Old 6th Sep 2023, 09:02
  #258 (permalink)  
 
Join Date: Nov 2008
Location: Scotland
Posts: 16
Likes: 0
Received 0 Likes on 0 Posts
Frequentis Comsoft from 2018 - based on earlier good experience

https://www.airport-suppliers.com/su...-plan-gateway/
tbwtg is offline  
Old 6th Sep 2023, 09:05
  #259 (permalink)  
 
Join Date: Jul 2013
Location: UK
Posts: 22
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Flying Wild
The report, submitted earlier this week to the Civil Aviation Authority, identified an issue with a flight plan processing sub-system called Flight Plan Reception Suite Automated – Replacement (FPRSA-R).

A small but important part of NATS’ overall air traffic control technical infrastructure, it was found to have encountered an extremely rare set of circumstances presented by a flight plan that included two identically named, but separate waypoint markers outside of UK airspace.

This led to a ‘critical exception’ whereby both the primary system and its backup entered a fail-safe mode. The report details how, in these circumstances, the system could not reject the flight plan without a clear understanding of what possible impact it may have had. Nor could it be allowed through and risk presenting air traffic controllers with incorrect safety critical information. On the day, the time taken to recover was driven by the need to identify the problem and the specific data, isolate and remove it in a controlled way, and then test it to ensure it could be returned safely into operation.

This scenario had never been encountered before, with the system having previously processed more than 15 million flight plans over the 5 years it has been in service. Steps have been taken to ensure the incident cannot be repeated.
It seems that internationally there is no requirement for waypoints to be uniquely named if I've understood

https://aviation.stackexchange.com/q...supposed-to-be

correctly

What can possibly go wrong? It's not like it is an unknown issue.


ExRR is offline  
Old 6th Sep 2023, 09:18
  #260 (permalink)  
 
Join Date: Dec 2015
Location: Budapest
Posts: 275
Received 108 Likes on 59 Posts
, the system could not reject the flight plan without a clear understanding of what possible impact it may have had
​​​​​​​That's the bit I don't understand.
Expatrick is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service

Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.