U.K. NATS Systems Failure
Join Date: Oct 2004
Location: Southern England
Posts: 485
Likes: 0
Received 0 Likes
on
0 Posts
NATS is the only ANSP that has ever operated the "Swanwick system" which may also be referred to as NERC or the LACC system. It was part of that system which failed in 2014. I can't rule out a failure of that system as the issue this time but I would expect the regulations and operational effect to be different than they were. The En-Route operation has several different civil operations, all seem to have been affected but only one of those uses that system. There are several other systems in the Flight Data thread before it gets to the "Swanwick system". My money is on one of those. One of them is NAS which is only used by NATS in its current form. The others are variations of systems used all over the World.
I expect the CEO has spent the weekend locked in a room with a few people attempting the futile task of condensing a complex technical report into something the Secretary of State and the average Daily Mail reader can understand. Pointless really because the minister's catchy three word response written and the Mail editorial have probably already been written.
I expect the CEO has spent the weekend locked in a room with a few people attempting the futile task of condensing a complex technical report into something the Secretary of State and the average Daily Mail reader can understand. Pointless really because the minister's catchy three word response written and the Mail editorial have probably already been written.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
I expect the CEO has spent the weekend locked in a room with a few people attempting the futile task of condensing a complex technical report into something the Secretary of State and the average Daily Mail reader can understand. Pointless really because the minister's catchy three word response written and the Mail editorial have probably already been written.
But, you have to be careful eginyt, it sounds like you just just want this problem to go away. What if post-enquiry the headline is (justifiably): NATS System Failure 'Inevitable' ?
Join Date: Oct 2004
Location: Southern England
Posts: 485
Likes: 0
Received 0 Likes
on
0 Posts
That headline is justified.The NATS system failed. Dodgy French data or not this was a failing of a NATS system. We know it failed, the investigation will need to explain why but far more important is which of the following three cases applies.
Either:
That failure was unforeseen, in my experience unlikely for NAS for which there are precursor failures, and I would hope unlikely for the other systems in the thread given the obvious possibility, however rare, of common software failure.
The failure was foreseen but the impact was not correctly assessed
The failure was foreseen and the impact correctly assessed but the controls expected to contain such a failure didn't work as expected.
It's always a bit tricky for systems like this because of the interaction between flow regulation to maintain safety and the business impact that results from that regulation.
Either:
That failure was unforeseen, in my experience unlikely for NAS for which there are precursor failures, and I would hope unlikely for the other systems in the thread given the obvious possibility, however rare, of common software failure.
The failure was foreseen but the impact was not correctly assessed
The failure was foreseen and the impact correctly assessed but the controls expected to contain such a failure didn't work as expected.
It's always a bit tricky for systems like this because of the interaction between flow regulation to maintain safety and the business impact that results from that regulation.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
That headline is justified.The NATS system failed. Dodgy French data or not this was a failing of a NATS system. We know it failed, the investigation will need to explain why but far more important is which of the following three cases applies.
Either:
That failure was unforeseen, in my experience unlikely for NAS for which there are precursor failures, and I would hope unlikely for the other systems in the thread given the obvious possibility, however rare, of common software failure.
The failure was foreseen but the impact was not correctly assessed
The failure was foreseen and the impact correctly assessed but the controls expected to contain such a failure didn't work as expected.
It's always a bit tricky for systems like this because of the interaction between flow regulation to maintain safety and the business impact that results from that regulation.
Either:
That failure was unforeseen, in my experience unlikely for NAS for which there are precursor failures, and I would hope unlikely for the other systems in the thread given the obvious possibility, however rare, of common software failure.
The failure was foreseen but the impact was not correctly assessed
The failure was foreseen and the impact correctly assessed but the controls expected to contain such a failure didn't work as expected.
It's always a bit tricky for systems like this because of the interaction between flow regulation to maintain safety and the business impact that results from that regulation.
a. Not possible, you can’t build without fallbacks and say a crash can’t happen, NATS was living on hope (and tbh, tempting fate)
b. Ditto, as the system was never fully stress
tested
c. What controls; there were no fallbacks (tbc)?
Conclusion, the headline’s fully justified. Let’s see.
Join Date: Oct 2004
Location: Southern England
Posts: 485
Likes: 0
Received 0 Likes
on
0 Posts
The word is there were no fallbacks. So, assuming that’s correct, the options don’t look good:
a. Not possible, you can’t build without fallbacks and say a crash can’t happen, NATS was living on hope (and tbh, tempting fate)
b. Ditto, as the system was never fully stress
tested
c. What controls; there were no fallbacks (tbc)?
Conclusion, the headline’s fully justified. Let’s see.
a. Not possible, you can’t build without fallbacks and say a crash can’t happen, NATS was living on hope (and tbh, tempting fate)
b. Ditto, as the system was never fully stress
tested
c. What controls; there were no fallbacks (tbc)?
Conclusion, the headline’s fully justified. Let’s see.
We don't know if operating at its limits was an issue so the latter point may or may not be relevant. We know the number of atomic functions was the issue in 2014 but that limitation is unique to the NERC system and different boundary conditions apply elsewhere in the Flight Data thread. We don't know which system so how do we know whether those conditions were tested?
We know any fallback was ineffective but not what it was. I would expect the first level to be provided by an identical system because that is normal in this thread. I'd expect the last fallback to be manual fallback. We don't know what if anything was expected to happen between those.
Join Date: Aug 2023
Location: England
Posts: 7
Likes: 0
Received 0 Likes
on
0 Posts
The failure was foreseen and the impact correctly assessed but the controls expected to contain such a failure didn't work as expected.
It's always a bit tricky for systems like this because of the interaction between flow regulation to maintain safety and the business impact that results from that regulation.
It's always a bit tricky for systems like this because of the interaction between flow regulation to maintain safety and the business impact that results from that regulation.
What is less certain is how the backup plan worked this time. It looks to me like the normal backup plan of manual operation, whilst safe, went a bit awry in this case as the impact was longer lasting. Or was it just that being a bank holiday, with its high pax volumes, delays were longer?
I fail to see how parliament can really help here. NATS is semi-private and semi-controlled by the UK airlines. Unless government said “Here is £1b to build a duplicate system or lets structure NATS in a completely different way” it’s up to the airlines to decide how much NATS spends to upgrade its systems and pass that cost back to the airlines and customers through the route charges. Neither government (even a Labour one) nor the airlines are likely to agree to say £1b when NATS operations are actually amongst the most reliable in the world. Especially as the ATCOs don’t go on strike, unlike many others.
By the way, a week on, have all the pax that were delayed got alternative flights now?
I work in the world of complex software, testing is a nightmare. As others have asked, at what point do you stop? How many bugs do you release (bearing in mind you may not know about them)? These are questions we're constantly asking and we never ever get to a stage where we satisfy everyone.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
If I understand it correctly, the data provided to NATS has been pre-processed and so any errors (mistakes) should have been caught by the system doing the pre-processing. If my understanding is correct, I would expect that the assumption is that the data is now ONLY valid data, so this failure case isn't as straight forward as you may believe.
I work in the work of complex software, testing is a nightmare. As others have asked, at what point do you stop? How many bugs do you release (bearing in mind you may not know about them)? These are questions we're constantly asking and we never ever get to a stage where we satisfy everyone.
I work in the work of complex software, testing is a nightmare. As others have asked, at what point do you stop? How many bugs do you release (bearing in mind you may not know about them)? These are questions we're constantly asking and we never ever get to a stage where we satisfy everyone.
'at what point do you stop?'. Er, when you've built one fall back, and preferably two - if, as you say, this is really a mission critical system.
Join Date: Oct 2004
Location: Southern England
Posts: 485
Likes: 0
Received 0 Likes
on
0 Posts
This answer (aka 'excuse') has been repeated so often, ie 'testing's soooo complicated, we couldn't possibly capture every permutation (which actually means not process an incorrectly formatted message). Really??
'at what point do you stop?'. Er, when you've built one fall back, and preferably two - if, as you say, this is really a mission critical system.
'at what point do you stop?'. Er, when you've built one fall back, and preferably two - if, as you say, this is really a mission critical system.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
You’re scenario’s surely hypothetical as I can’t imagine anyone - with the facts - would vote down building proper fall backs, whether it meant saving hundreds of millions of pounds in lost business, and costs, let alone more in reputational damage.
In fact, I’m sure they’d assume no one would consider building such a system WITHOUT fall backs in place.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
Join Date: Oct 2004
Location: Southern England
Posts: 485
Likes: 0
Received 0 Likes
on
0 Posts
I must have imagined the scrutiny applied to the NATS Investment Plan at every licence renewal. Maybe it was just a bad dream. Similarly the need to halt the investment programme at the start of the pandemic because without income there was no way to fund it.
If it turns out the system, which is still unknown, didn't have appropriate fallbacks then any inquiry should examine why that is including any issues about funding any resilience required.
If it turns out the system, which is still unknown, didn't have appropriate fallbacks then any inquiry should examine why that is including any issues about funding any resilience required.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
I must have imagined the scrutiny applied to the NATS Investment Plan at every licence renewal. Maybe it was just a bad dream. Similarly the need to halt the investment programme at the start of the pandemic because without income there was no way to fund it.
If it turns out the system, which is still unknown, didn't have appropriate fallbacks then any inquiry should examine why that is including any issues about funding any resilience required.
If it turns out the system, which is still unknown, didn't have appropriate fallbacks then any inquiry should examine why that is including any issues about funding any resilience required.
NATS took £1.5BILLION in debt financing during Covid (and still sacked all 120 ‘apprentice’ ATCOs)!
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
Join Date: Oct 2004
Location: Southern England
Posts: 485
Likes: 0
Received 0 Likes
on
0 Posts
I'm a bit old for Corporate Comms and lacking in the Facebook/Twitter skills probably required nowadays.
I wouldn't claim to understand NATS financing, I got confused when the Airline Group bought half of NATS for several hundred million and somehow NATS paid the Treasury. However I think you need to look at total borrowing before and after that deal and how much of that profit over the years was reinvested.
I am however in awe of the gentleman who somehow sold bonds for a company with no real income in the middle of a pandemic. I'm not surprised they gave him a pay rise.
I wouldn't claim to understand NATS financing, I got confused when the Airline Group bought half of NATS for several hundred million and somehow NATS paid the Treasury. However I think you need to look at total borrowing before and after that deal and how much of that profit over the years was reinvested.
I am however in awe of the gentleman who somehow sold bonds for a company with no real income in the middle of a pandemic. I'm not surprised they gave him a pay rise.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
I'm a bit old for Corporate Comms and lacking in the Facebook/Twitter skills probably required nowadays.
I wouldn't claim to understand NATS financing, I got confused when the Airline Group bought half of NATS for several hundred million and somehow NATS paid the Treasury. However I think you need to look at total borrowing before and after that deal and how much of that profit over the years was reinvested.
I am however in awe of the gentleman who somehow sold bonds for a company with no real income in the middle of a pandemic. I'm not surprised they gave him a pay rise.
I wouldn't claim to understand NATS financing, I got confused when the Airline Group bought half of NATS for several hundred million and somehow NATS paid the Treasury. However I think you need to look at total borrowing before and after that deal and how much of that profit over the years was reinvested.
I am however in awe of the gentleman who somehow sold bonds for a company with no real income in the middle of a pandemic. I'm not surprised they gave him a pay rise.
Pegase Driver
That is one of the main issue of privatized ATC, no revenue,? reduce controllers workforce, , traffic restarts , no staff.as it takes minimum 3 years from recruitment to fully validated .Same old , same old....
Join Date: Oct 2004
Location: Southern England
Posts: 485
Likes: 0
Received 0 Likes
on
0 Posts
The CEO renumeration did dip when income dipped, remember most of financial year 2019/2020 wasn't affected and the critical summer period was the best ever. It has rebounded as traffic did, to my mind rather better than NATS staff in general who actually gave back their 2019/20 payrise when the pandemic hit.
Join Date: Oct 2004
Location: Southern England
Posts: 485
Likes: 0
Received 0 Likes
on
0 Posts
a bit off topic but someone knows what happened to those 120 trainees. were they ever rehired now that traffic is back to pre-Covid 2019 levels ? If not , how is NATS going to cope with the planned retirements in the next 2-3 years ?.
That is one of the main issue of privatized ATC, no revenue,? reduce controllers workforce, , traffic restarts , no staff.as it takes minimum 3 years from recruitment to fully validated .Same old , same old....
That is one of the main issue of privatized ATC, no revenue,? reduce controllers workforce, , traffic restarts , no staff.as it takes minimum 3 years from recruitment to fully validated .Same old , same old....
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
I believe those that wished were put at the front of the queue when abinitio recruitment was restarted recently. Trainees have to factor in living on a minimal income during training for later reward and not all of the 120 would have been in a position to have another go.
You put a fantastic burnish on everything NATS does, with real detail (hence 'Corporate Comms'!), but this was clearly the wrong thing to do.