U.K. NATS Systems Failure
Join Date: Sep 2008
Location: River Thames
Age: 69
Posts: 27
Likes: 0
Received 0 Likes
on
0 Posts
Clearly, why a single point of failure has such an impact is a reasonable question, especially as several similar events have previously occurred (google for public domain examples of investigations). One hopes that e.g. a corrupt datalink message to an A350 wouldn't cause reversion to manual operation.
Join Date: Oct 2004
Location: Southern England
Posts: 480
Likes: 0
Received 0 Likes
on
0 Posts
I would hope that a number of people had a very good idea of what happened & why before the decision was made to remove flow yesterday afternoon. If not then NATS crisis management has changed for the worse since 2014. They will hopefully share that with the wider populace in due course although I'd love to be in the room when the Transport Secretary is presented with the report if it's as technical as they normally are.
Not sure when this was released, lots of puff little substance - report to Sec of State for Transport to be delivered on Monday.
Statement from NATS CEO Martin Rolfe, 29 August “I would like to apologise again for our technical failure yesterday. While we resolved the problem quickly, I am very conscious that the knock-on effects at such a busy time of year are still being felt by many people travelling in and out of the UK.
“I would like to reassure everyone that since yesterday afternoon all of our systems have been running normally to support airline and airport operations as they recover from this incident.
“NATS exists to allow everyone flying in UK airspace to do so safely. Our systems enable our air traffic controllers to deliver this service all year round. These have several levels of backup and allow us to manage around 2 million flights per year in some of the busiest and most complex airspace in the world safely and efficiently.
“Very occasionally technical issues occur that are complex and take longer to resolve. In the event of such an issue our systems are designed to isolate the problem and prioritise continued safe air traffic control.
“This is what happened yesterday. At no point was UK airspace closed but the number of flights was significantly reduced.
“Initial investigations into the problem show it relates to some of the flight data we received. Our systems, both primary and the back-ups, responded by suspending automatic processing to ensure that no incorrect safety-related information could be presented to an air traffic controller or impact the rest of the air traffic system. There are no indications that this was a cyber-attack.
“We have well established procedures, overseen by the CAA, to investigate incidents. We are already working closely with them to provide a preliminary report to the Secretary of State for Transport on Monday. The conclusions of this report will be made public.
“I would like again to apologise to everyone who has been affected.”
Statement from NATS CEO Martin Rolfe, 29 August “I would like to apologise again for our technical failure yesterday. While we resolved the problem quickly, I am very conscious that the knock-on effects at such a busy time of year are still being felt by many people travelling in and out of the UK.
“I would like to reassure everyone that since yesterday afternoon all of our systems have been running normally to support airline and airport operations as they recover from this incident.
“NATS exists to allow everyone flying in UK airspace to do so safely. Our systems enable our air traffic controllers to deliver this service all year round. These have several levels of backup and allow us to manage around 2 million flights per year in some of the busiest and most complex airspace in the world safely and efficiently.
“Very occasionally technical issues occur that are complex and take longer to resolve. In the event of such an issue our systems are designed to isolate the problem and prioritise continued safe air traffic control.
“This is what happened yesterday. At no point was UK airspace closed but the number of flights was significantly reduced.
“Initial investigations into the problem show it relates to some of the flight data we received. Our systems, both primary and the back-ups, responded by suspending automatic processing to ensure that no incorrect safety-related information could be presented to an air traffic controller or impact the rest of the air traffic system. There are no indications that this was a cyber-attack.
“We have well established procedures, overseen by the CAA, to investigate incidents. We are already working closely with them to provide a preliminary report to the Secretary of State for Transport on Monday. The conclusions of this report will be made public.
“I would like again to apologise to everyone who has been affected.”
Hal. HAL!Last edited by Clare Prop; 30th Aug 2023 at 06:45.
That logic, if not an inoperative humour detector, is likely the root cause.
It still plays on any page, Oveur...
Underflow, overflow, division by zero...
Ref the recovery from these ATC strikes, I’m presuming there are no changes to the curfew timings?
If not, considering probably 500,000 passengers minimum have been affected by this issue, the airports not just Heathrow should be cut some slack. And should be the same in any other exceptional event like snow for example.
Ryanair for example are operating some “rescue flights” but with every flight running at capacity it’s going to be days before everyone gets back. Being the last week of the school holidays, flights were probably already at 95% of capacity at least in some cases and you could genuinely see some people not getting back within 7 days.
You need to try and get some more capacity in the system somewhere.
If not, considering probably 500,000 passengers minimum have been affected by this issue, the airports not just Heathrow should be cut some slack. And should be the same in any other exceptional event like snow for example.
Ryanair for example are operating some “rescue flights” but with every flight running at capacity it’s going to be days before everyone gets back. Being the last week of the school holidays, flights were probably already at 95% of capacity at least in some cases and you could genuinely see some people not getting back within 7 days.
You need to try and get some more capacity in the system somewhere.
Join Date: Apr 2013
Location: Neither here or there
Posts: 317
Likes: 0
Received 0 Likes
on
0 Posts
Pull the other one Rolfe, there is no one at the CAA who competently oversees what you do. They are expired relics and ex librarians who don't understand and see anything beyond their nose.
Join Date: Mar 2008
Location: London
Age: 69
Posts: 148
Likes: 0
Received 0 Likes
on
0 Posts
The Walmsley Report after the 2014 incident https://www.caa.co.uk/media/r42hircd...port-2-0-1.pdf included this snippet on NATS paying compensation
For a techie, human-system view of problems with high tech failures; complexity, see:-
https://snafucatchers.github.io
https://snafucatchers.github.io
Join Date: Jan 2007
Location: Oblivion
Posts: 189
Likes: 0
Received 0 Likes
on
0 Posts
If Airlines had enough resilience to produce manual flight plans would this have reduced their own impact. And reputation hit.
Depending heavily on an automated system with little back up and resilience down to aggressively minimum levels really doesnt seem to make much sense.
Suppose finger pointing at NATs in a blame game post event would be about the only defence Airlines would have if they were inept enough to have little resilience.
No automated system is infallible. Its unlikely it ever will be either.
Depending heavily on an automated system with little back up and resilience down to aggressively minimum levels really doesnt seem to make much sense.
Suppose finger pointing at NATs in a blame game post event would be about the only defence Airlines would have if they were inept enough to have little resilience.
No automated system is infallible. Its unlikely it ever will be either.
Join Date: Jul 2008
Location: Poole
Posts: 11
Likes: 0
Received 0 Likes
on
0 Posts
Never mind SRA what about PAR?
First did a PAR in a Chipmunk. "Turn 2 degrees right". With a big glass compass between my legs calibrated every 10 degrees I complained to my instructor. " Well turn 12 degrees right then 10 degrees left".
Not strictly on topic but happy days.
Join Date: May 2001
Location: England
Posts: 1,904
Likes: 0
Received 0 Likes
on
0 Posts
So, a faulty bit of data basically bombed out the application service responsible for deciphering flight plan requests? In software development we have something called "Exception Handling". You will be familiar with the below kind of message. It basically causes the application to crash because the error type was never coded for/handled. Let's say you entered a "O" where a "0" (zero) was expected. It seems highly unlikely to me that in 10+ years, a corrupt bit of data didn't find its way to the same application service before. There is something else behind this that they've not divulged yet.
Keeping Danny in Sandwiches
Join Date: May 1999
Location: UK
Age: 76
Posts: 1,294
Likes: 0
Received 0 Likes
on
0 Posts
"If Airlines had enough resilience to produce manual flight plans would this have reduced their own impact. And reputation hit."
In my day they were called Airline Pilots. Normal operation in the 1970's.
In my day they were called Airline Pilots. Normal operation in the 1970's.
Join Date: Apr 2006
Location: ESSL
Age: 79
Posts: 61
Likes: 0
Received 0 Likes
on
0 Posts
Checking input
Join Date: Jul 2008
Location: Ex-EUROCONTROL land
Age: 75
Posts: 97
Received 0 Likes
on
0 Posts
Re Murty’s post #80
IFPS had, until recently, two Flight Plan Processing Units - IFPU1 at Haaren (Brussels) and IFPU2 at Bretigny (Paris). Each unit was responsible
for half of Europe. We at Bretigny were the responsible Unit for aircraft departing from Azores in the west, through Portugal, Spain, France, Italy, Greece, Turkey and the Med, along with airfields around the Med, for traffic entering European airspace.
We were not a back-up Unit for Brussels, rather a stand-alone Unit. Each Unit, however, backed up the other in case of a failure at either of the two Units. In fact, there were more “outages” at IFPU1 in Brussels than at Bretigny…
In 2006/7, the designators of the Units was changed to FP1 & FP2. During 2020, FP2 was closed down and the total airspace commitment transferred to FP1
IFPS had, until recently, two Flight Plan Processing Units - IFPU1 at Haaren (Brussels) and IFPU2 at Bretigny (Paris). Each unit was responsible
for half of Europe. We at Bretigny were the responsible Unit for aircraft departing from Azores in the west, through Portugal, Spain, France, Italy, Greece, Turkey and the Med, along with airfields around the Med, for traffic entering European airspace.
We were not a back-up Unit for Brussels, rather a stand-alone Unit. Each Unit, however, backed up the other in case of a failure at either of the two Units. In fact, there were more “outages” at IFPU1 in Brussels than at Bretigny…
In 2006/7, the designators of the Units was changed to FP1 & FP2. During 2020, FP2 was closed down and the total airspace commitment transferred to FP1
Last edited by IFPS man; 30th Aug 2023 at 10:21. Reason: Incorrect auto-prediction
Join Date: Oct 2004
Location: Southern England
Posts: 480
Likes: 0
Received 0 Likes
on
0 Posts
That Flight Plan was "checked" by at least 2 other systems before it got to where it caused the issue. There are perfectly valid flight plans which are known to cause the UK flight data processor issues and they are screened before they get there. It could be that a new one has now been added to the list.
Join Date: Jul 2007
Location: West Britain
Age: 74
Posts: 207
Likes: 0
Received 0 Likes
on
0 Posts
Back in the 70s on approach to Cork in the dear old gripper, we were asked if we would accept a PAR for controller training. My gallent captain said "yes" then leant across and turned off my ILS display. "Good traing for you boy". It would have been too but I never flew another PAR.
First did a PAR in a Chipmunk. "Turn 2 degrees right". With a big glass compass between my legs calibrated every 10 degrees I complained to my instructor. " Well turn 12 degrees right then 10 degrees left".
Not strictly on topic but happy days.
First did a PAR in a Chipmunk. "Turn 2 degrees right". With a big glass compass between my legs calibrated every 10 degrees I complained to my instructor. " Well turn 12 degrees right then 10 degrees left".
Not strictly on topic but happy days.