Go Back  PPRuNe Forums > Flight Deck Forums > Rumours & News
Reload this Page >

U.K. NATS Systems Failure

Rumours & News Reporting Points that may affect our jobs or lives as professional pilots. Also, items that may be of interest to professional pilots.

U.K. NATS Systems Failure

Old 29th Aug 2023, 21:51
  #101 (permalink)  
 
Join Date: Sep 2008
Location: River Thames
Age: 69
Posts: 27
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by ATC Watcher
As far as I know at this time , no-one really know what caused the system to crash , the root cause I mean , not yet but likely by tomorrow or in the next coming days we'll know
I'm a little confused by what you say. As mentioned upthread, it appears that the issue was a corrupt message from IFPS that the local (NATS) automated flight plan processing system couldn't reconcile. So it fell over. Without automated processing of flight plan data the NATS systems can't distribute flight data to the required sectors thus manual reversion and manual processing of flight plans is required.

Clearly, why a single point of failure has such an impact is a reasonable question, especially as several similar events have previously occurred (google for public domain examples of investigations). One hopes that e.g. a corrupt datalink message to an A350 wouldn't cause reversion to manual operation.
Janet Spongthrush is offline  
Old 29th Aug 2023, 22:01
  #102 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 488
Likes: 0
Received 0 Likes on 0 Posts
I would hope that a number of people had a very good idea of what happened & why before the decision was made to remove flow yesterday afternoon. If not then NATS crisis management has changed for the worse since 2014. They will hopefully share that with the wider populace in due course although I'd love to be in the room when the Transport Secretary is presented with the report if it's as technical as they normally are.
eglnyt is offline  
Old 29th Aug 2023, 22:06
  #103 (permalink)  
 
Join Date: Apr 2020
Location: Hampshire
Posts: 1,379
Received 149 Likes on 94 Posts
Not sure when this was released, lots of puff little substance - report to Sec of State for Transport to be delivered on Monday.

Statement from NATS CEO Martin Rolfe, 29 August “I would like to apologise again for our technical failure yesterday. While we resolved the problem quickly, I am very conscious that the knock-on effects at such a busy time of year are still being felt by many people travelling in and out of the UK.

“I would like to reassure everyone that since yesterday afternoon all of our systems have been running normally to support airline and airport operations as they recover from this incident.

“NATS exists to allow everyone flying in UK airspace to do so safely. Our systems enable our air traffic controllers to deliver this service all year round. These have several levels of backup and allow us to manage around 2 million flights per year in some of the busiest and most complex airspace in the world safely and efficiently.

“Very occasionally technical issues occur that are complex and take longer to resolve. In the event of such an issue our systems are designed to isolate the problem and prioritise continued safe air traffic control.

“This is what happened yesterday. At no point was UK airspace closed but the number of flights was significantly reduced.

“Initial investigations into the problem show it relates to some of the flight data we received. Our systems, both primary and the back-ups, responded by suspending automatic processing to ensure that no incorrect safety-related information could be presented to an air traffic controller or impact the rest of the air traffic system. There are no indications that this was a cyber-attack.

“We have well established procedures, overseen by the CAA, to investigate incidents. We are already working closely with them to provide a preliminary report to the Secretary of State for Transport on Monday. The conclusions of this report will be made public.

“I would like again to apologise to everyone who has been affected.”
SLXOwft is offline  
Old 29th Aug 2023, 22:25
  #104 (permalink)  
 
Join Date: Dec 2015
Location: U.K.
Posts: 192
Received 9 Likes on 5 Posts
A variation of Bobby Drop Tables?

https://imgs.xkcd.com/comics/exploits_of_a_mom.png




kghjfg is offline  
Old 29th Aug 2023, 22:30
  #105 (permalink)  
 
Join Date: Oct 2005
Location: Australia
Posts: 1,346
Received 268 Likes on 123 Posts
Hal. HAL!

Last edited by Clare Prop; 30th Aug 2023 at 06:45.
Clare Prop is offline  
Old 29th Aug 2023, 23:16
  #106 (permalink)  
 
Join Date: Jan 2008
Location: USA
Posts: 36
Received 0 Likes on 0 Posts
That logic, if not an inoperative humour detector, is likely the root cause.

Originally Posted by CBSITCB
You actually replied by linking back to this thread, itself. Was that to be substantive via ipso facto just because it's on post 11?

It still plays on any page, Oveur...

Underflow, overflow, division by zero...
moosepileit is offline  
Old 30th Aug 2023, 06:05
  #107 (permalink)  
 
Join Date: Dec 2011
Location: UK
Posts: 974
Received 5 Likes on 4 Posts
Ref the recovery from these ATC strikes, I’m presuming there are no changes to the curfew timings?

If not, considering probably 500,000 passengers minimum have been affected by this issue, the airports not just Heathrow should be cut some slack. And should be the same in any other exceptional event like snow for example.

Ryanair for example are operating some “rescue flights” but with every flight running at capacity it’s going to be days before everyone gets back. Being the last week of the school holidays, flights were probably already at 95% of capacity at least in some cases and you could genuinely see some people not getting back within 7 days.

You need to try and get some more capacity in the system somewhere.
Dannyboy39 is offline  
Old 30th Aug 2023, 06:45
  #108 (permalink)  
 
Join Date: Apr 2013
Location: Neither here or there
Posts: 317
Likes: 0
Received 0 Likes on 0 Posts
Pull the other one Rolfe, there is no one at the CAA who competently oversees what you do. They are expired relics and ex librarians who don't understand and see anything beyond their nose.
CW247 is offline  
Old 30th Aug 2023, 06:47
  #109 (permalink)  
 
Join Date: Mar 2016
Location: Location: Location
Posts: 59
Received 0 Likes on 0 Posts
Originally Posted by moosepileit
Was that to be substantive via ipso facto just because it's on post 11?
The link was actually to post 12. I don’t understand the rest of your comments.

Last edited by CBSITCB; 30th Aug 2023 at 07:09.
CBSITCB is offline  
Old 30th Aug 2023, 07:37
  #110 (permalink)  
 
Join Date: Mar 2008
Location: London
Age: 69
Posts: 148
Likes: 0
Received 0 Likes on 0 Posts
The Walmsley Report after the 2014 incident https://www.caa.co.uk/media/r42hircd...port-2-0-1.pdf included this snippet on NATS paying compensation

118.70 is offline  
Old 30th Aug 2023, 07:40
  #111 (permalink)  
 
Join Date: Dec 2002
Location: UK
Posts: 2,487
Likes: 0
Received 25 Likes on 12 Posts
For a techie, human-system view of problems with high tech failures; complexity, see:-

https://snafucatchers.github.io
safetypee is offline  
Old 30th Aug 2023, 07:47
  #112 (permalink)  
 
Join Date: Oct 1999
Location: the dark side
Posts: 1,122
Received 3 Likes on 3 Posts
Originally Posted by Dannyboy39
Ref the recovery from these ATC strikes, I’m presuming there are no changes to the curfew timings?
Correct, it’s unlikely to have any changes to curfew. The same with non Uk airports, which restricts flexibility in running extra flights, or any delayed legs, eg Amsterdam Geneva Zurich etc etc.
jumpseater is offline  
Old 30th Aug 2023, 08:09
  #113 (permalink)  
 
Join Date: Jan 2007
Location: Oblivion
Posts: 192
Likes: 0
Received 0 Likes on 0 Posts
If Airlines had enough resilience to produce manual flight plans would this have reduced their own impact. And reputation hit.

Depending heavily on an automated system with little back up and resilience down to aggressively minimum levels really doesnt seem to make much sense.

Suppose finger pointing at NATs in a blame game post event would be about the only defence Airlines would have if they were inept enough to have little resilience.

No automated system is infallible. Its unlikely it ever will be either.

Captain Boycott is offline  
Old 30th Aug 2023, 08:22
  #114 (permalink)  
 
Join Date: Jul 2008
Location: Poole
Posts: 11
Likes: 0
Received 0 Likes on 0 Posts
Never mind SRA what about PAR?

Originally Posted by classicwings
Love it!! Back in the 90's, my Dad, who was then an approach radar ATCO at West Drayton told me when he occasionally used to practice doing SRA's with pilots, he was certain that some crews had the ILS dialled up and followed that rather than ATC instructions!
Back in the 70s on approach to Cork in the dear old gripper, we were asked if we would accept a PAR for controller training. My gallent captain said "yes" then leant across and turned off my ILS display. "Good traing for you boy". It would have been too but I never flew another PAR.
First did a PAR in a Chipmunk. "Turn 2 degrees right". With a big glass compass between my legs calibrated every 10 degrees I complained to my instructor. " Well turn 12 degrees right then 10 degrees left".
Not strictly on topic but happy days.
100Series is offline  
Old 30th Aug 2023, 08:25
  #115 (permalink)  
 
Join Date: May 2001
Location: England
Posts: 1,904
Likes: 0
Received 0 Likes on 0 Posts
So, a faulty bit of data basically bombed out the application service responsible for deciphering flight plan requests? In software development we have something called "Exception Handling". You will be familiar with the below kind of message. It basically causes the application to crash because the error type was never coded for/handled. Let's say you entered a "O" where a "0" (zero) was expected. It seems highly unlikely to me that in 10+ years, a corrupt bit of data didn't find its way to the same application service before. There is something else behind this that they've not divulged yet.

Superpilot is offline  
Old 30th Aug 2023, 08:35
  #116 (permalink)  

Keeping Danny in Sandwiches
 
Join Date: May 1999
Location: UK
Age: 76
Posts: 1,294
Likes: 0
Received 0 Likes on 0 Posts
Talking

"If Airlines had enough resilience to produce manual flight plans would this have reduced their own impact. And reputation hit."

In my day they were called Airline Pilots. Normal operation in the 1970's.
sky9 is offline  
Old 30th Aug 2023, 08:43
  #117 (permalink)  
 
Join Date: Apr 2006
Location: ESSL
Age: 79
Posts: 61
Likes: 0
Received 0 Likes on 0 Posts
Checking input

Originally Posted by togsdragracing
An old tecchie writes: is there not some sort of validity check carried out on submitted plans? Be it syntax, format, whatever.
My PACS system that has been in use by airlines for 30 years has always had a simple check to check if the flight data is viable before making the calculation, any fault is flagged and sector not allowed with point of failure displayed.
FlightCosting is offline  
Old 30th Aug 2023, 08:48
  #118 (permalink)  
 
Join Date: Jul 2008
Location: Ex-EUROCONTROL land
Age: 76
Posts: 97
Received 0 Likes on 0 Posts
Re Murty’s post #80
IFPS had, until recently, two Flight Plan Processing Units - IFPU1 at Haaren (Brussels) and IFPU2 at Bretigny (Paris). Each unit was responsible
for half of Europe. We at Bretigny were the responsible Unit for aircraft departing from Azores in the west, through Portugal, Spain, France, Italy, Greece, Turkey and the Med, along with airfields around the Med, for traffic entering European airspace.

We were not a back-up Unit for Brussels, rather a stand-alone Unit. Each Unit, however, backed up the other in case of a failure at either of the two Units. In fact, there were more “outages” at IFPU1 in Brussels than at Bretigny…
In 2006/7, the designators of the Units was changed to FP1 & FP2. During 2020, FP2 was closed down and the total airspace commitment transferred to FP1

Last edited by IFPS man; 30th Aug 2023 at 10:21. Reason: Incorrect auto-prediction
IFPS man is offline  
Old 30th Aug 2023, 08:51
  #119 (permalink)  
 
Join Date: Oct 2004
Location: Southern England
Posts: 488
Likes: 0
Received 0 Likes on 0 Posts
That Flight Plan was "checked" by at least 2 other systems before it got to where it caused the issue. There are perfectly valid flight plans which are known to cause the UK flight data processor issues and they are screened before they get there. It could be that a new one has now been added to the list.
eglnyt is offline  
Old 30th Aug 2023, 08:53
  #120 (permalink)  
 
Join Date: Jul 2007
Location: West Britain
Age: 74
Posts: 211
Likes: 0
Received 1 Like on 1 Post
Originally Posted by 100Series
Back in the 70s on approach to Cork in the dear old gripper, we were asked if we would accept a PAR for controller training. My gallent captain said "yes" then leant across and turned off my ILS display. "Good traing for you boy". It would have been too but I never flew another PAR.
First did a PAR in a Chipmunk. "Turn 2 degrees right". With a big glass compass between my legs calibrated every 10 degrees I complained to my instructor. " Well turn 12 degrees right then 10 degrees left".
Not strictly on topic but happy days.
I seem to remember the Chippy having a vertical DI?
BristolScout is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service

Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.