Go Back  PPRuNe Forums > Flight Deck Forums > Rumours & News
Reload this Page >

Eurocontrol problems

Rumours & News Reporting Points that may affect our jobs or lives as professional pilots. Also, items that may be of interest to professional pilots.

Eurocontrol problems

Old 3rd Apr 2018, 13:37
  #1 (permalink)  
Thread Starter
 
Join Date: Mar 2007
Location: Here and there
Posts: 2,781
Likes: 0
Received 1 Like on 1 Post
Eurocontrol problems

The ETFMS system has failed and slot allocation by the system is not available. Expect big delays.

ATTN : ALL FMPS AND NOMINATED TOWERS
INFO : OTHER ATS UNITS AND ALL OPERATORS

ETFMS UNAVAILABILITY

DUE TO ETFMS SYSTEM FAILURE,
SLOT ALLOCATION BY THE ETFMS SYSTEM IS NOT AVAILABLE. ALL ATTEMPTS TO REACTIVATE ETFMS PROVED UNSUCCESSFUL.
IN ACCORDANCE WITH LOCAL PROCEDURES FMPs AND NOMINATED ATC TOWERS WILL HAVE TO APPLY THE ATFCM PROCEDURAL CONTINGENCY PLAN.

CONTINGENCY PLAN START TIME (CST) : WITH IMMEDIATE EFFECT
(TIME OF ENTERING THE OPERATIONAL CONTINGENCY PHASE)

-TOWERS WILL CONTINUE TO HONOUR CTOT WHICH FALL WITHIN THE PERIOD UP TO CST. AFTER THIS TIME OPERATION WILL BE BASED ON DEPARTURE INTERVALS ARRANGED BY NOMINATED TOWERS/FMPs. CTOT WHICH FALL IN THE PERIOD AFTER THE CST ARE IGNORED.

-THE EXTENDED SLOT TOLERANCE WINDOW (-5 TO +20 MIN) MENTIONED IN THE ALERT FLASH WILL CONTINUE TO APPLY. WHERE THE APPLICATION OF THE SLOT TOLERANCE WINDOW TAKES FLIGHTS INTO THE OPERATIONAL CONTINGENCY PHASE THE FLIGHTS WILL STILL BE ALLOWED TO DEPART AS
LONG AS THEY REMAIN WITHIN THE EXTENDED SLOT TOLERANCE WINDOW.

-FLIGHTS DEPARTING FROM NON-NOMINATED AIRFIELDS ARE NOT AFFECTED BY THE CONTINGENCY PLAN.

-FLIGHT PLAN AND FLIGHT PLAN UPDATE MSGS (INCLUDING CNL) ARE TO BE SENT TO THE IFPS UNITS THROUGHOUT THE WHOLE PERIOD OF OPERATION OF THE ATFCM CONTINGENCY PLAN.

-FOR ADDITIONAL INFORMATION PLEASE CALL YOUR LOCAL FMP.

NMOC Brussels
tubby linton is offline  
Old 3rd Apr 2018, 13:45
  #2 (permalink)  
 
Join Date: Jan 2004
Location: London
Posts: 650
Received 9 Likes on 5 Posts
More here:

Eurocontrol public portal
Del Prado is offline  
Old 3rd Apr 2018, 14:19
  #3 (permalink)  
 
Join Date: Jun 2001
Location: Rockytop, Tennessee, USA
Posts: 5,898
Likes: 0
Received 1 Like on 1 Post
Hopefully some good news:

The issue with the ETFMS has been identified and work is progressing to recover the system, which is expected to occur late this evening. Aircraft operators are requested to refile any flight plans for flights not yet operated and that were originally filed before 10.26UTC
Airbubba is offline  
Old 3rd Apr 2018, 15:08
  #4 (permalink)  
AF1
 
Join Date: Sep 2000
Location: Aotearoa
Age: 54
Posts: 63
Likes: 0
Received 0 Likes on 0 Posts
Update from OpsGroup:
---
Hi everyone,

The European ATC system is in crisis mode at the moment, due to a failure of the slot computer with multiple knock on effects. Here is our update of the situation:

OPSGROUP Update - 1455Z Tuesday
- Based on direct news from Eurocontrol, a Conference Call completed at 1415Z, and Operator input to Opsgroup

1. The system that allocates ATC slots to flights, and therefore manages the flow of traffic across Europe, failed today at 1026 UTC. It’s called the ETFMS (Enhanced Tactical Flow Management System), but aka “The Slot Computer”

2. There is a Contingency Plan for this situation. Airports are supposed to use this, which gives a quick table of departure intervals allowed according to the destination. You can view the plan here and see what it looks like for the airport you’re using: https://ops.group/files/NM.Contingency.Summer.2018.pdf

3. Some airlines reported that Istanbul, amongst others, were holding all departures. During the call, Istanbul at least were informed there was no need to do this, but others may still be unclear as to how to handle this.

4. All flight plans filed before 1026Z have been lost. You need to re-file your FPL if filed before this time, but in reality, it’s probably a good idea to re-file any FPL you have for today or tomorrow, to make sure it’s OK.

5. Eurocontrol says 10% capacity reduction across the whole of Europe, and expect ’high delays’. Actual delay numbers - usually available on the NOP - will be inaccurate because of all the missing FPL’s.

6. Eurocontrol will only switch back on the Slot computer once they reach a critical mass of filed flight plans - so the Contingency Plan will continue for all of today and probably into tomorrow. Eurocontrol are calling round the 70 main airports to make sure they know what they’re supposed to do.

7. Normally, Eurocontrol will re-address your FPL to ATC Centres outside the IFPZ. This needs to be done manually, ie. find the FIR’s you will cross, get their AFTN address like HECCZQZX, and send them your FPL - if you can get that info.

8. The actual system failure has been fixed as of about 1400Z, but Eurocontrol says the ATC system will not recover until well into tomorrow, earliest.

9. The Eurocontrol hotline for operators to call for more information about the contingency plan is: +32 496 XXX XXX. Go easy on them, they’ll be swamped.

10. Ramps may be congested, there may be arrival delays, so carry extra fuel.

For updates:
Monitor the Eurocontrol NOP
Monitor OpsGroup Slack - the #todays-ops channel

Kind regards,
The Team.
Quick links - OPSGROUP members:
Login to the Dashboard
Launch OpsGroup slack
Check Airport Spy
Upgrade to Team or Flight Department membership
Check the FSB blog
Email the OpsGroup Team
AF1 is offline  
Old 3rd Apr 2018, 15:56
  #5 (permalink)  
Pegase Driver
 
Join Date: May 1997
Location: Europe
Age: 73
Posts: 3,671
Likes: 0
Received 0 Likes on 0 Posts
All flight plans filed before 1026Z have been lost
That I have difficulty with . That would mean that both IFPS failed , the one in Brussels and the one in Bretigny at the same time ? or that the one in Bretigny could not take over, but that was the main reason why it was designed in the first place.
Both systems were supposed to be totally independent and act as a back up to one another for that very reason.
I will be curious to read the report and the explanations.
ATC Watcher is offline  
Old 3rd Apr 2018, 17:19
  #6 (permalink)  
 
Join Date: Aug 2013
Location: PA
Age: 59
Posts: 30
Likes: 0
Received 0 Likes on 0 Posts
damn.......

they probably just figured out that the information is not mirrored to the fail-over system...

or, the Russians have hacked the aviation system...again.

Last edited by underfire; 3rd Apr 2018 at 17:43.
underfire is offline  
Old 3rd Apr 2018, 20:23
  #7 (permalink)  
 
Join Date: Jul 2005
Location: Centre of old Europe
Posts: 59
Likes: 0
Received 0 Likes on 0 Posts
Latest info network manager

MESSAGE FROM DIRECTOR NETWORK MANAGER

EUROCONTROL very much regrets the disruption that has been caused to passengers and airlines due to today’s outage of the Enhanced Tactical Flow Management System (ETFMS).

The contingency procedures put in place ensured that safety was not compromised at any time.

As of 18.00 UTC, following the resumption of the ETFMS, the EUROCONTROL Network Manager has restarted normal flow management operations. This has been done after extensive internal testing and in coordination with airports, airlines and air traffic control across Europe and beyond.

In over 20 years of operation, the ETFMS has only had one other outage which occurred in 2001. The system currently manages up to 36,000 flights a day.


NMOC Brussels
songbird29 is offline  
Old 3rd Apr 2018, 22:05
  #8 (permalink)  
 
Join Date: Jun 2002
Location: Wor Yerm
Age: 68
Posts: 4
Likes: 0
Received 0 Likes on 0 Posts
It’s been a brilliant day for me. I hope EuroControl never fix their wretched systems. Over four sectors our longest departure delay was 11 minutes, our latest arrival delay was 12 minutes. The Karma was ATC having to work harder, like we have to do EVERY day in order to minimise the delays we are given.

PM

ps. I bet EuroControl will still charge full whack for their “Service”.
Piltdown Man is offline  
Old 4th Apr 2018, 02:50
  #9 (permalink)  
 
Join Date: Jul 2013
Location: Australia
Posts: 305
Received 7 Likes on 5 Posts
Originally Posted by underfire
damn.......

they probably just figured out that the information is not mirrored to the fail-over system...

or, the Russians have hacked the aviation system...again.
Or the errors were propagated over to the failover system.

Happened to the Australian Tax Office. They lost everything due to data corruption in the primary SAN being mirrored over to the backup SAN.
RickNRoll is online now  
Old 4th Apr 2018, 08:40
  #10 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by ATC Watcher
That I have difficulty with . That would mean that both IFPS failed , the one in Brussels and the one in Bretigny at the same time ? or that the one in Bretigny could not take over, but that was the main reason why it was designed in the first place.
Both systems were supposed to be totally independent and act as a back up to one another for that very reason.
I will be curious to read the report and the explanations.
If you have two identical systems running as primary and backup, and someone puts in a malformed message of some type to one that crashes it. The backup picks up and reads same malformed message and will also crash. Engineers reboot the systems they reread the same malformed message...crash... Rinse and repeat. Same can be caused by any common mode fault, which can be expected with identical primary and backup systems.

This is a system design error that I can remember EUROCONTROL being warned about and they 'patted the people raising the problem on the head and told them not to worry' and go away.

Systems often throw away the recent inputs after a crash and then restart hoping that the particular fault will go away when the messages are reinput. It sounds like this deletion of input messages may have been manually implemented given the precise time of all "lost flight plans".

It will be interesting to find out the cause - if it is ever admitted to.
Ian W is offline  
Old 4th Apr 2018, 09:17
  #11 (permalink)  
 
Join Date: Nov 2015
Location: 15km SE of YMML
Posts: 37
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by ATC Watcher
That I have difficulty with . That would mean that both IFPS failed , the one in Brussels and the one in Bretigny at the same time ? or that the one in Bretigny could not take over, but that was the main reason why it was designed in the first place.
Both systems were supposed to be totally independent and act as a back up to one another for that very reason.
I will be curious to read the report and the explanations.
I wonder if the lines to the IFPS nodes failed. Communications these days are cheaper, faster and less redundant than in the past. PRM in Sydney failed when it couldn't contact the FDP in Melbourne.
c_coder is offline  
Old 4th Apr 2018, 09:26
  #12 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by c_coder
I wonder if the lines to the IFPS nodes failed. Communications these days are cheaper, faster and less redundant than in the past. PRM in Sydney failed when it couldn't contact the FDP in Melbourne.
With one node in Brussels and the other South of Paris a common comms failure is unlikely. A more complex failure of the sharing mechanism could cause issues, but it sounds more like a common mode failure of the two IFPS themselves.
Ian W is offline  
Old 4th Apr 2018, 11:21
  #13 (permalink)  
 
Join Date: Nov 2015
Location: 15km SE of YMML
Posts: 37
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Ian W
With one node in Brussels and the other South of Paris a common comms failure is unlikely. A more complex failure of the sharing mechanism could cause issues, but it sounds more like a common mode failure of the two IFPS themselves.
Yes I have seen that sort of common mode failure, but on HMI nodes, which segfaulted. Flow management or flight data software should be more robust.

Though the operative word is should and we will find out eventually I suppose.
c_coder is offline  

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service

Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.