View Full Version : Eurocontrol problems
tubby linton
3rd Apr 2018, 13:37
The ETFMS system has failed and slot allocation by the system is not available. Expect big delays.
ATTN : ALL FMPS AND NOMINATED TOWERS
INFO : OTHER ATS UNITS AND ALL OPERATORS
ETFMS UNAVAILABILITY
DUE TO ETFMS SYSTEM FAILURE,
SLOT ALLOCATION BY THE ETFMS SYSTEM IS NOT AVAILABLE. ALL ATTEMPTS TO REACTIVATE ETFMS PROVED UNSUCCESSFUL.
IN ACCORDANCE WITH LOCAL PROCEDURES FMPs AND NOMINATED ATC TOWERS WILL HAVE TO APPLY THE ATFCM PROCEDURAL CONTINGENCY PLAN.
CONTINGENCY PLAN START TIME (CST) : WITH IMMEDIATE EFFECT
(TIME OF ENTERING THE OPERATIONAL CONTINGENCY PHASE)
-TOWERS WILL CONTINUE TO HONOUR CTOT WHICH FALL WITHIN THE PERIOD UP TO CST. AFTER THIS TIME OPERATION WILL BE BASED ON DEPARTURE INTERVALS ARRANGED BY NOMINATED TOWERS/FMPs. CTOT WHICH FALL IN THE PERIOD AFTER THE CST ARE IGNORED.
-THE EXTENDED SLOT TOLERANCE WINDOW (-5 TO +20 MIN) MENTIONED IN THE ALERT FLASH WILL CONTINUE TO APPLY. WHERE THE APPLICATION OF THE SLOT TOLERANCE WINDOW TAKES FLIGHTS INTO THE OPERATIONAL CONTINGENCY PHASE THE FLIGHTS WILL STILL BE ALLOWED TO DEPART AS
LONG AS THEY REMAIN WITHIN THE EXTENDED SLOT TOLERANCE WINDOW.
-FLIGHTS DEPARTING FROM NON-NOMINATED AIRFIELDS ARE NOT AFFECTED BY THE CONTINGENCY PLAN.
-FLIGHT PLAN AND FLIGHT PLAN UPDATE MSGS (INCLUDING CNL) ARE TO BE SENT TO THE IFPS UNITS THROUGHOUT THE WHOLE PERIOD OF OPERATION OF THE ATFCM CONTINGENCY PLAN.
-FOR ADDITIONAL INFORMATION PLEASE CALL YOUR LOCAL FMP.
NMOC Brussels
Del Prado
3rd Apr 2018, 13:45
More here:
Eurocontrol public portal (http://www.public.nm.eurocontrol.int/PUBPORTAL/gateway/spec/index.html)
Airbubba
3rd Apr 2018, 14:19
Hopefully some good news:
The issue with the ETFMS has been identified and work is progressing to recover the system, which is expected to occur late this evening. Aircraft operators are requested to refile any flight plans for flights not yet operated and that were originally filed before 10.26UTC
https://twitter.com/eurocontrol/status/981170816061661185
Update from OpsGroup:
---
Hi everyone,
The European ATC system is in crisis mode at the moment, due to a failure of the slot computer with multiple knock on effects. Here is our update of the situation:
OPSGROUP Update - 1455Z Tuesday
- Based on direct news from Eurocontrol, a Conference Call completed at 1415Z, and Operator input to Opsgroup
1. The system that allocates ATC slots to flights, and therefore manages the flow of traffic across Europe, failed today at 1026 UTC. It’s called the ETFMS (Enhanced Tactical Flow Management System), but aka “The Slot Computer”
2. There is a Contingency Plan for this situation. Airports are supposed to use this, which gives a quick table of departure intervals allowed according to the destination. You can view the plan here and see what it looks like for the airport you’re using: https://ops.group/files/NM.Contingency.Summer.2018.pdf
3. Some airlines reported that Istanbul, amongst others, were holding all departures. During the call, Istanbul at least were informed there was no need to do this, but others may still be unclear as to how to handle this.
4. All flight plans filed before 1026Z have been lost. You need to re-file your FPL if filed before this time, but in reality, it’s probably a good idea to re-file any FPL you have for today or tomorrow, to make sure it’s OK.
5. Eurocontrol says 10% capacity reduction across the whole of Europe, and expect ’high delays’. Actual delay numbers - usually available on the NOP - will be inaccurate because of all the missing FPL’s.
6. Eurocontrol will only switch back on the Slot computer once they reach a critical mass of filed flight plans - so the Contingency Plan will continue for all of today and probably into tomorrow. Eurocontrol are calling round the 70 main airports to make sure they know what they’re supposed to do.
7. Normally, Eurocontrol will re-address your FPL to ATC Centres outside the IFPZ. This needs to be done manually, ie. find the FIR’s you will cross, get their AFTN address like HECCZQZX, and send them your FPL - if you can get that info.
8. The actual system failure has been fixed as of about 1400Z, but Eurocontrol says the ATC system will not recover until well into tomorrow, earliest.
9. The Eurocontrol hotline for operators to call for more information about the contingency plan is: +32 496 XXX XXX. Go easy on them, they’ll be swamped.
10. Ramps may be congested, there may be arrival delays, so carry extra fuel.
For updates:
Monitor the Eurocontrol NOP
Monitor OpsGroup Slack - the #todays-ops channel
Kind regards,
The Team.
Quick links - OPSGROUP members:
Login to the Dashboard
Launch OpsGroup slack
Check Airport Spy
Upgrade to Team or Flight Department membership
Check the FSB blog
Email the OpsGroup Team
ATC Watcher
3rd Apr 2018, 15:56
All flight plans filed before 1026Z have been lost
That I have difficulty with . That would mean that both IFPS failed , the one in Brussels and the one in Bretigny at the same time ? or that the one in Bretigny could not take over, but that was the main reason why it was designed in the first place.
Both systems were supposed to be totally independent and act as a back up to one another for that very reason.
I will be curious to read the report and the explanations.
underfire
3rd Apr 2018, 17:19
damn.......
they probably just figured out that the information is not mirrored to the fail-over system...
or, the Russians have hacked the aviation system...again.
songbird29
3rd Apr 2018, 20:23
MESSAGE FROM DIRECTOR NETWORK MANAGER
EUROCONTROL very much regrets the disruption that has been caused to passengers and airlines due to today’s outage of the Enhanced Tactical Flow Management System (ETFMS).
The contingency procedures put in place ensured that safety was not compromised at any time.
As of 18.00 UTC, following the resumption of the ETFMS, the EUROCONTROL Network Manager has restarted normal flow management operations. This has been done after extensive internal testing and in coordination with airports, airlines and air traffic control across Europe and beyond.
In over 20 years of operation, the ETFMS has only had one other outage which occurred in 2001. The system currently manages up to 36,000 flights a day.
NMOC Brussels
Piltdown Man
3rd Apr 2018, 22:05
It’s been a brilliant day for me. I hope EuroControl never fix their wretched systems. Over four sectors our longest departure delay was 11 minutes, our latest arrival delay was 12 minutes. The Karma was ATC having to work harder, like we have to do EVERY day in order to minimise the delays we are given.
PM
ps. I bet EuroControl will still charge full whack for their “Service”.
RickNRoll
4th Apr 2018, 02:50
damn.......
they probably just figured out that the information is not mirrored to the fail-over system...
or, the Russians have hacked the aviation system...again.
Or the errors were propagated over to the failover system.
Happened to the Australian Tax Office. They lost everything due to data corruption in the primary SAN being mirrored over to the backup SAN.
That I have difficulty with . That would mean that both IFPS failed , the one in Brussels and the one in Bretigny at the same time ? or that the one in Bretigny could not take over, but that was the main reason why it was designed in the first place.
Both systems were supposed to be totally independent and act as a back up to one another for that very reason.
I will be curious to read the report and the explanations.
If you have two identical systems running as primary and backup, and someone puts in a malformed message of some type to one that crashes it. The backup picks up and reads same malformed message and will also crash. Engineers reboot the systems they reread the same malformed message...crash... Rinse and repeat. Same can be caused by any common mode fault, which can be expected with identical primary and backup systems.
This is a system design error that I can remember EUROCONTROL being warned about and they 'patted the people raising the problem on the head and told them not to worry' and go away.
Systems often throw away the recent inputs after a crash and then restart hoping that the particular fault will go away when the messages are reinput. It sounds like this deletion of input messages may have been manually implemented given the precise time of all "lost flight plans".
It will be interesting to find out the cause - if it is ever admitted to.
c_coder
4th Apr 2018, 09:17
That I have difficulty with . That would mean that both IFPS failed , the one in Brussels and the one in Bretigny at the same time ? or that the one in Bretigny could not take over, but that was the main reason why it was designed in the first place.
Both systems were supposed to be totally independent and act as a back up to one another for that very reason.
I will be curious to read the report and the explanations.
I wonder if the lines to the IFPS nodes failed. Communications these days are cheaper, faster and less redundant than in the past. PRM in Sydney failed when it couldn't contact the FDP in Melbourne.
I wonder if the lines to the IFPS nodes failed. Communications these days are cheaper, faster and less redundant than in the past. PRM in Sydney failed when it couldn't contact the FDP in Melbourne.
With one node in Brussels and the other South of Paris a common comms failure is unlikely. A more complex failure of the sharing mechanism could cause issues, but it sounds more like a common mode failure of the two IFPS themselves.
c_coder
4th Apr 2018, 11:21
With one node in Brussels and the other South of Paris a common comms failure is unlikely. A more complex failure of the sharing mechanism could cause issues, but it sounds more like a common mode failure of the two IFPS themselves.
Yes I have seen that sort of common mode failure, but on HMI nodes, which segfaulted. Flow management or flight data software should be more robust.
Though the operative word is should and we will find out eventually I suppose.