Go Back  PPRuNe Forums > Flight Deck Forums > Rumours & News
Reload this Page >

'System outage' grounds Delta flights

Rumours & News Reporting Points that may affect our jobs or lives as professional pilots. Also, items that may be of interest to professional pilots.

'System outage' grounds Delta flights

Old 8th Aug 2016, 09:17
  #1 (permalink)  
Thread Starter
 
Join Date: Jul 2013
Location: Genève
Posts: 35
'System outage' grounds Delta flights

Delta airlines says all flights suspended "due to system outage nationwide"

'System outage' grounds Delta flights - BBC News

Twitter

Last edited by Dimitri Cherchenko; 8th Aug 2016 at 09:37.
Dimitri Cherchenko is offline  
Old 8th Aug 2016, 12:12
  #2 (permalink)  
 
Join Date: Jul 2001
Location: UK
Posts: 195
Must be a nightmare for all concerned and affected - any word as to the root cause?
Porky Speedpig is offline  
Old 8th Aug 2016, 12:15
  #3 (permalink)  
 
Join Date: Feb 2004
Location: USA
Posts: 2,876
This:

http://finance.yahoo.com/news/delta-...111236208.html
bafanguy is offline  
Old 8th Aug 2016, 12:20
  #4 (permalink)  
 
Join Date: Oct 2013
Location: ALandDownUnder
Posts: 115
8am and the world's busiest airport is very quiet! Not a good week for those traveling though major airports
log0008 is offline  
Old 8th Aug 2016, 12:24
  #5 (permalink)  
 
Join Date: Jul 2001
Location: UK
Posts: 195
Wow, that's one heck of a power outage. No doubt 2-3 back up power systems too so it will be interesting to see why one couldn't kick in.
Porky Speedpig is offline  
Old 8th Aug 2016, 12:58
  #6 (permalink)  
 
Join Date: Apr 2016
Location: localhost
Age: 23
Posts: 220
Surely in the age of the cloud they should have multiple data centres?!
crablab is offline  
Old 8th Aug 2016, 13:12
  #7 (permalink)  
 
Join Date: Apr 2007
Location: London
Age: 62
Posts: 110
Power cut crashes Delta's worldwide flight update systems ? The Register
Alanwsg is offline  
Old 8th Aug 2016, 16:12
  #8 (permalink)  
 
Join Date: Jul 2014
Location: England
Posts: 373
From experience elsewhere (not being able to see behind the curtain at Delta) I'd hazard a guess that their IT guys have been beating on the management for years to get proper hot fall-back for what is nowadays a mission-critical system, but management (supported by bean-counters) have been stalling on the necessary investment.

OTOH, they may have all the fall-back in the world but they've never actually exercised it properly, so when it's called on for real, it doesn't work ...
OldLurker is offline  
Old 8th Aug 2016, 16:21
  #9 (permalink)  
 
Join Date: Aug 2009
Location: Texas
Age: 62
Posts: 5,848
Originally Posted by OldLurker View Post
OTOH, they may have all the fall-back in the world but they've never actually exercised it properly, so when it's called on for real, it doesn't work ...
I wonder. A whole lot of folks ran "business continuity plans" and tests before the dreaded "Y2K" event (which wasn't all that it threatened to be) so perhaps management consider that event the IT guys "crying wolf."

So now they lost a bit of this week's flock/wool ...
Lonewolf_50 is offline  
Old 8th Aug 2016, 17:01
  #10 (permalink)  
 
Join Date: Jul 2001
Location: UK
Posts: 195
A veritable squadron of off schedule Delta birds on the Atlantic now most heading for JFK and ATL and arriving in quick succession. Likely to be a nightmare at CBP.
Porky Speedpig is offline  
Old 8th Aug 2016, 17:17
  #11 (permalink)  
 
Join Date: Jun 2006
Location: Brisbane
Posts: 265
the dreaded "Y2K" event (which wasn't all that it threatened to be)
Y2J wasn't all that it threatened to be because they identified it beforehand and fixed it (at huge cost in some areas). You obviously missed that bit.
Derfred is offline  
Old 8th Aug 2016, 17:24
  #12 (permalink)  
Paxing All Over The World
 
Join Date: May 2001
Location: Hertfordshire, UK.
Age: 65
Posts: 9,645
Correct Derfred. Y2K was fixed in time but it suits everyone to say we cryed wolf. Then it is easier to not give credit where it's due.

Correct OldLurker. I was in IT for 27 years and because it works 99.9% of the time, they think it will be acceptable when it fails. As they say in the airline industry, "If you think preventative maintenance is expensive - try a crash for size" For years I saw IT starved of investment and then, when it did go wrong, they gave us the money we'd been asking for.
PAXboy is online now  
Old 8th Aug 2016, 18:00
  #13 (permalink)  
 
Join Date: Apr 2007
Location: moraira,spain-Norfolk, UK
Age: 80
Posts: 387
Y2K, I made a lot of money out of this. I invested a smallish amount into a company
working on Y2K solutions. Then I forgot about it for a couple of years, investment
had improved in value by about 8 times. On the subject of power supplies the last Data
Centre which I managed had 2*Ships diesels, each capable of carrying the mainframe &
ancilliary equipment load. I wonder in the case of Delta if it was some equipment other
than the computers which failed. I remember in NZ a few years ago the highly robust
point of sale network went down when someone found and cut the only "single point failure" cable.
esa-aardvark is offline  
Old 8th Aug 2016, 18:06
  #14 (permalink)  
c52
 
Join Date: Sep 2007
Location: Surrey
Posts: 2,197
My near-invariable experience in IT was that there would be a total failure when the annual test of the uninterruptible power supply took place.
c52 is offline  
Old 8th Aug 2016, 18:28
  #15 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
I wonder how long it will be for Delta to have a redundant system set up say at MSP? If they had done that the failure could have been transparent to the airline apart from the people directly involved at ATL. It seems that all the airline beancounters would prefer to upset their customers and give dispatch a really hard problem to solve at vast expense to the airline (just think of the EU mandated payments!) rather than have an efficient system that is fault tolerant. Perhaps, if IT had asked for the back up and it been refused the costs of the global ground stop and recovery could be put on the accountancy head count budget? That might concentrate their minds.
Ian W is offline  
Old 8th Aug 2016, 18:35
  #16 (permalink)  
 
Join Date: Jul 2013
Location: Pittsburgh
Posts: 129
What's interesting is Georgia Power is denying that there was an outage at all. None of their customers had a loss and all their equipment was running.
Delta called them to have them look at a master switch of some sort which had failed. Hmm.

What is that called.....single point of, something.....what is it again?

If it is something that silly and bad, heads will roll.
Smott999 is offline  
Old 8th Aug 2016, 18:40
  #17 (permalink)  
 
Join Date: Oct 2002
Location: Silicon Hills
Posts: 234
We had a lightning induced power failure years ago. Also had a bank of batteries and diesel generators to take over that failed to work. Seems the folks that maintained the UPS system monthly failed to disconnect the dummy load used for testing.....
vector4fun is offline  
Old 8th Aug 2016, 18:52
  #18 (permalink)  
 
Join Date: Jul 2013
Location: Pittsburgh
Posts: 129
I used to work at an intl bank, and every 6-9 months we had to do a full DIsaster recovery drill. Took it very seriously. If these guys had their whole global network sitting on one switch....yikes.
Smott999 is offline  
Old 8th Aug 2016, 18:55
  #19 (permalink)  
 
Join Date: Aug 2009
Location: Texas
Age: 62
Posts: 5,848
Originally Posted by Derfred View Post
Y2J wasn't all that it threatened to be because they identified it beforehand and fixed it (at huge cost in some areas). You obviously missed that bit.
No, I didn't miss anything as I was involved in three different BCP's for Y2K. Thanks for playing. I just didn't add all of the bloody detail to that post, so perhaps I overdid the brevity.


My point is that some in current management, who probably weren't in management then, may perceive through MBA eyes that Y2K was "crying wolf" since they didn't know what it took to mitigate it. As you are probably aware, management "up and out" and medium to high turnover is common.
Lonewolf_50 is offline  
Old 8th Aug 2016, 19:51
  #20 (permalink)  
 
Join Date: Sep 2015
Location: UK
Posts: 96
Ars Technica has this:

"According to the flight captain of JFK-SLC this morning, a routine scheduled switch to the backup generator this morning at 2:30am caused a fire that destroyed both the backup and the primary. "

If true: oops.

Data center disaster disrupts Delta Air Lines | Ars Technica
Joe_K is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service - Do Not Sell My Personal Information -

Copyright © 2021 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.