Go Back  PPRuNe Forums > Flight Deck Forums > Rumours & News
Reload this Page >

'System outage' grounds Delta flights

Rumours & News Reporting Points that may affect our jobs or lives as professional pilots. Also, items that may be of interest to professional pilots.

'System outage' grounds Delta flights

Old 8th Aug 2016, 21:19
  #21 (permalink)  
 
Join Date: Jul 2013
Location: Pittsburgh
Posts: 129
Likes: 0
Received 0 Likes on 0 Posts
So no separate data center? Or backup system on a different power circuit?
Smott999 is offline  
Old 8th Aug 2016, 21:29
  #22 (permalink)  
 
Join Date: Jul 2001
Location: Ottawa, Canada
Age: 69
Posts: 202
Likes: 0
Received 0 Likes on 0 Posts
We had a lightning induced power failure years ago. Also had a bank of batteries and diesel generators to take over that failed to work. Seems the folks that maintained the UPS system monthly failed to disconnect the dummy load used for testing.....
Or my personal favorite from ATC. We religiously switched to the standby generator on the first Sunday of the month to make sure all was good, it always was. One day we lost power, generator fired up and all was good...... for a couple of minutes, then it went very dark and very quiet.

Seems the monthly startup / shutdown checklists made no mention of fuel quantity !!!
ex-EGLL is offline  
Old 8th Aug 2016, 21:33
  #23 (permalink)  
 
Join Date: Mar 2004
Location: Baltimore, MD
Posts: 266
Likes: 0
Received 0 Likes on 0 Posts
When everything is fully redundant, Murphy refocuses his effort on the part that controls both.
FakePilot is offline  
Old 8th Aug 2016, 21:37
  #24 (permalink)  
Resident insomniac
 
Join Date: Aug 2005
Location: N54 58 34 W02 01 21
Age: 78
Posts: 1,871
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by ex-EGLL View Post
Or my personal favorite from ATC. We religiously switched to the standby generator on the first Sunday of the month to make sure all was good, it always was. One day we lost power, generator fired up and all was good...... for a couple of minutes, then it went very dark and very quiet.

Seems the monthly startup / shutdown checklists made no mention of fuel quantity !!!
I think that was what 'sunk' New Orleans during Katrina - the auxiliary generators that were backup for the pumps 'ran out of fuel'.
G-CPTN is offline  
Old 8th Aug 2016, 21:59
  #25 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Smott999 View Post
So no separate data center? Or backup system on a different power circuit?
This is the fact that makes no sense. If you are running a global system and it must be up and running 24/7 then you must not have all your systems in one place. Not only should there be a separate backup system it must be a geographically separate backup system ideally in another State - Delta could have theirs at MSP. Both the ATL and MSP systems should be running in parallel, with their own local backup for fault tolerance. Both should be able to support all operations 24/7, but when both are up they could share the load and keep each other in synch. Banks do this all the time. If ATL had gone down under this scheme only those people in the ATL center would have noticed. All users would have the same service from the one site.

As said before this sounds like a beancounter enforced lack of fault tolerance.
Ian W is offline  
Old 8th Aug 2016, 22:12
  #26 (permalink)  
 
Join Date: Sep 2007
Location: London
Posts: 561
Likes: 0
Received 0 Likes on 0 Posts
Here are some questions for IT experts - and I am sure there are plenty of issues I haven't even thought about.

If DL's computers are inop could other airlines help out, for example KLM produce flight plans for DL flights departing AMS or would there be problems owing to different regulatory regimes, etc? I know that there are specific rules relating to despatchers who have to be licenced in the US.

Where you have codeshare partners - lets say you are a DL ticketed passenger travelling on a KL flight - would your reservation appear in both systems which would effectively provide a backup? I have to say that I have visions of pax arriving at airports having to prove that they have a reservation - useful to have printed off a confirmation.

Presumably VS still has its own computer system as its ops appear to be unaffected.
Peter47 is offline  
Old 8th Aug 2016, 22:15
  #27 (permalink)  
 
Join Date: Apr 2016
Location: localhost
Age: 23
Posts: 220
Likes: 0
Received 0 Likes on 0 Posts
#1 Nah, all the Delta databases will have been in that datacenter (from the sounds of it) - you'd need all that information to produce anything useful and it would be a right pain to modify the KLM systems to handle the data etc. You might as well do it by hand.

#2 I believe all passenger reservations are stored on a central system to aid access of the TSA etc. Like APIS. So I guess you might be able to get data off of that?
crablab is offline  
Old 8th Aug 2016, 22:32
  #28 (permalink)  
 
Join Date: Jul 2013
Location: Pittsburgh
Posts: 129
Likes: 0
Received 0 Likes on 0 Posts
Indeed Ian, makes no sense if they were without a physically remote hot backup data center with fully redundant data, available to switch on promptly should the main site be lost.
I wonder if their are or will be regs about that kind of thing.

Speaking of regs, what about EC 261, that automatic-compensation for delayed/cancelled flights in EU? I've used it myself a few hrs back.

Lot of folks stranded in LHR or Amsterdam might want to make use of it! I wonder how many Yanks know about it though...
Smott999 is offline  
Old 8th Aug 2016, 22:41
  #29 (permalink)  
 
Join Date: Aug 2007
Location: United Kingdom
Posts: 375
Likes: 0
Received 0 Likes on 0 Posts
I wonder how many Yanks know about it though...
Probably not many, but you can be sure their attorneys will
Logohu is offline  
Old 8th Aug 2016, 22:51
  #30 (permalink)  
 
Join Date: Aug 2005
Location: Albuquerque USA
Posts: 172
Likes: 0
Received 0 Likes on 0 Posts
Need to use your backups, not just test them

One lesson I think I learned in another industry: if you want your backups and protections to work when they are needed, you have to actually integrate usage of them into your standard operations. No amount of "really really careful" testing is an adequate substitute.

True story: the factory site at which I worked which at times was probably on the worldwide top ten list for dollar value added across all industries, was so concerned about the single point failure of losing utility power (yes, we had lots of stuff on UPS, but some big stuff of interest was not) that they paid to have several miles of high-voltage connection made to a second point within the utility network. There was a nifty switch on our premises which at need would transfer our load from the one string of power towers to the other.

Came the day we needed the backup connection to work--not because of a failure of the utility, but because a forklift operator on our own premises accidentally damaged a very late connection line by swinging a load up into it.

The post-mortem established that the nifty transfer switch had a battery which needed to be alive for the transfer to happen. And there was no maintenance plan for looking after the battery, which had probably been dead for some time by the day of our need. That one cost a very, very, large amount of money.

Yes, a suitable test would have caught that one, but I'll still hold out for the higher standard of usage. That way people take it seriously, and people notice and fix the troubles.
archae86 is offline  
Old 9th Aug 2016, 00:39
  #31 (permalink)  
 
Join Date: Mar 2008
Location: In the Air
Age: 76
Posts: 15
Likes: 0
Received 0 Likes on 0 Posts
Reservation System Computer Back-up

I understand BA have a fully duplicated back-up in a secret location where it can't be sabotaged. Maybe DL ought to visit BA and learn something?
lincman is offline  
Old 9th Aug 2016, 00:45
  #32 (permalink)  
 
Join Date: May 2011
Location: NEW YORK
Posts: 1,351
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by lincman View Post
I understand BA have a fully duplicated back-up in a secret location where it can't be sabotaged. Maybe DL ought to visit BA and learn something?
Hope the transfer switch battery has been checked regularly.
etudiant is offline  
Old 9th Aug 2016, 02:03
  #33 (permalink)  
 
Join Date: Mar 2015
Location: Washington state
Posts: 209
Likes: 0
Received 0 Likes on 0 Posts
If it makes anyone feel better, years ago I worked for Microsoft in Redmond, Wa. Cost for our data centers was no object and we had the best backup power systems that money could buy at the time. Comes along a big storm in Redmond that took out the power and you guesed it, the lights go out, my screen goes dark, and the fire doors slammed shut..

I think lightning hit the backup generators. The good news is that my house was near "campus" and I got power restored days before anybody else.
Water pilot is offline  
Old 9th Aug 2016, 02:51
  #34 (permalink)  
 
Join Date: Aug 2013
Location: PA
Age: 57
Posts: 30
Likes: 0
Received 0 Likes on 0 Posts
From my experience, including working in Bldg #2 at Redmond, these 'experts' in backup and redundancy are akin to the 'experts' in aviation, always 'formerly' employed in the business.
Everything has relied on boilerplate checklists, with single point failures at virtually every point, so on paper it looks fine, but in operation, it falls apart.
In Seattle, the Police Department had backup generators for the systems. The weekly tests of the systems went fine. When an earthquake happened, the backup generator systems went down 2 hours after startup. The weekly tests had been using up the fuel storage, and there was never a contract in place to keep the tanks topped off.
underfire is offline  
Old 9th Aug 2016, 03:59
  #35 (permalink)  
 
Join Date: May 2002
Location: MA
Posts: 300
Likes: 0
Received 0 Likes on 0 Posts
Most "yanks" don't consult attorneys regarding canceled flights. DL has offered a $200 travel voucher to anyone who was canceled or delayed greater than 3 hours.
RobertS975 is offline  
Old 9th Aug 2016, 04:06
  #36 (permalink)  
 
Join Date: Aug 2013
Location: PA
Age: 57
Posts: 30
Likes: 0
Received 0 Likes on 0 Posts
go for the 15K FF miles!
underfire is offline  
Old 9th Aug 2016, 07:10
  #37 (permalink)  
 
Join Date: May 2009
Location: Confoederatio Helvetica
Age: 67
Posts: 2,847
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by RobertS975 View Post
Most "yanks" don't consult attorneys regarding canceled flights. DL has offered a $200 travel voucher to anyone who was canceled or delayed greater than 3 hours.
Yet 261 provides for €400/600 in CASH not in vouchers. It applies to all DL flights departing from an EU airport.

As passengers can't waive their rights they would be entitled to this in addition to their voucher.

Last edited by ExXB; 9th Aug 2016 at 07:35. Reason: Clarification
ExXB is offline  
Old 9th Aug 2016, 09:57
  #38 (permalink)  
 
Join Date: Mar 2002
Location: EDI, LHR, NQY
Posts: 395
Likes: 0
Received 0 Likes on 0 Posts
Although passengers can't waive their rights, airlines do not have to offer compensation where a passenger has agreed an alternative. Accepting the voucher could be argued as agreement to an alternative offer.

But yes, definitely worth lodging an EC261 as 600 Euro is way more use than an MCO/voucher that probably comes with a string of restrictions.
ajamieson is offline  
Old 9th Aug 2016, 12:15
  #39 (permalink)  
 
Join Date: Jul 2013
Location: Pittsburgh
Posts: 129
Likes: 0
Received 0 Likes on 0 Posts
I once had to go EC261 on United as they stranded me in London.
It literally took 18 months and they denied everything and said their cancellation was "force majeur " and I was entitled to nothing. Until the courts ruled and it was time to pay up, then they tried to bribe me to drop the case. I told them they were in violation of law for contacting me directly instead of my attny and hung up.
There are actually firms in EU that do nothing but prosecute EC261. I think they took about 15% and I had to do nothing except email them my boarding pass and other info. Not bad, but United was just appalling about it.
Smott999 is offline  
Old 9th Aug 2016, 12:18
  #40 (permalink)  
 
Join Date: Sep 2009
Location: Haifa
Age: 67
Posts: 18
Likes: 0
Received 0 Likes on 0 Posts
Any idea why are DL flights still being cancelled today (Tuesday)? Positioning? Some loss of data?
sabbasolo is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service - Do Not Sell My Personal Information

Copyright © 2022 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.