Go Back  PPRuNe Forums > Misc. Forums > Passengers & SLF (Self Loading Freight)
Reload this Page >

BA delays at LHR - Computer issue

Passengers & SLF (Self Loading Freight) If you are regularly a passenger on any airline then why not post your questions here?

BA delays at LHR - Computer issue

Old 27th May 2017, 19:06
  #61 (permalink)  
 
Join Date: Jan 2002
Location: Back of beyond
Posts: 793
Received 4 Likes on 3 Posts
Any decent data centre has an array of batteries that kick in as soon as one of the main power supplies( you'll normally have 3) fails and keep the machines running until the whacking great diesel generator (kept at operating temperature) takes over. It'll have fuel for the next 48 hours.
And you'll have your core systems mirrored.
This is industry standard.
RevMan2 is offline  
Old 27th May 2017, 19:08
  #62 (permalink)  
 
Join Date: Jul 2013
Location: Eastern Anglia
Age: 75
Posts: 131
Likes: 0
Received 0 Likes on 0 Posts
You would think they would have removed the tweet that immediately precedes the two 'we can't do anything today' tweets which reads:
******
Summer isn't just about beaches - check our new High Life Special & plan your own 'oh my god' trip: (my italics)
******
I presume departure halls the world over are filled with BA pax saying those exact words at this very moment?

fenland787 is offline  
Old 27th May 2017, 19:15
  #63 (permalink)  
 
Join Date: Jul 2014
Location: England
Posts: 396
Received 1 Like on 1 Post
Murphy can bite you in the ankle any time. One company I worked with seemed to have done most things right, including two mains feeds coming into the building from two separate grids (three would have been better, but there wasn't a third available). What they didn't realise was that in one place in the street the contractors had run the two cables side by side, in a shallow conduit because of some obstruction underneath. Inevitably some fool with a backhoe began to dig a hole in just that place (he should have been on the other side of the street) and chopped both cables at once ... Thankfully the company did have not only a UPS that worked, but also a backup generator that started on demand!
OldLurker is offline  
Old 27th May 2017, 19:17
  #64 (permalink)  
 
Join Date: Aug 2007
Location: Ireland
Posts: 216
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by RevMan2
Any decent data centre has an array of batteries that kick in as soon as one of the main power supplies( you'll normally have 3) fails and keep the machines running until the whacking great diesel generator (kept at operating temperature) takes over. It'll have fuel for the next 48 hours.
And you'll have your core systems mirrored.
This is industry standard.
It might be the "industrial standard" but util the power actually fails, you don't know if that generator will actually start or be switched in.
A good datacenter will have at least 2 separate ups's for A and B feeds to servers, and failover supplies for equipment that have only 1 actual power supply. But very few will have a second generator setup that will start if the first one don't. Then you have only minutes of ups power to find out why the generator didn't start. And complicated servers/storage and relational databases don't like sudden stops.
vikingivesterled is offline  
Old 27th May 2017, 19:18
  #65 (permalink)  
 
Join Date: Jan 2008
Location: Hotel Sheets, Downtown Plunketville
Age: 76
Posts: 0
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by KelvinD
The CEO has spoken. He says it is all down to a power supply failure.
I was surprised he had a degree in engineering. I had supposed his degree may haven been in kidology!
If he is not out the door by Tuesday, then Willie Walsh should be considering his own position.
POWER SUPPLY FAILURE! That`s got to be the biggest moronic explanation from a bloke with a degree in engineering. Maybe he thinks UPS stands for that parcel lugging company and not for Uninterrupted Power Supply.
Chronus is offline  
Old 27th May 2017, 19:25
  #66 (permalink)  
 
Join Date: Jul 2005
Location: SoCal
Posts: 1,929
Likes: 0
Received 0 Likes on 0 Posts
While the scale is certainly much bigger, this is not an isolated incident. Over the last few weeks I had repeated interactions with BA due to having to change a couple of long-haul flights several times. On more than one of these occasions I was asked to call back later, as their systems were down, sometimes for hours. Not good.
172driver is offline  
Old 27th May 2017, 19:28
  #67 (permalink)  
 
Join Date: Apr 2012
Location: Perth
Posts: 154
Received 5 Likes on 2 Posts
Just had a text message from a work colleague who was travelling back from PHL to EDI through LHR. He's now found some floor space in T5 which is going to be his bed for the night. Obviously hotel rooms are as rare as rocking horse poo tonight at Heathrow. He is not alone with this predicament and you have to wonder how much this is going to cost BA. Hopefully, it will hit them hard.
Speedywheels is offline  
Old 27th May 2017, 19:33
  #68 (permalink)  
 
Join Date: Aug 2001
Location: se england
Posts: 1,570
Likes: 0
Received 44 Likes on 20 Posts
Well the CEO seems to have started out alright with a tech degree and then working with AA and Sabre BUT then he turned to the dark side and got an MBA and worked for 'Management Consulting' companies.

After that he ran a rinky dink start up in Spain and moved to that paragon of quality Vueling. Any one who ran an airline with that paint scheme (vomit on a dirty white background) must have something wrong with them.

In these instances the culture espoused by the CEO starts the rot, if its cut cut cut then line managers will think it is Ok or even their responsibility to pare back on things previously considered essentials , It is hard to resist when your job and bonus depend on following the party line or company culture as opposed to your experience or professional training. I have worked in more than one company where experienced chartered engineers were ridiculed for putting forward professional sound proposals because they were 'too belt and braces' or not in line with current best practice . The latter being something a 'manager' read in a magazine about vaguely similar equipment in a completely different industry
pax britanica is offline  
Old 27th May 2017, 19:36
  #69 (permalink)  
 
Join Date: Jan 2009
Location: Oxford
Posts: 138
Received 0 Likes on 0 Posts
Last time I was dependent upon the (monstrous thing newly installed with much press hoorahing) generator kicking in when a power cut hit us was when sitting in Ghana Airways headquarters in Accra as Deputy Chief Exec of Something or Other ...

It did fire up, but lasted less than a minute because the local lads had drained the humungus diesel tank to set-up a highly profitable local business of their own
Senior Paper Monitor is offline  
Old 27th May 2017, 19:41
  #70 (permalink)  
 
Join Date: Dec 2007
Location: wishaw strathclyde
Posts: 34
Likes: 0
Received 0 Likes on 0 Posts
Not surprising Mr Walsh had little time to appear on national TV explaining the outage.He was far too busy on telephone being expertly advised by his IT consultants to unplug the router and wait 5 minutes. 🛩
aligee is offline  
Old 27th May 2017, 19:48
  #71 (permalink)  
 
Join Date: Oct 2002
Location: London UK
Posts: 7,631
Likes: 0
Received 16 Likes on 14 Posts
Failure because of a "power supply problem" :

Our (small scale in comparison) IT system is 'hosted' in a failure-proof computer hosting centre. There are multiple power supplies, diesel backup generators, batteries, and the whole thing is mirrored in real time to a comparable centre in another continent with automatic failover and restart. Multiple redundancy, something an airline should be familiar with. These organisations nowadays may be Clever Stuff, but are standard, not hugely expensive, and shared by all sorts of major IT operations, banks, credit card companies ... and airlines. I remember the presentation.

I would like to enquire just where the power supply failure actually was so we can find how long the local public electricity supply organisation took to fix it, and which hosting organisation was involved. Journalists, over to you.
WHBM is offline  
Old 27th May 2017, 19:50
  #72 (permalink)  
 
Join Date: Dec 2006
Location: near BHX
Age: 59
Posts: 30
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by vikingivesterled
And complicated servers/storage and relational databases don't like sudden stops.
I did operational IT for a living, rising to CIO, before I went into academia. I got bored with being told that filesystems and databases wouldn't stand sudden stops (ACID properties, right?). I was expected to buy exotic database products from Larry Ellison, tended by smug contractors who had a million and one reasons Postgres just wouldn't do. So I made it a point of acceptance testing from development into production that the systems I was expected to run had to survive a sudden stop. Salesmen from Oracle and NetApp talk about journalling, so let's see it: we're going to flip the power at the time of our choice during your testing, and your product will survive it, and we'll do it again a few times for fun, or you can all go back to your offices and fix it. It's not the 1990s, and fsck isn't a thing any more. They'd whinge and whine that I should be doing an orderly shutdown, but I genuinely meant "I will go into the development lab and flip switches at random".

I flushed out any number of problems with this approach.
xyzzy is offline  
Old 27th May 2017, 19:54
  #73 (permalink)  
 
Join Date: Jul 2007
Location: Hadlow
Age: 60
Posts: 597
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Speedywheels
Just had a text message from a work colleague who was travelling back from PHL to EDI through LHR. He's now found some floor space in T5 which is going to be his bed for the night. Obviously hotel rooms are as rare as rocking horse poo tonight at Heathrow. He is not alone with this predicament and you have to wonder how much this is going to cost BA. Hopefully, it will hit them hard.

Doesn't the law state that pax are entitled to hotel rooms in these cases?
Super VC-10 is offline  
Old 27th May 2017, 20:00
  #74 (permalink)  
 
Join Date: Jun 2009
Location: Canada
Posts: 464
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by xyzzy
I was expected to buy exotic database products from Larry Ellison, tended by smug contractors who had a million and one reasons Postgres just wouldn't do.
We've had a power outage on one of our test servers trash the Postgres database. But that was because the operating system truncated a bunch of files to zero bytes when it rebooted, so I doubt Oracle would have handled it any better.

That's the only system that doesn't have a battery-backed disk cache, which almost certainly would have prevented the problem.
MG23 is offline  
Old 27th May 2017, 20:05
  #75 (permalink)  
 
Join Date: Aug 2007
Location: Ireland
Posts: 216
Likes: 0
Received 0 Likes on 0 Posts
CEO Cruz has Navitaire experience. You would think he'd know the advantage of decentralized and independent systems, but maybe he either could justify the regular but well planned and scheduled downtimes sample Ryanair had for changeovers, upgrades and maintenance. This could be just the incident to spur him on to implement changes, if he gets WW's continued trust.
vikingivesterled is offline  
Old 27th May 2017, 20:12
  #76 (permalink)  
 
Join Date: Dec 2014
Location: The wrong timezone
Posts: 264
Received 11 Likes on 3 Posts
Perhaps the management should read this thread for some expert advice on how to run their IT systems - it seems that the world's supply of experts' opinions are here for the taking.
anson harris is online now  
Old 27th May 2017, 20:16
  #77 (permalink)  
ImageGear
Guest
 
Posts: n/a
Hardware is cheap, Linux is cheap - buy cheap, get cheap. Power failures, totally unacceptable, but billed as an acceptable risk against cost.

If this were Japan, (No single critical system failure in the lifetime of the product), the President would be bowing and sobbing, before being up for the chop, literally.

In the West, disgrace in business is unheard of, can't see Willy doing the honourable thing. Probably he'll take a bigger wedge this time for future protection.

Imagegear
 
Old 27th May 2017, 20:24
  #78 (permalink)  
 
Join Date: Apr 2012
Location: Perth
Posts: 154
Received 5 Likes on 2 Posts
Originally Posted by Super VC-10
Doesn't the law state that pax are entitled to hotel rooms in these cases?
Well he has just texted to say he is being told everybody has to leave T5, he can't stay there overnight. He has nowhere to go so he might have the pleasure of a night in a police cell 😄
Speedywheels is offline  
Old 27th May 2017, 20:30
  #79 (permalink)  
 
Join Date: Jul 2001
Location: Ottawa, Canada
Age: 70
Posts: 213
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Super VC-10
Doesn't the law state that pax are entitled to hotel rooms in these cases?
Not sure what they would do if there are no hotel rooms left, it's not like they could fly them to Birmingham for the night to get a room
ex-EGLL is offline  
Old 27th May 2017, 20:45
  #80 (permalink)  
 
Join Date: Jan 2016
Location: Going left then going right
Posts: 101
Likes: 0
Received 0 Likes on 0 Posts
Planefinder shows BA215 in the air, off Heathrow circa 21:40 BST off to Boston.

Saturday 27th May 2017.
nohold is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service

Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.