BA delays at LHR - Computer issue
Joined: Jan 2002
Posts: 796
Likes: 4
From: Back of beyond
Any decent data centre has an array of batteries that kick in as soon as one of the main power supplies( you'll normally have 3) fails and keep the machines running until the whacking great diesel generator (kept at operating temperature) takes over. It'll have fuel for the next 48 hours.
And you'll have your core systems mirrored.
This is industry standard.
And you'll have your core systems mirrored.
This is industry standard.
Joined: Jul 2013
Posts: 131
Likes: 0
From: Eastern Anglia
You would think they would have removed the tweet that immediately precedes the two 'we can't do anything today' tweets which reads:
******
Summer isn't just about beaches - check our new High Life Special & plan your own 'oh my god' trip: (my italics)
******
I presume departure halls the world over are filled with BA pax saying those exact words at this very moment?
******
Summer isn't just about beaches - check our new High Life Special & plan your own 'oh my god' trip: (my italics)
******
I presume departure halls the world over are filled with BA pax saying those exact words at this very moment?
Joined: Jul 2014
Posts: 548
Likes: 62
From: England
Murphy can bite you in the ankle any time. One company I worked with seemed to have done most things right, including two mains feeds coming into the building from two separate grids (three would have been better, but there wasn't a third available). What they didn't realise was that in one place in the street the contractors had run the two cables side by side, in a shallow conduit because of some obstruction underneath. Inevitably some fool with a backhoe began to dig a hole in just that place (he should have been on the other side of the street) and chopped both cables at once ... Thankfully the company did have not only a UPS that worked, but also a backup generator that started on demand!
Joined: Aug 2007
Posts: 216
Likes: 0
From: Ireland
Any decent data centre has an array of batteries that kick in as soon as one of the main power supplies( you'll normally have 3) fails and keep the machines running until the whacking great diesel generator (kept at operating temperature) takes over. It'll have fuel for the next 48 hours.
And you'll have your core systems mirrored.
This is industry standard.
And you'll have your core systems mirrored.
This is industry standard.
A good datacenter will have at least 2 separate ups's for A and B feeds to servers, and failover supplies for equipment that have only 1 actual power supply. But very few will have a second generator setup that will start if the first one don't. Then you have only minutes of ups power to find out why the generator didn't start. And complicated servers/storage and relational databases don't like sudden stops.
Joined: Jan 2008
Posts: 0
Likes: 0
From: Hotel Sheets, Downtown Plunketville
POWER SUPPLY FAILURE! That`s got to be the biggest moronic explanation from a bloke with a degree in engineering. Maybe he thinks UPS stands for that parcel lugging company and not for Uninterrupted Power Supply.
Joined: Jul 2005
Posts: 1,929
Likes: 0
From: SoCal
While the scale is certainly much bigger, this is not an isolated incident. Over the last few weeks I had repeated interactions with BA due to having to change a couple of long-haul flights several times. On more than one of these occasions I was asked to call back later, as their systems were down, sometimes for hours. Not good.
Joined: Apr 2012
Posts: 174
Likes: 14
From: Perth
Just had a text message from a work colleague who was travelling back from PHL to EDI through LHR. He's now found some floor space in T5 which is going to be his bed for the night. Obviously hotel rooms are as rare as rocking horse poo tonight at Heathrow. He is not alone with this predicament and you have to wonder how much this is going to cost BA. Hopefully, it will hit them hard.
Joined: Aug 2001
Aviation Qualifications: Spotter
Posts: 1,829
Likes: 165
From: se england
Well the CEO seems to have started out alright with a tech degree and then working with AA and Sabre BUT then he turned to the dark side and got an MBA and worked for 'Management Consulting' companies.
After that he ran a rinky dink start up in Spain and moved to that paragon of quality Vueling. Any one who ran an airline with that paint scheme (vomit on a dirty white background) must have something wrong with them.
In these instances the culture espoused by the CEO starts the rot, if its cut cut cut then line managers will think it is Ok or even their responsibility to pare back on things previously considered essentials , It is hard to resist when your job and bonus depend on following the party line or company culture as opposed to your experience or professional training. I have worked in more than one company where experienced chartered engineers were ridiculed for putting forward professional sound proposals because they were 'too belt and braces' or not in line with current best practice . The latter being something a 'manager' read in a magazine about vaguely similar equipment in a completely different industry
After that he ran a rinky dink start up in Spain and moved to that paragon of quality Vueling. Any one who ran an airline with that paint scheme (vomit on a dirty white background) must have something wrong with them.
In these instances the culture espoused by the CEO starts the rot, if its cut cut cut then line managers will think it is Ok or even their responsibility to pare back on things previously considered essentials , It is hard to resist when your job and bonus depend on following the party line or company culture as opposed to your experience or professional training. I have worked in more than one company where experienced chartered engineers were ridiculed for putting forward professional sound proposals because they were 'too belt and braces' or not in line with current best practice . The latter being something a 'manager' read in a magazine about vaguely similar equipment in a completely different industry
Joined: Jan 2009
Posts: 147
Likes: 14
From: Oxford
Last time I was dependent upon the (monstrous thing newly installed with much press hoorahing) generator kicking in when a power cut hit us was when sitting in Ghana Airways headquarters in Accra as Deputy Chief Exec of Something or Other ...
It did fire up, but lasted less than a minute because the local lads had drained the humungus diesel tank to set-up a highly profitable local business of their own
It did fire up, but lasted less than a minute because the local lads had drained the humungus diesel tank to set-up a highly profitable local business of their own
Joined: Dec 2007
Posts: 34
Likes: 0
From: wishaw strathclyde
Not surprising Mr Walsh had little time to appear on national TV explaining the outage.He was far too busy on telephone being expertly advised by his IT consultants to unplug the router and wait 5 minutes. 🛩
Joined: Oct 2002
Aviation Qualifications: PPL
Posts: 8,201
Likes: 347
From: London UK
Failure because of a "power supply problem" :
Our (small scale in comparison) IT system is 'hosted' in a failure-proof computer hosting centre. There are multiple power supplies, diesel backup generators, batteries, and the whole thing is mirrored in real time to a comparable centre in another continent with automatic failover and restart. Multiple redundancy, something an airline should be familiar with. These organisations nowadays may be Clever Stuff, but are standard, not hugely expensive, and shared by all sorts of major IT operations, banks, credit card companies ... and airlines. I remember the presentation.
I would like to enquire just where the power supply failure actually was so we can find how long the local public electricity supply organisation took to fix it, and which hosting organisation was involved. Journalists, over to you.
Our (small scale in comparison) IT system is 'hosted' in a failure-proof computer hosting centre. There are multiple power supplies, diesel backup generators, batteries, and the whole thing is mirrored in real time to a comparable centre in another continent with automatic failover and restart. Multiple redundancy, something an airline should be familiar with. These organisations nowadays may be Clever Stuff, but are standard, not hugely expensive, and shared by all sorts of major IT operations, banks, credit card companies ... and airlines. I remember the presentation.
I would like to enquire just where the power supply failure actually was so we can find how long the local public electricity supply organisation took to fix it, and which hosting organisation was involved. Journalists, over to you.
Joined: Dec 2006
Posts: 36
Likes: 29
From: near BHX
I flushed out any number of problems with this approach.
Joined: Jul 2007
Posts: 597
Likes: 0
From: Hadlow
Just had a text message from a work colleague who was travelling back from PHL to EDI through LHR. He's now found some floor space in T5 which is going to be his bed for the night. Obviously hotel rooms are as rare as rocking horse poo tonight at Heathrow. He is not alone with this predicament and you have to wonder how much this is going to cost BA. Hopefully, it will hit them hard.
Doesn't the law state that pax are entitled to hotel rooms in these cases?
Joined: Jun 2009
Posts: 463
Likes: 0
From: Canada
That's the only system that doesn't have a battery-backed disk cache, which almost certainly would have prevented the problem.
Joined: Aug 2007
Posts: 216
Likes: 0
From: Ireland
CEO Cruz has Navitaire experience. You would think he'd know the advantage of decentralized and independent systems, but maybe he either could justify the regular but well planned and scheduled downtimes sample Ryanair had for changeovers, upgrades and maintenance. This could be just the incident to spur him on to implement changes, if he gets WW's continued trust.
Guest
Posts: n/a
Hardware is cheap, Linux is cheap - buy cheap, get cheap. Power failures, totally unacceptable, but billed as an acceptable risk against cost.
If this were Japan, (No single critical system failure in the lifetime of the product), the President would be bowing and sobbing, before being up for the chop, literally.
In the West, disgrace in business is unheard of, can't see Willy doing the honourable thing. Probably he'll take a bigger wedge this time for future protection.
Imagegear
If this were Japan, (No single critical system failure in the lifetime of the product), the President would be bowing and sobbing, before being up for the chop, literally.
In the West, disgrace in business is unheard of, can't see Willy doing the honourable thing. Probably he'll take a bigger wedge this time for future protection.
Imagegear
Joined: Apr 2012
Posts: 174
Likes: 14
From: Perth
Joined: Jul 2001
Posts: 227
Likes: 9
From: Ottawa, Canada