Go Back  PPRuNe Forums > Misc. Forums > Passengers & SLF (Self Loading Freight)
Reload this Page >

BA delays at LHR - Computer issue

Passengers & SLF (Self Loading Freight) If you are regularly a passenger on any airline then why not post your questions here?

BA delays at LHR - Computer issue

Old 1st Jun 2017, 22:07
  #441 (permalink)  
 
Join Date: May 2012
Location: Stralya
Posts: 71
Received 0 Likes on 0 Posts
I'm not sure if the two things can be related, but years ago we had a power spike in a fixed GPU that wiped out half the black boxes on one of our aircraft. The GPU was inspected, 'no fault found'.

Is it possible something similar has happened here?
wondrousbitofrough is offline  
Old 2nd Jun 2017, 00:05
  #442 (permalink)  
Paxing All Over The World
 
Join Date: May 2001
Location: Hertfordshire, UK.
Age: 67
Posts: 10,125
Received 58 Likes on 48 Posts
It's not really a question of what started the problem - as problems happen. BUT how it rippled throughout their network and, then, why it took them so long to get it all back. The usual barriers that prevent corrupted data flowing outward appear to have failed and the sequence in which the systems were restored, appears to have been deficient.

Every global corporation must be (silently) thanking BA and warning them of what can happen and givem the chance to check their plans.
PAXboy is offline  
Old 2nd Jun 2017, 05:53
  #443 (permalink)  
 
Join Date: Aug 2003
Location: FR
Posts: 233
Likes: 0
Received 0 Likes on 0 Posts
I read that by outsourcing IT they save 90M a year ... and now possibly this failure will cost about the same. If it only happens every other year, they still are making money, I don't see why they need to fix anything.
pax2908 is offline  
Old 2nd Jun 2017, 05:56
  #444 (permalink)  
 
Join Date: Feb 2005
Location: Botswana
Posts: 887
Likes: 0
Received 0 Likes on 0 Posts
This will cost them a hell of a lot more than £90 Million. The compensation alone is going to cost upwards of £150 Million and that's if you take the lower end of the spectrum. That's before you get into the costs of the loss of confidence in the airline and people taking their business elsewhere in future. This will escalate big time if serious measures aren't put in place to address the fundamental failings of last weekend. But to just focus on the immediate figures with no foresight of the larger problem are words spoken like a true BA manager. Pax2908 are you sure you're not Alex?
RexBanner is offline  
Old 2nd Jun 2017, 07:50
  #445 (permalink)  
aox
 
Join Date: Mar 2015
Location: UK
Posts: 227
Received 0 Likes on 0 Posts
A power supply unit at the centre of last weekend’s British Airways fiasco was in perfect working order but was deliberately shut down in a catastrophic blunder, The Times has learnt.

An investigation into the incident, which disrupted the travel plans of 75,000 passengers, is likely to centre on human error rather than any equipment failure at BA, it emerged.


https://www.thetimes.co.uk/edition/n...nder-tbfhxwsw2


n.b. The Times has a paywall restriction. You can sign up free for two articles a week. I'm up to my limit at the moment, so can't see and quote from the whole article, but the first two paragraphs seem to be sufficient to get the idea (though not later comment, if there is any, about alleged source).
aox is offline  
Old 2nd Jun 2017, 08:04
  #446 (permalink)  
 
Join Date: Dec 2012
Location: Nr Aston Down, Cotswolds
Posts: 158
Likes: 0
Received 0 Likes on 0 Posts
Quote from Guardian (free app)

"An investigation into the power outage that led to chaos for British Airways over the bank holiday weekend is likely to focus on human error rather than any equipment failure.

A contractor doing maintenance work at a BA data centre is said to have inadvertently switched off the power supply, knocking out the airline’s computer systems, according to a report in The Times.

Quoting a BA source, the newspaper said the power supply unit that prompted the IT failure was working perfectly but was accidentally shut down by a worker. An investigation into the power outage is likely to focus on human error rather than any equipment failure, it said.

AdvertisementHide
BA had to cancel all flights from London’s Heathrow and Gatwick airports last Saturday, leaving 75,000 passengers stranded. It blamed a power surge that knocked out its computer system, disrupting flight operations, call centres and its website.

Willie Walsh, the chief executive of British Airways’ parent company IAG, on Thursday praised BA bosses for “doing everything possible” after the IT meltdown.

Walsh’s comments came amid reports that the BA board is demanding an external inquiry into the fiasco, which is expected to land the airline with a compensation bill of more than £100m.

He said the investigation would take time but insisted that BA’s management, led by the Spanish chief executive, Alex Cruz, “did everything they could in the circumstances”.

The airline has been criticised for not promising full compensation to all those affected, but Walsh pledged: “Our focus will be on making sure that any of our customers who experienced disruption are satisfied with how we handled things. Clearly we will do everything we can to make up for the disruption that they suffered.”

However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.

What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
Epsomdog is offline  
Old 2nd Jun 2017, 08:19
  #447 (permalink)  
 
Join Date: Feb 2000
Location: solent-on-sea
Posts: 443
Likes: 0
Received 0 Likes on 0 Posts
Interesting that in relation to this incident, people are quoting newspapers as though they are the all seeing eye of wisdom. In my 25 years in 'aviation', the one thing I can say it's definitively taught me is that if it's in the press, it's at very best partially correct, but more likely completely factually inaccurate.
Not Long Now is offline  
Old 2nd Jun 2017, 08:26
  #448 (permalink)  
Paxing All Over The World
 
Join Date: May 2001
Location: Hertfordshire, UK.
Age: 67
Posts: 10,125
Received 58 Likes on 48 Posts
I agree Not Long Now but it's all we've got and it looks like the press are going to pursue this.

The Guardian: https://www.theguardian.com/business...-reports-claim

Unfortunately 'human error' is no excuse - it is STILL the mgmt who are responsible. When I worked in mission critical environments, we had a process that no one who was not approved could work in the comms centre and all work had to be scheduled and any new person to the environment had to be escorted and monitored. But doing THAT costs money. Who said, "A stitch in time saves nine"?
PAXboy is offline  
Old 2nd Jun 2017, 08:35
  #449 (permalink)  
 
Join Date: Dec 2012
Location: Nr Aston Down, Cotswolds
Posts: 158
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Not Long Now
Interesting that in relation to this incident, people are quoting newspapers as though they are the all seeing eye of wisdom. In my 25 years in 'aviation', the one thing I can say it's definitively taught me is that if it's in the press, it's at very best partially correct, but more likely completely factually inaccurate.
Well in my 47 years in the aviation industry, I've learned to keep an open mind, gather as much information, from ALL sources, then make an appraisal. I don't know if the papers are right or wrong, but I certainly won't dismiss them at this stage, just because they're a newspaper!
Epsomdog is offline  
Old 2nd Jun 2017, 08:39
  #450 (permalink)  
aox
 
Join Date: Mar 2015
Location: UK
Posts: 227
Received 0 Likes on 0 Posts
Originally Posted by Not Long Now
Interesting that in relation to this incident, people are quoting newspapers as though they are the all seeing eye of wisdom. In my 25 years in 'aviation', the one thing I can say it's definitively taught me is that if it's in the press, it's at very best partially correct, but more likely completely factually inaccurate.
I didn't say it like that, and I included "(though not later comment, if there is any, about alleged source)."

It turns out this is alleged to come from someone at BA, presumably very unofficial and unauthorised, given the boss warning to avoid comment, which may or may not be slightly better than journalists who deal with football transfer rumours and copy them from each other, turning the word reportedly into a new cliché.
aox is offline  
Old 2nd Jun 2017, 09:36
  #451 (permalink)  
 
Join Date: Jun 2016
Location: Cheshire
Posts: 141
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Epsomdog
Quote from Guardian (free app)

However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.

What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
The CAA, who are responsible for overseeing how major incidents like this are dealt with, seem to have abdicated their responsibilities.

It's no wonder that passengers are still so confused.
Trav a la is offline  
Old 2nd Jun 2017, 09:37
  #452 (permalink)  
 
Join Date: Dec 2012
Location: Nr Aston Down, Cotswolds
Posts: 158
Likes: 0
Received 0 Likes on 0 Posts
What really worries me, is that as an industry, we appear to be losing the ability to learn from our mistakes!
As an example, my own airline has gradually stepped back from its own policy of, "No Fuelling with pax on board". Fire cover, no fire cover, headset, no headset, boarding, no boarding. To the point now where the matter is totally disregarded. the bean counters have forgotten that a fire during fuelling, in the past, has actually happened! The odds as far as they are concerned are probably against it. I suspect the Bean Counters feel sufficiently removed, that they won't have to suffer the consequences when disaster strikes!
Epsomdog is offline  
Old 2nd Jun 2017, 09:40
  #453 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by pax2908
I read that by outsourcing IT they save 90M a year ... and now possibly this failure will cost about the same. If it only happens every other year, they still are making money, I don't see why they need to fix anything.
I have no doubt that point has already been made by the IAG bean counters.

However, another failure like this one and people will vote with their wallets (as much as the IAG monopoly at LHR will allow them to). But I already meet people who will fly all sorts of multiple layover routes to avoid flying with IAG airlines, BA in particular. Spanish ownership of LHR and BA has not been good for 'the brand'. The best thing for BA would have been a Boris Island or similar new airport in the Thames estuary. The competition would have stopped the silly games from Willie Walsh and IAG and force them to run a good airline for a change.
Ian W is offline  
Old 2nd Jun 2017, 10:16
  #454 (permalink)  
 
Join Date: May 2008
Location: Paris
Age: 60
Posts: 101
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by B Fraser
I was asked to look at ways to reduce fault rates for a famous global telecoms provider. I established that a large proportion of faults were due to kit being unplugged at the customer premises. The cleaner and the hoover were regular culprits. My solution was to provide stickers that read "Do not turn off. Call 0800-123-4567 before disconnecting this power supply".


The idea was not implemented on the grounds of cost.
To be honest, you failed in your remit.


I've designed several datacenters. You simply make it impossible for such unplugging. I'd have been horrified if you made this suggestion to me.
Nialler is offline  
Old 2nd Jun 2017, 10:21
  #455 (permalink)  
 
Join Date: May 2008
Location: Paris
Age: 60
Posts: 101
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Epsomdog
Quote from Guardian (free app)

"An investigation into the power outage that led to chaos for British Airways over the bank holiday weekend is likely to focus on human error rather than any equipment failure.

A contractor doing maintenance work at a BA data centre is said to have inadvertently switched off the power supply, knocking out the airline’s computer systems, according to a report in The Times.

Quoting a BA source, the newspaper said the power supply unit that prompted the IT failure was working perfectly but was accidentally shut down by a worker. An investigation into the power outage is likely to focus on human error rather than any equipment failure, it said.

AdvertisementHide
BA had to cancel all flights from London’s Heathrow and Gatwick airports last Saturday, leaving 75,000 passengers stranded. It blamed a power surge that knocked out its computer system, disrupting flight operations, call centres and its website.

Willie Walsh, the chief executive of British Airways’ parent company IAG, on Thursday praised BA bosses for “doing everything possible” after the IT meltdown.

Walsh’s comments came amid reports that the BA board is demanding an external inquiry into the fiasco, which is expected to land the airline with a compensation bill of more than £100m.

He said the investigation would take time but insisted that BA’s management, led by the Spanish chief executive, Alex Cruz, “did everything they could in the circumstances”.

The airline has been criticised for not promising full compensation to all those affected, but Walsh pledged: “Our focus will be on making sure that any of our customers who experienced disruption are satisfied with how we handled things. Clearly we will do everything we can to make up for the disruption that they suffered.”

However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.

What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
This is far far worse than human error regarding a single event. It's not just one contractor. I hate to repeat that I have decades of experience in this form of computing. If I operated at my current client site and had malicious intent it would take me about an hour to take their systems down. That is with access all areas clearance.


It would be quicker if there were inherent architectural design flaws.
Nialler is offline  
Old 2nd Jun 2017, 10:28
  #456 (permalink)  
 
Join Date: Aug 2001
Location: se england
Posts: 1,569
Likes: 0
Received 44 Likes on 20 Posts
Although the job of management is to make money for the shareholders the reality is these days they put making money for themselves first and this si where much of the problem lies. Any manager who said that its cheaper to leave things as they are because the cost of compensation is not much more than the annual savings should be sacked on the spot for disrespecting the customer base . lets face t a famous airline from one of the EU s most westerly states used to treat pax as dirt eventually had to change their view.

Many large corporations treat their customers with much more respect than this attitude and still make a decent profit . If that really is BAs policy then its EZ for me in the future as i thought at least the airline operations side of BA was the last bit of quality they had left
pax britanica is offline  
Old 2nd Jun 2017, 10:36
  #457 (permalink)  
 
Join Date: Jan 2008
Location: Reading, UK
Posts: 15,784
Received 196 Likes on 90 Posts
Originally Posted by Trav a la
The CAA, who are responsible for overseeing how major incidents like this are dealt with, seem to have abdicated their responsibilities.
"Passengers who wish to make a claim for out-of-pocket expenses or compensation following this disruption should contact British Airways directly in the first instance. If they are not satisfied with the response, they should refer their claim to British Airways' appointed dispute resolution service, CEDR. Dispute resolution services provide independent decisions on passengers' claims that the airline is contractually obliged to abide by."

Seems quite reasonable to me.

CAA statement on the disruption affecting British Airways flights
DaveReidUK is offline  
Old 2nd Jun 2017, 10:36
  #458 (permalink)  
 
Join Date: May 2008
Location: Paris
Age: 60
Posts: 101
Likes: 0
Received 0 Likes on 0 Posts
You know those films where it takes forever to kill the baddie? Those films where he (or she) keeps coming back despite apparently having been killed? That's what a GDSP (geographically dispersed sysplex) is like.


Yeah, I know that any mainframers reading this will nod knowingly. In one sense it's easy to bring down a system. However, when you have multiple hot swaps available, so long as they are fit for purpose the requirement is to take all of them down first - to cut the links from the running system to them, and, if properly configured, to take them down rather than have them polling the active primary and swinging into action if there is no response. So, You get onto all of those, issue a "z eod" command, wait until they are all down before "eod"ing the active system.


45 minutes to an hour of activity, all of which is creating a huge amount of alerts. Think flashing lights and sirens. If I tried to take down such a system I'd expect to be stopped within five minutes. OK, I can take an axe to my primary system in France, but even with the TGV it is a bit of a haul to the secondary site in Luxembourg, and I might need a confederate in Brussels. They'll need an axe and access to the datacenter.


That is a description of resilience.
Nialler is offline  
Old 2nd Jun 2017, 10:41
  #459 (permalink)  
 
Join Date: Aug 2003
Location: FR
Posts: 233
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Ian W
I have no doubt that point has already been made by the IAG bean counters...
That comment (and a previous one I made about this event) was meant to sarcastic ...
pax2908 is offline  
Old 2nd Jun 2017, 10:44
  #460 (permalink)  
 
Join Date: Oct 2002
Location: London UK
Posts: 7,630
Likes: 0
Received 16 Likes on 14 Posts
I can only add that in the last day or two there is some hearsay starting that the initial failure may have been an inside job by some threatened redundancy/demotion/pay reduction IT staff, who knew the weak points in the logic. The fact that it happened just at the start of a holiday weekend with minimal competent staffing at the time being a bit of a pointer.

Of course, Willie and Alex love swinging their weight about saying they are the "cost saving supremos", without it seems any thought to anyone pushing back.
WHBM is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service

Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.