BA delays at LHR - Computer issue
Is it possible something similar has happened here?
Paxing All Over The World
It's not really a question of what started the problem - as problems happen. BUT how it rippled throughout their network and, then, why it took them so long to get it all back. The usual barriers that prevent corrupted data flowing outward appear to have failed and the sequence in which the systems were restored, appears to have been deficient.
Every global corporation must be (silently) thanking BA and warning them of what can happen and givem the chance to check their plans.
Every global corporation must be (silently) thanking BA and warning them of what can happen and givem the chance to check their plans.
Join Date: Aug 2003
Location: FR
Posts: 234
Likes: 0
Received 0 Likes
on
0 Posts
I read that by outsourcing IT they save 90M a year ... and now possibly this failure will cost about the same. If it only happens every other year, they still are making money, I don't see why they need to fix anything.
Join Date: Feb 2005
Location: Botswana
Posts: 890
Likes: 0
Received 0 Likes
on
0 Posts
This will cost them a hell of a lot more than £90 Million. The compensation alone is going to cost upwards of £150 Million and that's if you take the lower end of the spectrum. That's before you get into the costs of the loss of confidence in the airline and people taking their business elsewhere in future. This will escalate big time if serious measures aren't put in place to address the fundamental failings of last weekend. But to just focus on the immediate figures with no foresight of the larger problem are words spoken like a true BA manager. Pax2908 are you sure you're not Alex?
A power supply unit at the centre of last weekend’s British Airways fiasco was in perfect working order but was deliberately shut down in a catastrophic blunder, The Times has learnt.
An investigation into the incident, which disrupted the travel plans of 75,000 passengers, is likely to centre on human error rather than any equipment failure at BA, it emerged.
https://www.thetimes.co.uk/edition/n...nder-tbfhxwsw2
n.b. The Times has a paywall restriction. You can sign up free for two articles a week. I'm up to my limit at the moment, so can't see and quote from the whole article, but the first two paragraphs seem to be sufficient to get the idea (though not later comment, if there is any, about alleged source).
An investigation into the incident, which disrupted the travel plans of 75,000 passengers, is likely to centre on human error rather than any equipment failure at BA, it emerged.
https://www.thetimes.co.uk/edition/n...nder-tbfhxwsw2
n.b. The Times has a paywall restriction. You can sign up free for two articles a week. I'm up to my limit at the moment, so can't see and quote from the whole article, but the first two paragraphs seem to be sufficient to get the idea (though not later comment, if there is any, about alleged source).
Join Date: Dec 2012
Location: Nr Aston Down, Cotswolds
Posts: 160
Likes: 0
Received 0 Likes
on
0 Posts
Quote from Guardian (free app)
"An investigation into the power outage that led to chaos for British Airways over the bank holiday weekend is likely to focus on human error rather than any equipment failure.
A contractor doing maintenance work at a BA data centre is said to have inadvertently switched off the power supply, knocking out the airline’s computer systems, according to a report in The Times.
Quoting a BA source, the newspaper said the power supply unit that prompted the IT failure was working perfectly but was accidentally shut down by a worker. An investigation into the power outage is likely to focus on human error rather than any equipment failure, it said.
AdvertisementHide
BA had to cancel all flights from London’s Heathrow and Gatwick airports last Saturday, leaving 75,000 passengers stranded. It blamed a power surge that knocked out its computer system, disrupting flight operations, call centres and its website.
Willie Walsh, the chief executive of British Airways’ parent company IAG, on Thursday praised BA bosses for “doing everything possible” after the IT meltdown.
Walsh’s comments came amid reports that the BA board is demanding an external inquiry into the fiasco, which is expected to land the airline with a compensation bill of more than £100m.
He said the investigation would take time but insisted that BA’s management, led by the Spanish chief executive, Alex Cruz, “did everything they could in the circumstances”.
The airline has been criticised for not promising full compensation to all those affected, but Walsh pledged: “Our focus will be on making sure that any of our customers who experienced disruption are satisfied with how we handled things. Clearly we will do everything we can to make up for the disruption that they suffered.”
However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.
What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
"An investigation into the power outage that led to chaos for British Airways over the bank holiday weekend is likely to focus on human error rather than any equipment failure.
A contractor doing maintenance work at a BA data centre is said to have inadvertently switched off the power supply, knocking out the airline’s computer systems, according to a report in The Times.
Quoting a BA source, the newspaper said the power supply unit that prompted the IT failure was working perfectly but was accidentally shut down by a worker. An investigation into the power outage is likely to focus on human error rather than any equipment failure, it said.
AdvertisementHide
BA had to cancel all flights from London’s Heathrow and Gatwick airports last Saturday, leaving 75,000 passengers stranded. It blamed a power surge that knocked out its computer system, disrupting flight operations, call centres and its website.
Willie Walsh, the chief executive of British Airways’ parent company IAG, on Thursday praised BA bosses for “doing everything possible” after the IT meltdown.
Walsh’s comments came amid reports that the BA board is demanding an external inquiry into the fiasco, which is expected to land the airline with a compensation bill of more than £100m.
He said the investigation would take time but insisted that BA’s management, led by the Spanish chief executive, Alex Cruz, “did everything they could in the circumstances”.
The airline has been criticised for not promising full compensation to all those affected, but Walsh pledged: “Our focus will be on making sure that any of our customers who experienced disruption are satisfied with how we handled things. Clearly we will do everything we can to make up for the disruption that they suffered.”
However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.
What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
Join Date: Feb 2000
Location: solent-on-sea
Posts: 443
Likes: 0
Received 0 Likes
on
0 Posts
Interesting that in relation to this incident, people are quoting newspapers as though they are the all seeing eye of wisdom. In my 25 years in 'aviation', the one thing I can say it's definitively taught me is that if it's in the press, it's at very best partially correct, but more likely completely factually inaccurate.
Paxing All Over The World
I agree Not Long Now but it's all we've got and it looks like the press are going to pursue this.
The Guardian: https://www.theguardian.com/business...-reports-claim
Unfortunately 'human error' is no excuse - it is STILL the mgmt who are responsible. When I worked in mission critical environments, we had a process that no one who was not approved could work in the comms centre and all work had to be scheduled and any new person to the environment had to be escorted and monitored. But doing THAT costs money. Who said, "A stitch in time saves nine"?
The Guardian: https://www.theguardian.com/business...-reports-claim
Unfortunately 'human error' is no excuse - it is STILL the mgmt who are responsible. When I worked in mission critical environments, we had a process that no one who was not approved could work in the comms centre and all work had to be scheduled and any new person to the environment had to be escorted and monitored. But doing THAT costs money. Who said, "A stitch in time saves nine"?
Join Date: Dec 2012
Location: Nr Aston Down, Cotswolds
Posts: 160
Likes: 0
Received 0 Likes
on
0 Posts
Interesting that in relation to this incident, people are quoting newspapers as though they are the all seeing eye of wisdom. In my 25 years in 'aviation', the one thing I can say it's definitively taught me is that if it's in the press, it's at very best partially correct, but more likely completely factually inaccurate.
Interesting that in relation to this incident, people are quoting newspapers as though they are the all seeing eye of wisdom. In my 25 years in 'aviation', the one thing I can say it's definitively taught me is that if it's in the press, it's at very best partially correct, but more likely completely factually inaccurate.
It turns out this is alleged to come from someone at BA, presumably very unofficial and unauthorised, given the boss warning to avoid comment, which may or may not be slightly better than journalists who deal with football transfer rumours and copy them from each other, turning the word reportedly into a new cliché.
Join Date: Jun 2016
Location: Cheshire
Posts: 141
Likes: 0
Received 0 Likes
on
0 Posts
Quote from Guardian (free app)
However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.
What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.
What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
It's no wonder that passengers are still so confused.
Join Date: Dec 2012
Location: Nr Aston Down, Cotswolds
Posts: 160
Likes: 0
Received 0 Likes
on
0 Posts
What really worries me, is that as an industry, we appear to be losing the ability to learn from our mistakes!
As an example, my own airline has gradually stepped back from its own policy of, "No Fuelling with pax on board". Fire cover, no fire cover, headset, no headset, boarding, no boarding. To the point now where the matter is totally disregarded. the bean counters have forgotten that a fire during fuelling, in the past, has actually happened! The odds as far as they are concerned are probably against it. I suspect the Bean Counters feel sufficiently removed, that they won't have to suffer the consequences when disaster strikes!
As an example, my own airline has gradually stepped back from its own policy of, "No Fuelling with pax on board". Fire cover, no fire cover, headset, no headset, boarding, no boarding. To the point now where the matter is totally disregarded. the bean counters have forgotten that a fire during fuelling, in the past, has actually happened! The odds as far as they are concerned are probably against it. I suspect the Bean Counters feel sufficiently removed, that they won't have to suffer the consequences when disaster strikes!
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes
on
0 Posts
However, another failure like this one and people will vote with their wallets (as much as the IAG monopoly at LHR will allow them to). But I already meet people who will fly all sorts of multiple layover routes to avoid flying with IAG airlines, BA in particular. Spanish ownership of LHR and BA has not been good for 'the brand'. The best thing for BA would have been a Boris Island or similar new airport in the Thames estuary. The competition would have stopped the silly games from Willie Walsh and IAG and force them to run a good airline for a change.
Join Date: May 2008
Location: Paris
Age: 60
Posts: 101
Likes: 0
Received 0 Likes
on
0 Posts
I was asked to look at ways to reduce fault rates for a famous global telecoms provider. I established that a large proportion of faults were due to kit being unplugged at the customer premises. The cleaner and the hoover were regular culprits. My solution was to provide stickers that read "Do not turn off. Call 0800-123-4567 before disconnecting this power supply".
The idea was not implemented on the grounds of cost.
The idea was not implemented on the grounds of cost.
I've designed several datacenters. You simply make it impossible for such unplugging. I'd have been horrified if you made this suggestion to me.
Join Date: May 2008
Location: Paris
Age: 60
Posts: 101
Likes: 0
Received 0 Likes
on
0 Posts
Quote from Guardian (free app)
"An investigation into the power outage that led to chaos for British Airways over the bank holiday weekend is likely to focus on human error rather than any equipment failure.
A contractor doing maintenance work at a BA data centre is said to have inadvertently switched off the power supply, knocking out the airline’s computer systems, according to a report in The Times.
Quoting a BA source, the newspaper said the power supply unit that prompted the IT failure was working perfectly but was accidentally shut down by a worker. An investigation into the power outage is likely to focus on human error rather than any equipment failure, it said.
AdvertisementHide
BA had to cancel all flights from London’s Heathrow and Gatwick airports last Saturday, leaving 75,000 passengers stranded. It blamed a power surge that knocked out its computer system, disrupting flight operations, call centres and its website.
Willie Walsh, the chief executive of British Airways’ parent company IAG, on Thursday praised BA bosses for “doing everything possible” after the IT meltdown.
Walsh’s comments came amid reports that the BA board is demanding an external inquiry into the fiasco, which is expected to land the airline with a compensation bill of more than £100m.
He said the investigation would take time but insisted that BA’s management, led by the Spanish chief executive, Alex Cruz, “did everything they could in the circumstances”.
The airline has been criticised for not promising full compensation to all those affected, but Walsh pledged: “Our focus will be on making sure that any of our customers who experienced disruption are satisfied with how we handled things. Clearly we will do everything we can to make up for the disruption that they suffered.”
However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.
What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
"An investigation into the power outage that led to chaos for British Airways over the bank holiday weekend is likely to focus on human error rather than any equipment failure.
A contractor doing maintenance work at a BA data centre is said to have inadvertently switched off the power supply, knocking out the airline’s computer systems, according to a report in The Times.
Quoting a BA source, the newspaper said the power supply unit that prompted the IT failure was working perfectly but was accidentally shut down by a worker. An investigation into the power outage is likely to focus on human error rather than any equipment failure, it said.
AdvertisementHide
BA had to cancel all flights from London’s Heathrow and Gatwick airports last Saturday, leaving 75,000 passengers stranded. It blamed a power surge that knocked out its computer system, disrupting flight operations, call centres and its website.
Willie Walsh, the chief executive of British Airways’ parent company IAG, on Thursday praised BA bosses for “doing everything possible” after the IT meltdown.
Walsh’s comments came amid reports that the BA board is demanding an external inquiry into the fiasco, which is expected to land the airline with a compensation bill of more than £100m.
He said the investigation would take time but insisted that BA’s management, led by the Spanish chief executive, Alex Cruz, “did everything they could in the circumstances”.
The airline has been criticised for not promising full compensation to all those affected, but Walsh pledged: “Our focus will be on making sure that any of our customers who experienced disruption are satisfied with how we handled things. Clearly we will do everything we can to make up for the disruption that they suffered.”
However, customers have been further angered by BA’s online claim forms, which instructs them to first seek compensation via their travel insurance – leaving many liable to pay an excess. The Association of British Insurers has complained to the airline that it is giving misleading information and that BA should be the first point of call.
What level of compensation the airline intends to provide will be of intense interest to customers still waiting for their bags to be returned – even though they were unable to fly."
It would be quicker if there were inherent architectural design flaws.
Although the job of management is to make money for the shareholders the reality is these days they put making money for themselves first and this si where much of the problem lies. Any manager who said that its cheaper to leave things as they are because the cost of compensation is not much more than the annual savings should be sacked on the spot for disrespecting the customer base . lets face t a famous airline from one of the EU s most westerly states used to treat pax as dirt eventually had to change their view.
Many large corporations treat their customers with much more respect than this attitude and still make a decent profit . If that really is BAs policy then its EZ for me in the future as i thought at least the airline operations side of BA was the last bit of quality they had left
Many large corporations treat their customers with much more respect than this attitude and still make a decent profit . If that really is BAs policy then its EZ for me in the future as i thought at least the airline operations side of BA was the last bit of quality they had left
Seems quite reasonable to me.
CAA statement on the disruption affecting British Airways flights
Join Date: May 2008
Location: Paris
Age: 60
Posts: 101
Likes: 0
Received 0 Likes
on
0 Posts
You know those films where it takes forever to kill the baddie? Those films where he (or she) keeps coming back despite apparently having been killed? That's what a GDSP (geographically dispersed sysplex) is like.
Yeah, I know that any mainframers reading this will nod knowingly. In one sense it's easy to bring down a system. However, when you have multiple hot swaps available, so long as they are fit for purpose the requirement is to take all of them down first - to cut the links from the running system to them, and, if properly configured, to take them down rather than have them polling the active primary and swinging into action if there is no response. So, You get onto all of those, issue a "z eod" command, wait until they are all down before "eod"ing the active system.
45 minutes to an hour of activity, all of which is creating a huge amount of alerts. Think flashing lights and sirens. If I tried to take down such a system I'd expect to be stopped within five minutes. OK, I can take an axe to my primary system in France, but even with the TGV it is a bit of a haul to the secondary site in Luxembourg, and I might need a confederate in Brussels. They'll need an axe and access to the datacenter.
That is a description of resilience.
Yeah, I know that any mainframers reading this will nod knowingly. In one sense it's easy to bring down a system. However, when you have multiple hot swaps available, so long as they are fit for purpose the requirement is to take all of them down first - to cut the links from the running system to them, and, if properly configured, to take them down rather than have them polling the active primary and swinging into action if there is no response. So, You get onto all of those, issue a "z eod" command, wait until they are all down before "eod"ing the active system.
45 minutes to an hour of activity, all of which is creating a huge amount of alerts. Think flashing lights and sirens. If I tried to take down such a system I'd expect to be stopped within five minutes. OK, I can take an axe to my primary system in France, but even with the TGV it is a bit of a haul to the secondary site in Luxembourg, and I might need a confederate in Brussels. They'll need an axe and access to the datacenter.
That is a description of resilience.
Join Date: Aug 2003
Location: FR
Posts: 234
Likes: 0
Received 0 Likes
on
0 Posts
I can only add that in the last day or two there is some hearsay starting that the initial failure may have been an inside job by some threatened redundancy/demotion/pay reduction IT staff, who knew the weak points in the logic. The fact that it happened just at the start of a holiday weekend with minimal competent staffing at the time being a bit of a pointer.
Of course, Willie and Alex love swinging their weight about saying they are the "cost saving supremos", without it seems any thought to anyone pushing back.
Of course, Willie and Alex love swinging their weight about saying they are the "cost saving supremos", without it seems any thought to anyone pushing back.