BA delays at LHR - Computer issue

[southern duel]

Originally Posted by wiggy

I know why it's fashionable and understandable to give BA a kicking but I seem to recall the volcanic ash brought most European airlines to a grinding halt.....

I'd have to say some of the other incidents mentioned at Heathrow (e.g. Buncefield) certainly also impacted operators other than BA, but with it being BAs main base they suffered the greatest disruption....

Yes it did affect other carriers but my point is that as the "home carrier" they should have had some sort of contingency in place. Instead of running around like headless chickens and not fulfilling their duties to the public.

[crippen]

did I miss it??

No mention of Willie Walsh. Left Aer lingus in a Huff. Lots of strikes and lock-outs at BA whilst there, and now this.

[wiggy]

Originally Posted by southern duel

Yes it did affect other carriers but my point is that as the "home carrier" they should have had some sort of contingency in place. Instead of running around like headless chickens and not fulfilling their duties to the public.

I'd agree it would be nice to have a sure fire contingency plan to keep the whole operation running on schedule no matter what the problem...but how do you keep the whole operation running without a glitch if say there is e.g. another Buncefield?

You can never guarantee to keep the operation going, it's how you handle things when the music stops that then counts...and where I would agree with you was that on Saturday one had the impression there was of a lack of visible leadership and a breakdown in comms...

[Nialler]

Originally Posted by Cazalet33

The power supply excuse doesn't quite sound credible for such a wide ranging series of failures.

They only way I can see it happening is as follows :

You're running x number of data centers each geographically remote of each and each mirroring the active Datacenter and each designed to perform a seamless horse swap.

On the active data centre DC1 you have kicked a batch job consisting of a volume of updates within the database subsystem. The power cut hits with uncommitted updates still in full flow.

DC2 takes over, continues, but because DC1 didn't have time to communicate a synch point, and as a result of network latency, DC2 resumes at the incorrect moment of processing. Now may have a corrupt database. Worse, subsequent batch jobs in the queue may use it as input, spreading the damage. Even worse, this corruption has been faithfully reproduced on the other data centers, so you now have no hot swap capability.

[nicolai]

Originally Posted by BEagle

So waiting for ba to sort out claims through fistfuls of receipts won't be an adequate response. The CAA has clearly decided that ba's passenger delays were not caused by 'extraordinary circumstances'; the EU law on flight compensation uses the term 'extraordinary circumstances' to refer to situations where delays or cancellations have been caused by things that are not the responsibility of the airline, whereas the airline clearly is responsible for its IT systems having sufficient resilience and redundancy.

Thus full compensation under EU law must be paid.

I'm afraid I don't agree with the CAA saying that. The CAA is saying that passengers are "passengers affected ... are protected under EU law", and this law provides for both a duty of care (in all circumstances) and compensation (in some circumstances). So the EU law will protect all passengers in case of travel disruption but may only require a duty of care.

The CAA is being reassuring to passengers and warning BA (and other airlines who may be watching) not to neglect the duty of care, rather than making any comments on compensation.

The duty of care does include getting passengers on their way as quickly as reasonable possible, for example - not excluding booking onto other carriers, which is often a sticking point for tight-fisted airlines. This may be one of the things the CAA is trying to remind the airlines of, again without any reference to compensation.

[crewmeal]

Although the rot really set in when Bob Ayling took over

Correct. Also wasn't he responsible for the demise of the BA franchises that were working so well for BA? To me that's when the rot really set in. Since then BMI, GB Airways, Maersk to name a few have all gone.

[FlightCosting]

BA will have to pay

Originally Posted by nicolai

I'm afraid I don't agree with the CAA saying that. The CAA is saying that passengers are "passengers affected ... are protected under EU law", and this law provides for both a duty of care (in all circumstances) and compensation (in some circumstances). So the EU law will protect all passengers in case of travel disruption but may only require a duty of care.

The CAA is being reassuring to passengers and warning BA (and other airlines who may be watching) not to neglect the duty of care, rather than making any comments on compensation.

The duty of care does include getting passengers on their way as quickly as reasonable possible, for example - not excluding booking onto other carriers, which is often a sticking point for tight-fisted airlines. This may be one of the things the CAA is trying to remind the airlines of, again without any reference to compensation.

In a roundabout way what the CAP is saying is that the airline scam of using a blanket extraordinary circumstances to avoid pay compensation is over the ECJ has ruled that the practice of unforseen technical delay being used to exclude claims when the aircraft was late from maintenance is no excuse. Neither will an act of God unless Willy Wonka is still considered to be .God

[p.j.m]

Originally Posted by Sonny_Jim

Would not be surprised in the slightest if the 'power supply problem' was actually a botched upgrade.'.

The "Hi Tech" parks in India where outsourcers run their operations from OFTEN have power outages (once a day is common). Believe it or not, they also have rats running across the floors.

[DuncanF]

According to someone who knows someone ... this is not a simple power failure. There has been a "failure" of core routers via the power supplies/backup power supplies to these routers. A targeted attack has not been ruled out. In some areas the network has up to six levels of resiliency, and these outer resilient layers have been targeted first.

This has been across multiple data centres, not all in the UK. In fact the primary UK data centre was the last to fall off the network. The underlying applications are mostly available, but inaccessible due to the network being down.

[BEagle]

Indeed, FlightCosting! I read the CAA statement as a firm reminder to Cruz that 'extraordinary circumstances' cannot be considered to apply in this case - he brought it on ba with his policies.

Doubly unfortunate for Cruz that amongst the passengers stuck in his Heathrow chaos there happened to be a Daily Mail journalist and his family: British Airways doesn't deserve name, says Robert Hardman | Daily Mail Online

I don't imagine that this comment from Cruz (again from the Daily Mail) will endear him to many:

Mr Cruz's email asking his workers not to comment

Guys,

either you are part of the team working to fix this or you aren’t. We are not in the mode of ‘debriefing on what happened’ but rather ‘let’s fix this mode’.

I have been answering some emails from colleagues (thank you for the support) and I just finished a video for media and another for all of us. I suppose you will have seen our Twitter account and BA.com.

Now, your interest on today’s events is well noted. I am wondering if you would like to help out? Either LHR or LGW or anyone of our stations could use you, now. If you indeed can, drop me a message and I will connect you or go straight to the airport and make yourself available. I am sure they will appreciate your involvement.

In the meantime, if you do not want to get involved or cannot get involved, I would kindly ask you to refrain from live commentary, unless it is a message of support to the thousands of colleagues that love BA as much as you do.

Thanks

Full article at British Airways' cost-cutting boss gags his staff | Daily Mail Online

[p.j.m]

Originally Posted by DuncanF

The underlying applications are mostly available, but inaccessible due to the network being down.

So its not a HP SAN fault? lol

[Nialler]

Originally Posted by BEagle

Indeed, FlightCosting! I read the CAA statement as a firm reminder to Cruz that 'extraordinary circumstances' cannot be considered to apply in this case - he brought it on ba with his policies.

Doubly unfortunate for Cruz that amongst the passengers stuck in his Heathrow chaos there happened to be a Daily Mail journalist and his family: British Airways doesn't deserve name, says Robert Hardman | Daily Mail Online

I don't imagine that this comment from Cruz (again from the Daily Mail) will endear him to many:



Full article at British Airways' cost-cutting boss gags his staff | Daily Mail Online

Asking staff to remain schtum in these events is standard industry practice. It's generally done for legal reasons. If someone names a piece of kit as the culprit and the subsequent enquiry reveals, for example, that it failed because it was being asked to operate outside its documented operational limits, then you will be in receipt of letters from Messrs Sue, Grabbit, and Runne pronto.


As is being indicated, there may be huge sums of money at the core of this.


Remember in this, that it seems to have happened at a time of very heavy use.

[Heathrow Harry]

Looks like they are cherry picking shorthaul flights today - one each way to Aberdeen cancelled mid- morning for example

BBC this morning

British Airways passengers are facing a third day of disruption at Heathrow as the airline deals with the impact of a worldwide computer system crash.
BA says it aims to operate a full long-haul schedule and a "high proportion" of short-haul services after the outage caused by a power failure.

So far on Monday, 13 short-haul flights at Heathrow have been cancelled.

All flights operated from Gatwick on Sunday but more than a third of services from Heathrow - mostly to short-haul destinations - were cancelled.

In a statement, BA said its IT systems were moving "closer to full operational capacity", and chief executive Alex Cruz has posted videos on Twitter apologising for what he called a "horrible time for passengers". But no-one from the airline has been made available to explain the cause of the system crash, and it has not explained why there was no back-up system in place.
BA added: "At Heathrow, we have operated virtually all our scheduled long-haul flights, though the knock-on effects of Saturday's disruption resulted in a reduced short-haul programme.
"We apologise again to customers for the frustration and inconvenience they are experiencing and thank them for their continued patience."

One of the affected passengers is Ian Sanderson who is stuck in-transit in London after setting off from his home in Luxembourg to travel to Iceland.
He said he was "incandescent with rage" after being unable to rebook his flight, or speak to a member of staff.
Image copyright Twitter Image caption Thousands of customers have taken to Twitter to vent their frustration Speaking on Sunday evening, he said: "I've bombarded them with about 100 tweets in the last 24 hours. I know that's annoying but there's nothing else I can do.
"We've tried to call them on the numbers they give and all we've got is the same recorded message which then cuts off at the end."

Former Virgin Airlines spokesman Paul Charles said: "What seems remarkable is there was no back-up system kicking in within a few minutes system failing.
"Businesses of this type need systems backing up all the time, and this is what passengers expect."

BA is liable to reimburse thousands of passengers for refreshments and hotel expenses, and travel industry commentators have suggested the cost to the company - part of Europe's largest airline group IAG - could run in to tens of millions of pounds.Customers displaced by flight cancellations can claim up to £200 a day for a room (based on two people sharing), £50 for transport between the hotel and airport, and £25 a day per adult for meals and refreshments.
On Saturday, travellers spent the night sleeping on terminal floors at Heathrow on yoga mats provided by BA.
The disruption continued into Sunday, with queues building up as passengers tried to rebook flights. Conference rooms at the airport were opened to provide somewhere more comfortable for passengers to rest.

BA said Heathrow was still expected to be congested on Monday and urged travellers not to go to the airport unless they had a confirmed booking for a flight that was operating.
It said passengers could get a full refund or rebook to travel up to the end of November but recommend they use its website.

[tinyGiraffe]

Originally Posted by DuncanF

According to someone who knows someone ... this is not a simple power failure. There has been a "failure" of core routers via the power supplies/backup power supplies to these routers. A targeted attack has not been ruled out. In some areas the network has up to six levels of resiliency, and these outer resilient layers have been targeted first.

This has been across multiple data centres, not all in the UK. In fact the primary UK data centre was the last to fall off the network. The underlying applications are mostly available, but inaccessible due to the network being down.

Not entirely true - the datacentres are both at LHR, not worldwide. However I think "power failure" is a fig-leaf of an explanation.

[RichardBeeb]

Anyone used to work there?

I'm the transport corr at the BBC and I'm looking to talk to anyone who used to work in BA IT...I'm on [email protected]

[BigFrank]

EU261/ 2004 inter alia

Originally Posted by Heathrow Harry

BA is liable to reimburse thousands of passengers for refreshments and hotel expenses, and travel industry commentators have suggested the cost to the company - part of Europe's largest airline group IAG - could run in to tens of millions of pounds.

i) Customers displaced by flight cancellations can claim up to £200 a day for a room (based on two people sharing), £50 for transport between the hotel and airport, and £25 a day per adult for meals and refreshments.


ii) On Saturday, travellers spent the night sleeping on terminal floors at Heathrow on yoga mats provided by BA.

i) These figures do not exist anywhere in the EU261/ 2004 legislation.

ii) These people cannot claim for the costs of overnight accommodation under
the above provisions since patently they did not contract said accommodation.

Harsh evaluation ? Yes indeed. But who ever believed the politicians and the HR bods when they said repeatedly that "life was fair" ?



(Indeed the simple fact that the vast majority of passengers at both LHR and LGW could not contract such provisions due to the elementary logic which dictates that all hotels will have been full within, let's say 4 hours of the incident also substantially limits the de facto bill under EU261, I would say. Though obviously the world wide ramifications must be borne in mind as well. Likewise the issue of a much smaller number of potentially much more costly compensation claims furth of EU261.)

The question of some ex gratia offer being made in the next few hours to assuage the current tsunami of negative publicity is also a factor I believe though as, unlike several posters here it would appear, I only know of the major players involved through often ignorant and biassed press publicity filtered through expensive PR companies, I am not in a position to speculate on the probability.

[WHBM]

Here's what I get on the BA website, 48 hours after the "failure", when I click on Flight Information to find out how my forthcoming flight is going.

[rideforever]

Originally Posted by Nialler

DC2 takes over, continues, but because DC1 didn't have time to communicate a synch point, and as a result of network latency, DC2 resumes at the incorrect moment of processing. Now may have a corrupt database. Worse, subsequent batch jobs in the queue may use it as input, spreading the damage. Even worse, this corruption has been faithfully reproduced on the other data centers, so you now have no hot swap capability.

Easy, you mirror the uncommitted database queue as much as is feasible.
One changeover you have a procedure to immediately process that queue, and if a few bits get missed then you deal with it in the field and accept that loss.
If you have to restart procedures for certain events (passenger boarding for instance) ... then these are isolated and expedited. Meaning 5 plane loads of passengers are disembarked and reembarked.
In other words every part of the procedure is planned for.
Nothing is out of process.
And practiced, a failure is just another day at work.
If airlines do not do this, they are completely irresponsible.
This is normal.

[HZ123]

No doubt it has been said but when the dust settles Mixed Fleet will be looking to take further industrial action as there is no resolution to the previous issues.

AC presently rallying the troops and no doubt applauding them, when it gets back to normal his senior managers will be looking to outsource and reduce more personnel.

If the longest crew trips are 7 days then clearly crewing issues may not be resolved entirely until next week. Time to start looking for an inducement to bring customers back ; drinks and catering on shorthaul?

[xs-baggage]

[QUOTE=Caribbean Boy;9785587]Isn't BA's main data centre in Boudicea House near Hatton Cross, with an unmanned data centre in nearby Cranebank? If there was a failure of hardware or software in Boudicea House, I would have expected the operation to run from Cranebank. There is UPS in both buildings, so how could a power supply issue cause such massive failure.[/QUOTEnk.

The high-level network diagram used to be stuck to the wall over my desk back in the day. RTZ ended up being hosted by Amadeus, with resilient links to both Amadeus data centres from both BOHO and Cranebank. I don't know the power supply side of it, but I'd imagine from its location that BOHO has grid supplies from LHR's Iver and Staines feeders as well as local emergency generation.