BA delays at LHR - Computer issue

[fchan]

Originally Posted by fergusd

Anybody with the barest minimum of knowledge would deploy only server chassis fitted with two power supplies, each fed from diverse electrical feeds in the data center, backed up using different UPS's each connected to different diesel backup generators. Professional kit deployed by professionals can always be deployed with multiple power supplies. Short of a catastrophic event at the data center (bomb/flood/plane crash) you don't lose power on all power feeds.

According to the Yahoo video recently posted here this may not be a good strategy. It’s introducing so much complexity that it may reduce reliability, especially if you consider the extra possibilities of fire and human error.

In one centre where we tried to reliability model the power system it was so complex we could not mathematically model it. Intuitively it had so many backups it seemed to be bullet proof, but if it can’t be modelled how can we be really sure?

[Gertrude the Wombat]

Originally Posted by fchan

Intuitively it had so many backups it seemed to be bullet proof, but if it can’t be modelled how can we be really sure ...

... that the backups aren't eventually circular, so that if one minor component goes down it takes out everything else with it?

[Snyggapa]

foolproof systems are, in my experience, no match for a determined fool

[sceh]

power still?

So BA are still pushing the power supply narrative even though no-one in IT believes them.
They have two data centres in active-active mode so one copies the other if need be. FIrstly they are both within a mile of the airport which is gross incompetence anyway - they ought to be many miles apart and many miles away.
They are trying to get us to believe that there is only one UPS for both? No. This was the original story and now it has changed to a power surge or the power coming back on uncontrollably which damaged servers and data.
Let's just suppose it did. They have back-up data and spare servers or at least they should have and they should have had tested plans for getting back within a specified recovery time objective - if they didn't it is yet more incompetence.
Since they outsourced, even if the in-house IT guys didn't raise these issues years ago the outsourcing company would have done or they would never have taken the contract.
Sorry BA but this doesn't hang together- there are other issues they are trying to hide

[LTNman]

I used to install UPS's and they have a limited lifespan so I wonder if anyone changed the batteries when they time expired? Best to do it every 3 years but many companies forget which isn't helped when they sack their IT staff.

[fenland787]

Originally Posted by Snyggapa

foolproof systems are, in my experience, no match for a determined fool

...or.....as my first boss used to say "you can make things foolproof but not bl**dy fool proof?"

[EDLB]

BA is in the air transport industry. There are time tested methods to investigate accidents. What do you learn form secrecy and obfuscation? Those managers have no place in this industry.

[blind pew]

EDLB
You don't understand the system - it's about protecting the establishment and in civil aviation that starts with the Munich disaster, followed by Staines, the Heathrow flypass, Concorde landing on trim fuel, 777... being the notable known whitewashing

[RAT 5]

A Parliamentary committee of MP's spent many hours, embarrassing at times, grilling Sir Philip Green (BHS) & Mike Ashley (Sports Direct) over their handling of affairs in their companies. Much dirty laundry was aired. Surely this fiasco at a company with much more international conspicuity and affect on UK's image would merit such scrutiny. I would hope that WW and his cohorts do not have enough lobbying influence to divert such thoughts. If Sir Phillip could not avoid a public enquiry why should the top cats at BA manage it.

[XBA1709]

When BA outsourced it's IT it got rid of 80% of it's data network staff(59 down to 12 'heads' ) and 84% of it's Operations Control staff(152 down to 24 'heads'). Ops are the people monitoring the IT domain 24/7 365 days a year. The numbers left after the reductions were almost all contracts managers and project managers - not the technical staff. This was all done within a 1 year timeframe so the chance of 'the new guys' being able to cope and respond correctly seems slight. Also one of the components of FLY was outsourced to TCS India for maintenance and support early on in this. It was rumoured that within 8 months all the Indian contractors who had been trained by established BA IT staff to do the work had left.

[B Fraser]

They have two data centres in active-active mode so one copies the other if need be. FIrstly they are both within a mile of the airport which is gross incompetence anyway

Why do you say that ? Why is it perfectly acceptable to have hotels full of people within an airport boundary but not datacentres that are some considerable distance apart ? Many companies have key IT nodes adjoining the airport and not one has been damaged due to their proximity to aircraft operations.


I used to give presentations to customers on the capabilities of a telecoms company who had their operations centre in Reading. One smart alec suggested that having a building so close to LHR was bad planning. I replied that MI5, Downing Street, Westminster, Buckingham Palace and Windsor Castle were significantly closer and the consequences should they be hit by a wayward aircraft were considerably greater. Nobody had suggested moving those functions to a place of greater safety.


There was one incident where an aircraft damaged a telecoms facility, it was the Algerian (?) freighter that crashed near Coventry (?) and took out a transmission mast. As the network was ring-protected, the customers didn't notice any interruption but that same operations centre in Reading knew that something had happened within an instant.

[feueraxt]

BA IT systems failure: Uninterruptible Power Supply was interrupted

...potentially by a panicking contractor, if reports are to be believed.

BA IT systems failure: Uninterruptible Power Supply was interrupted ? The Register

Bill Francis, Head of Group IT at BA's owner International Airlines Group (IAG), has sent an email to staff saying an investigation so far had found that an Uninterruptible Power Supply to a core data centre at Heathrow was over-ridden on Saturday morning. He said: "This resulted in the total immediate loss of power to the facility, bypassing the backup generators and batteries. This in turn meant that the controlled contingency migration to other facilities could not be applied. "After a few minutes of this shutdown of power, it was turned back on in an unplanned and uncontrolled fashion, which created physical damage to the system, and significantly exacerbated the problem.

The rest of the article makes interesting reading.

[c52]

I feel they should answer to Parliament because they benefit from and contribute to Britain's reputation in a way an airline with any other name doesn't.

[HHornet]

Originally Posted by Ian W

In a machine room from my past, that no longer exists, a security guard groped for the light switch and powered down the research data center using the emergency switch despite its switch cover - rather obviously the lights did not come on although several smaller ones went off . So the guard groped further and found the light switch and realized that the wrong switch had been used so switched it back on again and went on his way not realizing the chaos caused by the few minutes interruption and uncontrolled restoration.

This sounds like a similar occurrence.

Hum, on a tiny tiny scale compared to BA and many years ago, my colleague's company UPS (due to lack of space) was based in a storeroom/kitchen area. The IT team came from head office for an inspection, but realised that the cleaners had struggled one night to find a free power socket, so had unplugged the UPS to vacuum the offices. As there was a lack of spare sockets, the Sony Walkman toting cleaners had moved on from the fridge plug which was labelled "fridge - do not turn off" to the UPS plug which wasn't labelled or secured. Staff lunches were perfectly chilled but luckily the mains power didn't go down. So nothing was lost, lots was learned.

[Piltdown Man]

Firstly, I don't have a dog in this fight as don't work for or have shares in BA. I do work for a competitor but have no desire to see BA uneccesssarily punished, if for no other reason than good friends of mine work there. What I do have is my stake in British business, the business that will have to exist for my children to have a job. So that's my interest.

So what went on? I'll happily buy that part of BA's IT infrastructure was taken down by an internal power failure and more damage was done when it was restored. But what I don't believe is that only a power failure caused this mess. This mess was a direct result of incompetence by BA's IT department. An IT system is a collection of hardware, software and infrastructure ordered to receive, process an deliver data. The data handling process is there to support the company. And as the company changes the way it does business, any system modifications have to be designed, built and tested beforehand. BA clearly haven't done this. Instead, they have disposed of in-house talent as their system has grown and placed the future if the company in jeopardy, exactly the place where it is today. An organisation saving system patch can not be written in a few days, so BA is still very vulnerable.

BA's problems are in the greedy and uncaring way it does business. It appoints grasping, greedy sociopaths without background experience to run highly sophisticated operations and as a result have ripped out the skill, expertise and more importantly the heart of BA. Turning a profit is one thing but doing so day after day, week after week without a heart will not work. Firstly, you won't keep your good employees, after all why should they care about working for a company who cares nothing for them. Secondly, you won't keep your customers. They have been lied and cheated out of their flights and they won't take much more. The edge that BA had was a quality product delivered by professional caring staff. But that has gone. BA have screwed up over the past few years by placing greedy incompetent managers in positions they weren't qualified to be in. Worse, they have outsourced the guts of the organisation to the third world and they are now at the mercy of people who don't care about BA who work on a different continent.

This should not only be BA's wake up call but the wake up call to all of British business. You are on the wrong track. Squeezing the last few quid out of a company might look good in the short term but in the long term there will be nothing left. No heart, no company, no jobs and no society.


ps. Apologies to the professional BA staff who still work there. I know how you feel. When the heart and soul of a company you work and care for gets ripped out it takes some getting used to.

[WHBM]

Originally Posted by c52

I feel they should answer to Parliament because they benefit from and contribute to Britain's reputation in a way an airline with any other name doesn't.

I agree. BA is one of those companies which nowadays operates on a "Now I'm one thing, now I'm not" approach. They are very happy to trade with the national name, to receive the lion's share of the grandfathered slots at Heathrow, to lobby for all that UK government passenger business, and so on. But they then take all the money and spend it in Madrid or Mumbai, say because they are an international business they shouldn't take part in anything from the UK Parliament or elsewhere that they find inconvenient, and such like.

[Chris2303]

One wonders how many businesses have said to themselves "OMG! Better have a thorough check of our arrangements"

[PAXboy]

What we are seeing is just the natural progression of a company:

• Start up.
• Early problems.
• Merger with other start ups.
• Expansion (post war).
• Nationalisation so they know that they are 'king'.
• Big expansion as they get into their stride.
• Some bad choices but the times are on their side so expansion continues.
• Privatisation works well so they REALLY know they are king.
• They buy up or squeeze out the competition, even though they have to go to court, they survive.
• Arrogance sets in.
• Bad choices are covered by the overall expansion of the era.
• Competition starts to bite - and they don't know what to do.
• They try to follow the competition.
• Complacency ...

By this time they are already over their peak - but they don't know it yet. They have been successful for so long - obviously they are doing things right. Usually - at this point - the end is only a matter of time. They are like 'Road Runner' who has run off the edge of the cliff - and just don't realise that their wheels are spinning in mid air ...

In my view, their peak was about 2005 (the date can be argued in many directions but I pick that as a starting point) after this, it is probably all over but they have not realised it and only something astounding can save them..

If you want to look at other companies that have recently followed this path (as all corporates do) consider (not all stages apply, of course as I have angled this to the BA path):

• Woolworths
• Marks & Spencer
• Most British car and motorcycle companies
• All political parties
• Large department store chains
• PanAm
• TransWorldAirlines
• Alitalia
• The United Kingdom
• [fill in the blanks]

[fltplanner]

PAXboy - this all sounds persuasive, but if things are so predictably dire why is BA share price so bouyant ?

[Jetstream67]

FLTplanner

Probably because the big fundholders are expecting they either

Recover or
Get taken over

Good bet either way with all those slots