BA delays at LHR - Computer issue
Join Date: Apr 2007
Location: UK
Posts: 38
Likes: 0
Received 0 Likes
on
0 Posts
In a machine room from my past, that no longer exists, a security guard groped for the light switch and powered down the research data center using the emergency switch despite its switch cover - rather obviously the lights did not come on although several smaller ones went off.
Another time a maintainer tried to shut down one poorly performing widget using the maintenance screen; his mouse pointer slipped down one line in the menu from “shut down this widget” to “shut down all widgets” and he did not spot it, with serious results for the many operators using the widgets. In the aviation no blame culture it’s not the fault of the maintainer or security guard, it’s poor design or procedures. Proper analysis and testing in the design and build phase should find this.
I am surprised to hear here that servers can be so sensitive to mains problems. I guess the problem is that that their designers assume they are always fed from a nice clean UPS supply and, if the UPS fails to filter the surges, brownouts and spikes, the server has no designed in protection against them. The UPS manufacturers will probably claim it can never happen.
Is it a fair guess that all other computers, maybe for miles around, on the same power network had problems at the same time on Saturday morning?
Thought not.
My experience of working with a UPS goes back to the 1980s when the one thing that could be guaranteed to cause an outage was a demo of the UPS.
Thought not.
My experience of working with a UPS goes back to the 1980s when the one thing that could be guaranteed to cause an outage was a demo of the UPS.
Join Date: Jan 2008
Location: Wintermute
Posts: 76
Likes: 0
Received 0 Likes
on
0 Posts
I am surprised to hear here that servers can be so sensitive to mains problems. I guess the problem is that that their designers assume they are always fed from a nice clean UPS supply and, if the UPS fails to filter the surges, brownouts and spikes, the server has no designed in protection against them. The UPS manufacturers will probably claim it can never happen.
Join Date: Jan 2008
Location: Somewhere colder than my clothes.
Age: 61
Posts: 59
Likes: 0
Received 5 Likes
on
3 Posts
At last, after an age of lurking, a subject where I do feel qualified to comment. In my organisation the operation of data rooms is split between IT and Real Estate - I am an engineer working for the latter and am responsible for all M+E services supporting the equipment. (Anything inside the racks belongs to IT).
For highly critical loads there is always more than a single UPS supplying critical power - at least 2 and more usually 4 or more. Cooling is also on a redundant backup system, details vary depending on the method used, ditto generators etc and we even consider water buffer tanks. Technically these are 2N services with distributed supplies and need to be fault tolerant AS A SYSTEM. There is also a similarly resilient backup data room in a distant location.
There's always scope for idiocy, but any work is controlled by a comprehensive change control system with at least 3 individual approvers before it's authorised. I can't comment with the same authority on the software side although an element of resilience is always included, and failover procedures are regularly practiced by IT. At least once per year I fail various elements of the support systems to test the backups and stress test the whole installation.
There is absolutely no way a well designed and maintained IT infrastructure could fail so catastrophically from a power irregularity - this is bread + butter stuff. An installation that is not well designed or maintained however.....
For highly critical loads there is always more than a single UPS supplying critical power - at least 2 and more usually 4 or more. Cooling is also on a redundant backup system, details vary depending on the method used, ditto generators etc and we even consider water buffer tanks. Technically these are 2N services with distributed supplies and need to be fault tolerant AS A SYSTEM. There is also a similarly resilient backup data room in a distant location.
There's always scope for idiocy, but any work is controlled by a comprehensive change control system with at least 3 individual approvers before it's authorised. I can't comment with the same authority on the software side although an element of resilience is always included, and failover procedures are regularly practiced by IT. At least once per year I fail various elements of the support systems to test the backups and stress test the whole installation.
There is absolutely no way a well designed and maintained IT infrastructure could fail so catastrophically from a power irregularity - this is bread + butter stuff. An installation that is not well designed or maintained however.....
Join Date: May 2017
Location: UK
Posts: 4
Likes: 0
Received 0 Likes
on
0 Posts
For those of you who didn't manage to read the 67 page Thomson Reuter transcript of the Nov 2015 IAG Capital Markets Day presentation here's a direct quote from page 46
'Willie Walsh - International Airlines Group - Group CEO
So I come back to what this is all about. Somebody asked me during the break there you know did I really give a speech around show me the money in a tapas bar in Madrid. And I have to say I was insulted because those of you that know me if you can picture me in the tapas bar with a glass of wine in my hand, I didn't say show me the money. I said show me the (expletive) money. Silvia stand up. Shout it out. Shout it out with pride. Come on.
Silvia Cairo Jordan - International Airlines Group - Head of Group Commercial Planning and Policy
Show me the (expletive) money.
Willie Walsh - International Airlines Group - Group CEO
All right. I'll bring you back to show me the (expletive) money. And we make no apologies for it. You know because we're in a new industry. And you've heard me talk about this. You've heard me talk about the airline industry being different. I honestly believe it is. Because it's no longer an industry that's driven by those capacity statistics, ASKs, people going out there talking about the way they're growing. It's an industry that's now driven by capacity balanced with unit revenue. Capacity balance by unit costs. Capacity balanced by return on investment. There are more and more airline leaders talk about return investment. So we're here to show you the (expletive) money today. Because look at what we're doing. We're increasing all of our targets. And we're increasing all of our targets because we are determined to deliver on them. We know we can deliver on them. We've got a proven track record of delivery. '
Although the quote is from the film Jerry Maguire I get the impression he thinks he is the new Gordon Gekko. He certainly demonstrates what he is concentrating on and it is certainly not service or the customer.
'Willie Walsh - International Airlines Group - Group CEO
So I come back to what this is all about. Somebody asked me during the break there you know did I really give a speech around show me the money in a tapas bar in Madrid. And I have to say I was insulted because those of you that know me if you can picture me in the tapas bar with a glass of wine in my hand, I didn't say show me the money. I said show me the (expletive) money. Silvia stand up. Shout it out. Shout it out with pride. Come on.
Silvia Cairo Jordan - International Airlines Group - Head of Group Commercial Planning and Policy
Show me the (expletive) money.
Willie Walsh - International Airlines Group - Group CEO
All right. I'll bring you back to show me the (expletive) money. And we make no apologies for it. You know because we're in a new industry. And you've heard me talk about this. You've heard me talk about the airline industry being different. I honestly believe it is. Because it's no longer an industry that's driven by those capacity statistics, ASKs, people going out there talking about the way they're growing. It's an industry that's now driven by capacity balanced with unit revenue. Capacity balance by unit costs. Capacity balanced by return on investment. There are more and more airline leaders talk about return investment. So we're here to show you the (expletive) money today. Because look at what we're doing. We're increasing all of our targets. And we're increasing all of our targets because we are determined to deliver on them. We know we can deliver on them. We've got a proven track record of delivery. '
Although the quote is from the film Jerry Maguire I get the impression he thinks he is the new Gordon Gekko. He certainly demonstrates what he is concentrating on and it is certainly not service or the customer.
Join Date: Jan 2006
Location: Gatwick
Posts: 117
Likes: 0
Received 0 Likes
on
0 Posts
I have worked in IT for about 30 years and have some experience with regard DR planning and requirements.
Datacentres are generally, as has been said already, fed by 2 or more diverse feeds from the grid. These then go via a UPS which is either in parallel or in series with the power feed. The major difference here is that the UPS can be used for removal of power spikes etc if it is in series and reduces the need for another switch that will flick in should the power fail.
If the input power does fail, the UPS will provide power to the building for anything from 5 minutes to several hours. This is dependant on the servers etc being powered. BAckup generators should kick in as soon as power fails. This gives them 5 minutes or more to warm up and settle down before the UPS batteries run out of power.
Assuming all of the above works, power then continues to be provided for the rest of time as long you have a plan in palce to refil the diesel tanks of the generators before they run dry. In reality, mains power comes back on line, the generators turn themselves off and normality is restored.
What tends to happen though is that the backup generator fails to start, the UPS batteries have not been replaced in the last 10 years and when the mains power goes down, you have all systems falling over in a heap. This is the time to switch to the DR data centre.
This is where the real problems begin. All too often this has not been tested properly. Systems have different RTO (recovery time objectives) which can vary from instant to weeks and many people don't realise that system#1 which has an instant RTO is and but cannot run as it is dependant on system#73 which is on a 3 day RTO. Not only that, when they tested it, the production system didn't fail, it was cleanly shut down, meaning that you now have corrupted data in system#1 and that brings an whole new head ach for people to sort out.
There is a common phrase I use which I don't think the forum software will allow but here it is in another format.
In IT, Sierra-Hotel-India-Tango happens.
Datacentres are generally, as has been said already, fed by 2 or more diverse feeds from the grid. These then go via a UPS which is either in parallel or in series with the power feed. The major difference here is that the UPS can be used for removal of power spikes etc if it is in series and reduces the need for another switch that will flick in should the power fail.
If the input power does fail, the UPS will provide power to the building for anything from 5 minutes to several hours. This is dependant on the servers etc being powered. BAckup generators should kick in as soon as power fails. This gives them 5 minutes or more to warm up and settle down before the UPS batteries run out of power.
Assuming all of the above works, power then continues to be provided for the rest of time as long you have a plan in palce to refil the diesel tanks of the generators before they run dry. In reality, mains power comes back on line, the generators turn themselves off and normality is restored.
What tends to happen though is that the backup generator fails to start, the UPS batteries have not been replaced in the last 10 years and when the mains power goes down, you have all systems falling over in a heap. This is the time to switch to the DR data centre.
This is where the real problems begin. All too often this has not been tested properly. Systems have different RTO (recovery time objectives) which can vary from instant to weeks and many people don't realise that system#1 which has an instant RTO is and but cannot run as it is dependant on system#73 which is on a 3 day RTO. Not only that, when they tested it, the production system didn't fail, it was cleanly shut down, meaning that you now have corrupted data in system#1 and that brings an whole new head ach for people to sort out.
There is a common phrase I use which I don't think the forum software will allow but here it is in another format.
In IT, Sierra-Hotel-India-Tango happens.
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes
on
0 Posts
The problem is often not hardware or software but 'wetware' as the human element is sometimes called. Or to put it another way - "Nothing is fool proof as fools are too ingenious".
Somewhere I worked in the past had a uninterruptable power supply providing smoothed power for the sensitive electronics being used - and the system would crash many evenings for no apparent reason. It was then found that a cleaner was using one of the smoothed power supply sockets in the machine room for a vacuum cleaner putting noise and spikes into the power not realizing the difference between the black sockets with smoothed power and the white sockets for 'domestic' power.
Somewhere I worked in the past had a uninterruptable power supply providing smoothed power for the sensitive electronics being used - and the system would crash many evenings for no apparent reason. It was then found that a cleaner was using one of the smoothed power supply sockets in the machine room for a vacuum cleaner putting noise and spikes into the power not realizing the difference between the black sockets with smoothed power and the white sockets for 'domestic' power.
Join Date: Sep 2004
Location: Sweden
Posts: 61
Likes: 0
Received 0 Likes
on
0 Posts
Tieto , one of the larger outsourcing companies in the Nordic countries had a major datacenter crash about two years. This was caused by a storage area network (SAN) turning corrupt during an upgrade. For redundancy they had a second storage set but that did not help as it also went corrupt doing mirroring in real time.
When changing recovery strategy to restore it was found out the backup was not restoreable due to upgraded OS (or something like that).
Service for several large organisation was down for almost a week.
The problems for BA at Heathrow reminds me about this as both the wide outage as well as the long time before being operational again are similar.
When changing recovery strategy to restore it was found out the backup was not restoreable due to upgraded OS (or something like that).
Service for several large organisation was down for almost a week.
The problems for BA at Heathrow reminds me about this as both the wide outage as well as the long time before being operational again are similar.
Join Date: Aug 2002
Location: Europe
Posts: 4
Likes: 0
Received 0 Likes
on
0 Posts
Totally agree
Pretty much by definition anyone who ends up running a large company is an ambitious sociopath, because if you are not, you will not get there. Two of the largest firms in the world (Amazon and Apple) were built by deeply unpleasant people (Bezos & Jobs). Society at large lets it persist because - "they get things done" and, why would anyone normal want the role anyway?
We all know you are 100% right. We need nice, kind human beings running our societies. Happiness levels are too low in high GDP countries, this is the direct result of uncontrolled corporate greed where sociopaths are significantly rewarded and those who want to improve happiness for most are excluded by the system.
Often the same sociopaths are heavily involved in charitable missions, such a paradox and so fake
Last edited by ILS27LEFT; 31st May 2017 at 22:33.
Join Date: Jan 2015
Location: Centre of Universe
Posts: 371
Likes: 0
Received 0 Likes
on
0 Posts
Was at LHR on Saturday. Bags pitched up on yesterdays flight so 2 days late.
(only problem is two of en famille are stuck in UK but that's a minor point)
From my perspective as one of xxxxx thousand punters - can't grumble too much, all forgotten and we'll survive. Thumbs up to BA front line staff Saturday.
(only problem is two of en famille are stuck in UK but that's a minor point)
From my perspective as one of xxxxx thousand punters - can't grumble too much, all forgotten and we'll survive. Thumbs up to BA front line staff Saturday.
Join Date: Oct 2016
Location: Ontario
Posts: 0
Likes: 0
Received 0 Likes
on
0 Posts
Looks like senior figures at BA/IAG want an independent investigation - I imagine it'll end up going to one of the big consulting companies (e.g. PwC). Quite how the incentives will play out when the consulting company has its own advisory services to sell is something that leaves me rather puzzled
BA board members are expected to request an inquiry into the IT fiasco - BBC News
BA board members are expected to request an inquiry into the IT fiasco - BBC News
Paxing All Over The World
Unfortunately, sending a 'heavy mob' in to ask questions - will not give you the answers. Everyone will now be out to protect their own @rse and point the blame. They will have only been there a short time, they were told to do this, they were in the process of checing procedures when X did Y and the Z happened. etcetera.
If the company had a strong line of communication, where the senior IT mgmt were KNOWN and VISIBLE? Then you can get the information you need.
Since it is clear that it was an utter failure, at every level, You might be better off starting form scratch. Trying to work out what went wrong and then patching it? Again? But, as davidjohnson6 says, the 'heavy mob' will be only too glad to specify a new system. That is, one that no one at the top understands. Oh, that looks like what they already have.
If the company had a strong line of communication, where the senior IT mgmt were KNOWN and VISIBLE? Then you can get the information you need.
Since it is clear that it was an utter failure, at every level, You might be better off starting form scratch. Trying to work out what went wrong and then patching it? Again? But, as davidjohnson6 says, the 'heavy mob' will be only too glad to specify a new system. That is, one that no one at the top understands. Oh, that looks like what they already have.
Last edited by PAXboy; 31st May 2017 at 22:58.
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes
on
0 Posts
This is one of the problems.
PR says give the punters an excuse that sounds good and won't affect pax or share prices.
Sensible organization management says: find out the truth so we don't go down the same hole again.
These are often conflicting aims.
PR says give the punters an excuse that sounds good and won't affect pax or share prices.
Sensible organization management says: find out the truth so we don't go down the same hole again.
These are often conflicting aims.
Thread Starter
Join Date: Apr 2010
Location: London
Posts: 7,072
Likes: 0
Received 0 Likes
on
0 Posts
I have to say that BA are losing the plot with every passing minute - I've spent the last few days traveling and the problems have come up in many conversations
Not one single person believes a word they issue
"the story changes once or twice a day "..
" where was the CEO?"
"all those poor people and not a single manager to be seen"
"Did no-one ever TEST these systems?"
This is heading towards being a "Ratner" moment if they aren't careful
Not one single person believes a word they issue
"the story changes once or twice a day "..
" where was the CEO?"
"all those poor people and not a single manager to be seen"
"Did no-one ever TEST these systems?"
This is heading towards being a "Ratner" moment if they aren't careful
I've been wading through it as well. All the breathless gushing may be necessary for investors, but are they actually saying anything, other than endless efforts to reduce the cost base ?. From what I can see, they are upgrading all their IT to a common standard across the group and that work is still in progress, so plenty of room for trouble as the old kit is replaced by new. The software complexity must be horrendous, not to mention all the hardware and networking kit assocuiated with it. Admirable aim anyway.
Couple of things sprang out:
“It's using the Cloud, and we're really positive that we've now got both the BA and IB selling teams, who are able to do dealing and, also, workflow within a single system in absolutely future technologies”…
and:
“But, equally, you're going to see a much more Windows environment”....
So, up to date with the latest trendy cloud tech, but is it organised in a fully redundant manner and with guaranteed real time perfomance and failover ?.
Since when was anything from uSoft ever considered robust enterprise class ?. Fine for desktops and order processing, but wouldn’t be using it for any critical infrastructure, anywhere.
If you want stuff that works from top to bottom, go to IBM or Oracle. Very expensive, but good reason for it…
Couple of things sprang out:
“It's using the Cloud, and we're really positive that we've now got both the BA and IB selling teams, who are able to do dealing and, also, workflow within a single system in absolutely future technologies”…
and:
“But, equally, you're going to see a much more Windows environment”....
So, up to date with the latest trendy cloud tech, but is it organised in a fully redundant manner and with guaranteed real time perfomance and failover ?.
Since when was anything from uSoft ever considered robust enterprise class ?. Fine for desktops and order processing, but wouldn’t be using it for any critical infrastructure, anywhere.
If you want stuff that works from top to bottom, go to IBM or Oracle. Very expensive, but good reason for it…
Join Date: May 2011
Location: Girona
Posts: 230
Likes: 0
Received 0 Likes
on
0 Posts
IAG CEO " soon to be spotted "
So far he has been conspicuous by his absence. Very.
But the BBC link above states that he will be on parade this morning at BCN launching Level; IAG's new trans continental LCC.
¿ Will he take questions from journos or will he emulate Spanish PM Señor Rajoy and deliver his address from behind a high tech (sic) one way mirror ?
( As a shareholder, one hopes at very least that Willie's pathetic barrack-room language, quoted above by a business news agency no less, will be kept under strict control. Or better still only deployed, strictly in private, on those responsible for the current fiasco.)
But the BBC link above states that he will be on parade this morning at BCN launching Level; IAG's new trans continental LCC.
¿ Will he take questions from journos or will he emulate Spanish PM Señor Rajoy and deliver his address from behind a high tech (sic) one way mirror ?
( As a shareholder, one hopes at very least that Willie's pathetic barrack-room language, quoted above by a business news agency no less, will be kept under strict control. Or better still only deployed, strictly in private, on those responsible for the current fiasco.)
“It's using the Cloud, and we're really positive that we've now got both the BA and IB selling teams, who are able to do dealing and, also, workflow within a single system in absolutely future technologies”…
and:
“But, equally, you're going to see a much more Windows environment”
and:
“But, equally, you're going to see a much more Windows environment”
Last edited by c52; 1st Jun 2017 at 07:02. Reason: Grammar (obviously, when I write something like that).
So the Cyber Security Governance Board reports to the main board via a professional spin doctor. Is this a person who's likely to tell inconvenient truth to power?
¿ Does the fact that she doesn't have a Spanish (or Irish, or for that matter French) -sounding name mean that she is in the clear ?
(Just in case anyone might be confused, 'communications' in the above means 'public relations', aka 'spin', not telecoms!)