BA delays at LHR - Computer issue
Join Date: Jan 2010
Location: London
Posts: 379
Likes: 0
Received 0 Likes
on
0 Posts
I don't imagine that this comment from Cruz (again from the Daily Mail) will endear him to many:
Full article at British Airways' cost-cutting boss gags his staff | Daily Mail Online
Full article at British Airways' cost-cutting boss gags his staff | Daily Mail Online
"Guys, either you are part of the team working to fix this or you aren't".
I have never heard of any chief executive use this sort of language. Now, I don't want to hear of everyone being part of the "team", which BA does ad nauseam, but it jars with me that all BA employees are not made to feel part of the engaged workforce to help BA in its moment of crisis. Surely, as CEO, he should be asking everyone to do their bit to keep BA running and, if they are able, to help the beleaguered operations people at LHR and LGW.
Furthermore, it's hardly sensible to ask employees to keep quiet. After all, this is the biggest crisis to hit BA since volcanic ash in 2010 caused shutdown of European airspace for six days, so of course people are going to talk about it.
"Guys, either you are part of the team working to fix this or you aren't"
A number of contributors have blamed the Accountants for this outage, e.g. excessive cost cutting, lack of funds for IT kit. In defence of my profession, the BA Board makes the strategic, financial and operational decisons for the airline. It is there where you should be pointing the finger (as well as the Head of IT!).
For an instance from another industry, a friend told me that in a major telecoms company many years ago the accountants took a huge amount of persuading by the engineers that there was a business case for what we now call broadband internet. Instead of daring to imagine the scope of what could happen, and consequent new income streams, they were paranoid about losing pstn call revenue.
Well, we know now that they still managed to cock things up, with a deal described later by the Financial Times as “in hindsight ... one of the worst strategic errors in UK corporate history, given the rapid growth in mobile communications, as it left the former state-owned UK telecoms group without a cellphone business”.
Breaking News on BBC:
BA boss 'won't resign' over flight chaos - BBC News
(Being updated)
BA chief executive Alex Cruz says he will not resign and that flight disruption had nothing to do with cutting costs
(Being updated)
Join Date: May 2011
Location: Girona
Posts: 230
Likes: 0
Received 0 Likes
on
0 Posts
Different classes of foreigners ?
Or what about comparing it to Qatar Airways?
After all they do own 20% of IAG making them the single biggest shareholder by a clear margin.
Or are Qatar just useless foreign clowns like Walshe and Cruz, to name but two, not quite at random?
Without mentioning the Frenchman on the board with his weird near Romanian name.
After all they do own 20% of IAG making them the single biggest shareholder by a clear margin.
Or are Qatar just useless foreign clowns like Walshe and Cruz, to name but two, not quite at random?
Without mentioning the Frenchman on the board with his weird near Romanian name.
Join Date: Apr 2008
Location: London
Age: 59
Posts: 50
Likes: 0
Received 0 Likes
on
0 Posts
There is no evidence that Tata advertised those jobs as required by law, nor did they TUPE over jobs that were replaced with Tier 2 ICTs.
Tier 2 ICTs had to be paid a minimum £24k at that time, although as of a few weeks ago the rules around ICT have changed dramatically due to the wholesale abuse of them such as in this case. This £24k includes both their salary and non-taxable expenditure such as housing and travel. So after expenses, HMRC gets to see tax on about £12k salary if they're lucky, as opposed to tax on, a local on, earning say £40k. Even if you could find locals to do the job for £24k, they would be taxed on housing and travel unlike imported labour.
What amused me about the interview and BA statements were the following
1 We would never put our IT systems at risk
2 It was nothing to do with outsourcing
3 I will not resign
What caused it all then-we dont' know that yet, so you do not know why it happened yet you can make assertions like that
1 We would never put our IT systems at risk
2 It was nothing to do with outsourcing
3 I will not resign
What caused it all then-we dont' know that yet, so you do not know why it happened yet you can make assertions like that
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes
on
0 Posts
It is starting to look like WW bet with R Branson could have been the other way around. Virgin still going strong and not been swallowed by Delta. I do hope payment of the bet will be publicized.
https://www.theguardian.com/business...llie-walsh-bet
https://www.theguardian.com/business...llie-walsh-bet
Why is Cruz calling all the staff guys? I thought guys referred to males. A truly unprofessional attitude to women and the position. If he won't resign lets hope the shareholders boot him out.
Probably depart with a massive payoff and then appear else where on the corporate scene at another enhanced salary... Has been known to happen before, or am I just being cynical?!!!
Join Date: Aug 2007
Location: Tullamore
Posts: 27
Likes: 0
Received 0 Likes
on
0 Posts
This on the register in the comments:
"
Comment from a Times article.
From the IT rumour mill
Allegedly, the staff at the Indian data centre were told to apply some security fixes to the computers in the data centre. The BA IT systems have two, parallel systems to cope with updates. What was supposed to happen was that they apply the fixes to the computers of the secondary system, and when all is working, apply to the computers of the primary system. In this way, the programs all keep running without any interruption.
What they actually did was apply the patches to _all_ the computers. Then they shutdown and restarted the entire data centre. Unfortunately, computers in these data centres are used to being up and running for lengthy periods of time. That means, when you restart them, components like memory chips and network cards fail. Compounding this, if you start all the systems at once, the power drain is immense and you may end up with not enough power going to the computers - this can also cause components to fail. It takes quite a long time to identify all the hardware that failed and replace it.
So the claim that it was caused by "power supply issues" is not untrue. Bluntly - some idiot shut down the power.
Would this have happened if outsourcing had not be done? Probably not, because prior to outsourcing you had BA employees who were experienced in maintaining BA computer systems, and know without thinking what the proper procedures are. To the offshore staff, there is no context, they've no idea what they're dealing with - it's just a bunch of computers that need to be patched. Job done, get bonus for doing it quickly, move on."
https://forums.theregister.co.uk/for..._supply_issue/
I'm astonished, absolutely gobsmacked if this is true - never mind cocking up the change, that happens all the time, it's doing the change on a busy weekend at all! This speaks volumes about commodity staff, unfamiliar with airlines - for everyone else, a bank holiday is an opportunity to do housekeeping at a quiet time. In an airline, though, a bank holiday is the busiest time of the year. The US company I work for has change freezes around every major US holiday. When I worked at somewhere close to BA many moons ago, there were even freezes for large customer demos and trade shows.
Words fail me. Well, after the words above and some more I won't repeat here!
"
Comment from a Times article.
From the IT rumour mill
Allegedly, the staff at the Indian data centre were told to apply some security fixes to the computers in the data centre. The BA IT systems have two, parallel systems to cope with updates. What was supposed to happen was that they apply the fixes to the computers of the secondary system, and when all is working, apply to the computers of the primary system. In this way, the programs all keep running without any interruption.
What they actually did was apply the patches to _all_ the computers. Then they shutdown and restarted the entire data centre. Unfortunately, computers in these data centres are used to being up and running for lengthy periods of time. That means, when you restart them, components like memory chips and network cards fail. Compounding this, if you start all the systems at once, the power drain is immense and you may end up with not enough power going to the computers - this can also cause components to fail. It takes quite a long time to identify all the hardware that failed and replace it.
So the claim that it was caused by "power supply issues" is not untrue. Bluntly - some idiot shut down the power.
Would this have happened if outsourcing had not be done? Probably not, because prior to outsourcing you had BA employees who were experienced in maintaining BA computer systems, and know without thinking what the proper procedures are. To the offshore staff, there is no context, they've no idea what they're dealing with - it's just a bunch of computers that need to be patched. Job done, get bonus for doing it quickly, move on."
https://forums.theregister.co.uk/for..._supply_issue/
I'm astonished, absolutely gobsmacked if this is true - never mind cocking up the change, that happens all the time, it's doing the change on a busy weekend at all! This speaks volumes about commodity staff, unfamiliar with airlines - for everyone else, a bank holiday is an opportunity to do housekeeping at a quiet time. In an airline, though, a bank holiday is the busiest time of the year. The US company I work for has change freezes around every major US holiday. When I worked at somewhere close to BA many moons ago, there were even freezes for large customer demos and trade shows.
Words fail me. Well, after the words above and some more I won't repeat here!
Last edited by yoganmahew; 29th May 2017 at 13:14. Reason: add linky
None but a blockhead
Join Date: Nov 1999
Location: London, UK
Posts: 535
Likes: 0
Received 0 Likes
on
0 Posts
The World At One BBC interview was fun - lots of dodging around the 'why didn't you have a backup?' with a 'the real failure only lasted a short time' answer. Funny, I could have sworn the IT was really failed for a lot longer than that, even if you don't count the knock-on scheduling nightmare. On outsourcing/sacking the experts - "This had nothing to do with it. They were local systems manned by local staff" - I'm afraid at this point I may have yelled something at the wireless impugning yer man's truthfulness and parentage.
So far, it's been 'a power fluctuation' that 'disrupted the network in the data centre' for a short while, and when the network came back up 'the messaging' wasn't working. Which, assuming some correlation with what actually happened, are three things that should have redundancy or resilience in themselves, and furthermore shouldn't then trigger the next one in the chain. I'm going with poor design, poor implementation, poor testing, poor ops and nobody left who understands the whole system - the latter especially so, because the prior four factors won't exist if there's someone in charge who knows how things work (and has the authority to make them so, but that's another story...)
As for who's ultimate responsible for this, it's the board. Half the stuff I've written for CIO consumption has been about how to make a case to the board for the resources and buy-in to do things properly, because y'know, IT is just a cost centre and why the hell are we spending all this money on it when Mrs Chairman can use Google for free? (I wish that wasn't an accurate parody, but...)
So far, it's been 'a power fluctuation' that 'disrupted the network in the data centre' for a short while, and when the network came back up 'the messaging' wasn't working. Which, assuming some correlation with what actually happened, are three things that should have redundancy or resilience in themselves, and furthermore shouldn't then trigger the next one in the chain. I'm going with poor design, poor implementation, poor testing, poor ops and nobody left who understands the whole system - the latter especially so, because the prior four factors won't exist if there's someone in charge who knows how things work (and has the authority to make them so, but that's another story...)
As for who's ultimate responsible for this, it's the board. Half the stuff I've written for CIO consumption has been about how to make a case to the board for the resources and buy-in to do things properly, because y'know, IT is just a cost centre and why the hell are we spending all this money on it when Mrs Chairman can use Google for free? (I wish that wasn't an accurate parody, but...)
Join Date: Jun 1999
Location: world
Posts: 3,424
Likes: 0
Received 0 Likes
on
0 Posts
Why is Cruz calling all the staff guys? I thought guys referred to males. A truly unprofessional attitude to women and the position.
Having said that I still think that Cruz should go
..and the likes of Emirates (just as one examples). Interesting to plot the decline of BA with the growth of Emirates
Emirates airline profits plunge 82.5% in past year | UAE News | Al Jazeera
Join Date: May 2008
Location: Paris
Age: 60
Posts: 101
Likes: 0
Received 0 Likes
on
0 Posts
This on the register in the comments:
"
Comment from a Times article.
From the IT rumour mill
Allegedly, the staff at the Indian data centre were told to apply some security fixes to the computers in the data centre. The BA IT systems have two, parallel systems to cope with updates. What was supposed to happen was that they apply the fixes to the computers of the secondary system, and when all is working, apply to the computers of the primary system. In this way, the programs all keep running without any interruption.
What they actually did was apply the patches to _all_ the computers. Then they shutdown and restarted the entire data centre. Unfortunately, computers in these data centres are used to being up and running for lengthy periods of time. That means, when you restart them, components like memory chips and network cards fail. Compounding this, if you start all the systems at once, the power drain is immense and you may end up with not enough power going to the computers - this can also cause components to fail. It takes quite a long time to identify all the hardware that failed and replace it.
So the claim that it was caused by "power supply issues" is not untrue. Bluntly - some idiot shut down the power.
Would this have happened if outsourcing had not be done? Probably not, because prior to outsourcing you had BA employees who were experienced in maintaining BA computer systems, and know without thinking what the proper procedures are. To the offshore staff, there is no context, they've no idea what they're dealing with - it's just a bunch of computers that need to be patched. Job done, get bonus for doing it quickly, move on."
https://forums.theregister.co.uk/for..._supply_issue/
I'm astonished, absolutely gobsmacked if this is true - never mind cocking up the change, that happens all the time, it's doing the change on a busy weekend at all! This speaks volumes about commodity staff, unfamiliar with airlines - for everyone else, a bank holiday is an opportunity to do housekeeping at a quiet time. In an airline, though, a bank holiday is the busiest time of the year. The US company I work for has change freezes around every major US holiday. When I worked at somewhere close to BA many moons ago, there were even freezes for large customer demos and trade shows.
Words fail me. Well, after the words above and some more I won't repeat here!
"
Comment from a Times article.
From the IT rumour mill
Allegedly, the staff at the Indian data centre were told to apply some security fixes to the computers in the data centre. The BA IT systems have two, parallel systems to cope with updates. What was supposed to happen was that they apply the fixes to the computers of the secondary system, and when all is working, apply to the computers of the primary system. In this way, the programs all keep running without any interruption.
What they actually did was apply the patches to _all_ the computers. Then they shutdown and restarted the entire data centre. Unfortunately, computers in these data centres are used to being up and running for lengthy periods of time. That means, when you restart them, components like memory chips and network cards fail. Compounding this, if you start all the systems at once, the power drain is immense and you may end up with not enough power going to the computers - this can also cause components to fail. It takes quite a long time to identify all the hardware that failed and replace it.
So the claim that it was caused by "power supply issues" is not untrue. Bluntly - some idiot shut down the power.
Would this have happened if outsourcing had not be done? Probably not, because prior to outsourcing you had BA employees who were experienced in maintaining BA computer systems, and know without thinking what the proper procedures are. To the offshore staff, there is no context, they've no idea what they're dealing with - it's just a bunch of computers that need to be patched. Job done, get bonus for doing it quickly, move on."
https://forums.theregister.co.uk/for..._supply_issue/
I'm astonished, absolutely gobsmacked if this is true - never mind cocking up the change, that happens all the time, it's doing the change on a busy weekend at all! This speaks volumes about commodity staff, unfamiliar with airlines - for everyone else, a bank holiday is an opportunity to do housekeeping at a quiet time. In an airline, though, a bank holiday is the busiest time of the year. The US company I work for has change freezes around every major US holiday. When I worked at somewhere close to BA many moons ago, there were even freezes for large customer demos and trade shows.
Words fail me. Well, after the words above and some more I won't repeat here!
Finally, given that they're still on a background of TPF, the machines running TPF are typically z-Series enterprise servers from IBM. i.e. designed with internal redundancy and with continuous uptime as one of the core aspects of their architecture. Their power requirements have shrunk from a time when, yes, the airport lights might flicker as the beast was woken up, through to today's models, which are CMOS based and run off little more than a kettle connection. The meantime between failures on these machines is measured in years. They do not fail in the type of circumstances described.
Thanks for posting it, though.
Join Date: May 2008
Location: Paris
Age: 60
Posts: 101
Likes: 0
Received 0 Likes
on
0 Posts
Join Date: Jun 2009
Location: Canada
Posts: 464
Likes: 0
Received 0 Likes
on
0 Posts
That rumour seems dubious to me, but having worked with Indian outsourcing in a previous life, it's not that dubious.
Join Date: Oct 2001
Location: LHR
Posts: 170
Likes: 0
Received 0 Likes
on
0 Posts
I think Alex Cruz is choosing his words carefully. There may well be local staff working on local systems. Those local staff are likely to be onshore Indian contractors. The permanent BA staff have been thinned out greatly by redundancies so TCS contractors are filling the gaps. They will have been onshore for no more than a few months due to visa restrictions.
I also agree rumour is credible. BA lost the watchlist for a weekend due to incorrect procedures being followed by a contractor.
Many BA IT staff are choosing to take redundancy rather than gamble on a now somewhat flimsy career path. The IAG head of BA IT was once the head of cabin crew during the 2010 dispute. He's determined to get 91m euro savings over a 3 year period. Just as well, those savings are going to be needed now.
I also agree rumour is credible. BA lost the watchlist for a weekend due to incorrect procedures being followed by a contractor.
Many BA IT staff are choosing to take redundancy rather than gamble on a now somewhat flimsy career path. The IAG head of BA IT was once the head of cabin crew during the 2010 dispute. He's determined to get 91m euro savings over a 3 year period. Just as well, those savings are going to be needed now.