BA delays at LHR - Computer issue
Join Date: Jan 2010
Location: London
Posts: 379
Likes: 0
Received 0 Likes
on
0 Posts
So, some airlines gained from shared development and running costs, but it looks unlikely to me that this would account for the IT underspend compared with other industries.
Join Date: May 2009
Location: Confoederatio Helvetica
Age: 68
Posts: 2,847
Likes: 0
Received 0 Likes
on
0 Posts
Revenues is a bad measurement in an industry that alost always failes to achieve a 5% ROI. Airlines make a few dollars per passenger, in the good years.
Join Date: Mar 2014
Location: Rainsville
Posts: 51
Likes: 0
Received 0 Likes
on
0 Posts
If that is a low percentage it is probably because airlines have such massive big budget items on this like airframe depreciation and fuel that perhaps it pushes the IT percentage down compared to say a bank which spends relatively more in percentage terms on IT because it doesn't have to buy things like A 380s or spend a fortune on fuel.
The difference is probably that banks face a competitive threat from tech startups. In a cashless economy, deposit-taking banks are potentially redundant (and who would mourn them?), but until one of us invents a teleportation device we'll probably still need physical aircraft.
Join Date: Sep 2009
Location: Haifa
Age: 69
Posts: 18
Likes: 0
Received 0 Likes
on
0 Posts
I remember a lightning strike that killed the UPS and cutover/bypass switch for a large data center. The result was an inability to restart anything until a new switch was installed. Then we discovered that about 20% of disks had failed - they had been spinning for years and many were damaged by the sudden power cut.
This was mid 90's - probably the same technology BA use now.
This was mid 90's - probably the same technology BA use now.
The cost of this single failure exceeds the cost of a same sized brand new data center by a factor between 5 and 10. That tells all about the risk assessment quality of BA IT.
Join Date: Jan 2016
Location: Uk
Posts: 74
Likes: 0
Received 0 Likes
on
0 Posts
to be fair, even if a data centre was wiped out by a power outage, and even if the power came back on haphazardly, and even if a number of devices physically failed due to this - a company of that size dependent on systems to run it's core business should be able to continue. It sounds like this BA have more that one data centre but for whatever reason failing over to it clearly didn't work as planned.
It doesn't matter how many data centres you have, if you can't reliably fail over then you may as well just have one. This smells like a failing in application/system design and testing.
However don't underestimate the complexity of designing systems this way that integrate with both legacy and external systems - it is very difficult to do, to test and is expensive. Every penny I spend on disaster planning is money that ultimately I hope to be wasted - but just sometimes, it is not and it pays itself back 100 fold
It doesn't matter how many data centres you have, if you can't reliably fail over then you may as well just have one. This smells like a failing in application/system design and testing.
However don't underestimate the complexity of designing systems this way that integrate with both legacy and external systems - it is very difficult to do, to test and is expensive. Every penny I spend on disaster planning is money that ultimately I hope to be wasted - but just sometimes, it is not and it pays itself back 100 fold
Last edited by Snyggapa; 8th Jun 2017 at 09:47. Reason: typo
Can I offer a perspective. I was around a company 12 years ago where someone did a Chernobyl experiment by disconnecting the UPS and trying something. This tripped the circuit breakers and killed the SAN (storage area network) and about 500 servers. Then the company discovered the failover data centre was incorrectly sized and went down as well. The SAN subsequently refused to restart and then the Japanese manufacturer informed the company that Sorry but the SAN was no longer supported as it was an old version. The company paid a lot of money to fly somebody from Japan and it eventually was restarted after a few days. But it cost the company £1m in penalty clause payments to their system users. There were a number of banks and Health systems involved in that but was kept quiet for obvious reasons.
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes
on
0 Posts
This is the way it always happens.
The actual failure (finger trouble) should have had no effect. BUT
The failover data centre was too small, possibly from inception or because nobody had thought to check the growth against backup. This is an IT management failure of testing and continual monitoring.
The SAN being out of date is again an IT management failure as the continued maintenance and support of special equipment that supports the business should be part of normal IT management checks.
The other management failure shown by both of these is the failure to set flags that 'when the system gets this big' or 'when the system gets this old' a new system will be needed. Those flags should have been set on acceptance of the system and a draft contingency plan for how to replace the system should have been written and maintained.
With a cost cutting management it can prove very difficult to persuade them that the IT system which is still humming along, will need replacing in 18 months time and as the draft replacement plan says it will take 15 months to replace work needs to be paid for now to initiate contracts for replacement.
The actual failure (finger trouble) should have had no effect. BUT
The failover data centre was too small, possibly from inception or because nobody had thought to check the growth against backup. This is an IT management failure of testing and continual monitoring.
The SAN being out of date is again an IT management failure as the continued maintenance and support of special equipment that supports the business should be part of normal IT management checks.
The other management failure shown by both of these is the failure to set flags that 'when the system gets this big' or 'when the system gets this old' a new system will be needed. Those flags should have been set on acceptance of the system and a draft contingency plan for how to replace the system should have been written and maintained.
With a cost cutting management it can prove very difficult to persuade them that the IT system which is still humming along, will need replacing in 18 months time and as the draft replacement plan says it will take 15 months to replace work needs to be paid for now to initiate contracts for replacement.
Join Date: Jan 2010
Location: London
Posts: 379
Likes: 0
Received 0 Likes
on
0 Posts
Why would the contracts need to be placed so early? If anything, I would have thought that the biggest delay would be in getting budget approval past the Finance committee.
Was the outsourcing Cruz's idea ?
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes
on
0 Posts
1. has just passed one of the indicators that it needs replacement that were declared at its installation. With say system hardware or software becoming unsupported in 2 years or in 2 years after a merger you will need to add a subsidiary airline to the system taking it above its design limits..
2. Installation and acceptance and operational roll out will take 15 months
3. Getting 'finance committee' or board approval will take at least 3 months
4. Contract creation, bid process, selection and formal contract let will take at least 3 months
Then you will have to proceed immediately initiating the replacement project for the system. This is when the biggest battle will be with the bean counters and the CFO and board. It is also where the poorest decisions are made.
Putting head of IT as reporting to the CFO is almost certain to lead to attempts to continue with the old system that the head of IT knows will fail. Head of IT has the choice of making noise to do the right thing that will often result in a pink slip along with the introduction of the outsourcing group who also know the system will fail, but know if it does they control the company as the outsourcing group are now the only people who can provide a solution at their own inflated cost.
Join Date: Nov 2015
Location: BRS/GVA
Posts: 342
Likes: 0
Received 0 Likes
on
0 Posts
Have a read of this https://www.theregister.co.uk/2017/0...path_analysis/
analysis by a leading IT industry news site
analysis by a leading IT industry news site
[Hoss
Thanks for that link to a very objective and analytical piece. I am sure that in BA thre is a need for various systems to interact. For example the check in process isnt confined to that action since the operational side of the airline needs to know the pax and baggage weights for fuel and also performance calculations and wight and balance. That could mean there is a need to connect between different systems but as the author points out hats ok if it is between two or four systems but dozens!! It is almost guaranteeign disaster when somethign goes wrong.
Some organsiations deal with this kind of complex interworking by ensuring their key peopel are retained because they built the systems and the interfaces and can fix them. very hard to outsource something very complex and customised . General accounting and sales functions maybe yes since theyare common to many companies. Airline IT systems are in a different elague.
The other point is that made continually on hear that a cost cutting management does not listen to 'this equipment is time expired it has to be replaced. Obviously in an airline aircraft and engines have that kind of condition attached to them but the management don't really have any choice because those issues are regulated but with things like IT systems and UPS there is huge temptation to defer replacements because anew UPS for Cranebank doesnt lower seat mile costs and doesnt produce any more bookings and so its only and immediate effect is on the cost side of the equation, that is of ciurse untiil the wheels come off like in this case.
Management platitudes are common to all companies and not just airlines but airlines are not immune. For abit of amusement I wrote to Easyjet a few years ago and saying that as they had introduced seat reservations Iwas tempted to try them but as they were low cost could I be sure they were safe. Of course I got a lovely PR answer back that safety was their first priority etc etc etc . I wrote back, after a little reserach on the company and said @Yes, i have heard that but if safety is your fist priority why isnt it a responsibility for any member of your board of directors who all seemed to have a fiance or marketing background.
And its the same here no one high enough up who can stand up to the Aer Iberia duo and say , you wont like this boss but our IT system is very very vulnerable and something needs to be done because they know the answer is a P45
Thanks for that link to a very objective and analytical piece. I am sure that in BA thre is a need for various systems to interact. For example the check in process isnt confined to that action since the operational side of the airline needs to know the pax and baggage weights for fuel and also performance calculations and wight and balance. That could mean there is a need to connect between different systems but as the author points out hats ok if it is between two or four systems but dozens!! It is almost guaranteeign disaster when somethign goes wrong.
Some organsiations deal with this kind of complex interworking by ensuring their key peopel are retained because they built the systems and the interfaces and can fix them. very hard to outsource something very complex and customised . General accounting and sales functions maybe yes since theyare common to many companies. Airline IT systems are in a different elague.
The other point is that made continually on hear that a cost cutting management does not listen to 'this equipment is time expired it has to be replaced. Obviously in an airline aircraft and engines have that kind of condition attached to them but the management don't really have any choice because those issues are regulated but with things like IT systems and UPS there is huge temptation to defer replacements because anew UPS for Cranebank doesnt lower seat mile costs and doesnt produce any more bookings and so its only and immediate effect is on the cost side of the equation, that is of ciurse untiil the wheels come off like in this case.
Management platitudes are common to all companies and not just airlines but airlines are not immune. For abit of amusement I wrote to Easyjet a few years ago and saying that as they had introduced seat reservations Iwas tempted to try them but as they were low cost could I be sure they were safe. Of course I got a lovely PR answer back that safety was their first priority etc etc etc . I wrote back, after a little reserach on the company and said @Yes, i have heard that but if safety is your fist priority why isnt it a responsibility for any member of your board of directors who all seemed to have a fiance or marketing background.
And its the same here no one high enough up who can stand up to the Aer Iberia duo and say , you wont like this boss but our IT system is very very vulnerable and something needs to be done because they know the answer is a P45
How on earth is WW going to determine who are the 'key people' that he needs to retain?
The system is in continual flux, as new services and new hardware are deployed, so the architecture may have changed as well. I don't think keeping people would have helped avert this disaster, rather BA needs to have some sort of fallback for when it all goes bad.
The system is in continual flux, as new services and new hardware are deployed, so the architecture may have changed as well. I don't think keeping people would have helped avert this disaster, rather BA needs to have some sort of fallback for when it all goes bad.
The other point is that made continually on hear that a cost cutting management does not listen to 'this equipment is time expired it has to be replaced. Obviously in an airline aircraft and engines have that kind of condition attached to them but the management don't really have any choice because those issues are regulated
A few years ago the internal state of the BA "European" 767 fleet, at a time when they were still employed on the Moscow route, had got into a very dishevelled state, principally due to engineering having to conform to tight budgets. Over on Flyertalk the Moscow regulars, generally seniors from the banking and oil industries, started to compile a Tech Log, by airframe registration (it was quite a small fleet), listing all the recurrent defects. One aircraft G-BZHC, was known to all as "the Disco" because of defective flashing lights. A number of the issues were indeed safety-related.
Came back on G-BZHC today. What a wreck that plane is.
Flashing disco lights started while still on the ground. Rectified by shutting down the overhead electrics, allowing it to "cool down", and leaving the door open to the IFE equipment for the entire duration of the flight, to stop it overheating and going into a disco light frenzy.
All 3 bulkhead monitors are broken and will never be fixed. In front of 2AB and 2JK completely dead. In front of 1DF, almost pitch black, you might be able to make out something with night vision glasses. Noone in the front row can see the screens, so none of us were able to see the safety demo.
One of the ceiling mounted TVs (above rows 4/5 on LHS) broken. So, a total of 4 out of 7 monitors broken in the forward cabin.
No sound was working in the JK seats throughout the plane.
The bulkhead mounted pocket infront of 1D has almost completely ripped off the wall, leaving the magazines and sickbags dangling into the floor area.
No magazines loaded. No menus loaded.
The oven in the forward galley barely works, so it took an hour to heat the food. Imagine what chicken and rice tastes like after an hour in a warm oven. It was completely hard and came out in lumps, but had to be left in the oven this long to heat through. The outward breakfast was similarly baked for an hour in the mal-functioning oven, and the scrambled eggs came out in one vile lump.
Flashing disco lights started while still on the ground. Rectified by shutting down the overhead electrics, allowing it to "cool down", and leaving the door open to the IFE equipment for the entire duration of the flight, to stop it overheating and going into a disco light frenzy.
All 3 bulkhead monitors are broken and will never be fixed. In front of 2AB and 2JK completely dead. In front of 1DF, almost pitch black, you might be able to make out something with night vision glasses. Noone in the front row can see the screens, so none of us were able to see the safety demo.
One of the ceiling mounted TVs (above rows 4/5 on LHS) broken. So, a total of 4 out of 7 monitors broken in the forward cabin.
No sound was working in the JK seats throughout the plane.
The bulkhead mounted pocket infront of 1D has almost completely ripped off the wall, leaving the magazines and sickbags dangling into the floor area.
No magazines loaded. No menus loaded.
The oven in the forward galley barely works, so it took an hour to heat the food. Imagine what chicken and rice tastes like after an hour in a warm oven. It was completely hard and came out in lumps, but had to be left in the oven this long to heat through. The outward breakfast was similarly baked for an hour in the mal-functioning oven, and the scrambled eggs came out in one vile lump.
WHBM
You are indeed right, I am myself aware that regulation is not foolproof having grown up on the doorstep to LHR with numerous friends and family friends working at BA and predecessors, indeed a close relative worked in BA Maintrol . I rather deliberately put in an 'official ' way just for comparsion but either way none of this shows BA up ina good light.
As to retaining key people which someone else commented on I am sure WW may not know who the right people to keep are but he should have enough about him to know that if your IT is a rats nest of systems then you lose corporate expertise at your peril. Firstly because the state of the systems/sub systems is clearly not well designed or truly desirable and secondly because in these systems of making do and customised interfaces there is often not a lot written down since that kind of good housekeeping practice doesnt really sit well with a make do and mend strategy does it
You are indeed right, I am myself aware that regulation is not foolproof having grown up on the doorstep to LHR with numerous friends and family friends working at BA and predecessors, indeed a close relative worked in BA Maintrol . I rather deliberately put in an 'official ' way just for comparsion but either way none of this shows BA up ina good light.
As to retaining key people which someone else commented on I am sure WW may not know who the right people to keep are but he should have enough about him to know that if your IT is a rats nest of systems then you lose corporate expertise at your peril. Firstly because the state of the systems/sub systems is clearly not well designed or truly desirable and secondly because in these systems of making do and customised interfaces there is often not a lot written down since that kind of good housekeeping practice doesnt really sit well with a make do and mend strategy does it
Join Date: Jan 2010
Location: London
Posts: 379
Likes: 0
Received 0 Likes
on
0 Posts
'You talk about it as cost-cutting, I talk about it as efficiency': BA boss Willie Walsh defends airline over IT meltdown - and says criticism has been 'unfair'
BA boss Willie Walsh defends airline over IT meltdown | Daily Mail Online
BA boss Willie Walsh defends airline over IT meltdown | Daily Mail Online
Join Date: Feb 2000
Location: London, UK
Age: 64
Posts: 103
Likes: 0
Received 0 Likes
on
0 Posts
He just doesn't get it does he. Efficiency? An efficient airline doesn't close down for a day and a half because of a failure to put in place proper resilience.
Join Date: Mar 2015
Location: North by Northwest
Posts: 476
Likes: 0
Received 0 Likes
on
0 Posts
Still charging extra to get seat assignments?
Meanwhile, in conversation with a peer Friday, he related the story that his sister had booked a trip US-Scotland-England and return for July and paid for business class seats. Airline will not give her seat assignments on the outbound leg unless she pays an extra $300 per seat. I remarked that it had to be BA. It is. Still trying to make up cost of IT debacle? Have to wonder what they will charge when she tries to get home.
Paxing All Over The World
He just doesn't get it does he.
The five stages, denial, anger, bargaining, depression and acceptance are a part of the framework that makes up our learning to live with the one we lost. They are tools to help us frame and identify what we may be feeling. But they are not stops on some linear timeline in grief.
No, he doesn't get it because he has lived all his life with cost cutting and knows nothing else. Having to throw away all that you know and believe in AND that has made you 'top dog' and very rich - is almost impossible.
Another current example is Teresa May.