Go Back  PPRuNe Forums > Misc. Forums > Passengers & SLF (Self Loading Freight)
Reload this Page >

BA delays at LHR - Computer issue

Passengers & SLF (Self Loading Freight) If you are regularly a passenger on any airline then why not post your questions here?

BA delays at LHR - Computer issue

Old 7th Jun 2017, 18:33
  #581 (permalink)  
 
Join Date: Jan 2010
Location: London
Posts: 379
Originally Posted by etudiant View Post
If memory serves, the airline IT systems were cooperatively developed/shared among carriers, which might help explain the below normal percentage of revenue invested.
This is partly true as three of the big four global distribution systems (GDSs) - Worldspan, Galilieo and Amadeus - were set up by airlines. In contrast, Sabre was built by American Airlines and IBM. Then there is BA which had its own British Airways Booking System (BABS), a development of BOAC's booking system launched in the late 1960s, which ran up to 2002 until the switch to Amadeus.

So, some airlines gained from shared development and running costs, but it looks unlikely to me that this would account for the IT underspend compared with other industries.
Caribbean Boy is offline  
Old 7th Jun 2017, 18:42
  #582 (permalink)  
 
Join Date: May 2009
Location: Confoederatio Helvetica
Age: 65
Posts: 2,846
Revenues is a bad measurement in an industry that alost always failes to achieve a 5% ROI. Airlines make a few dollars per passenger, in the good years.
ExXB is offline  
Old 8th Jun 2017, 03:02
  #583 (permalink)  
 
Join Date: Mar 2014
Location: Rainsville
Posts: 46
If that is a low percentage it is probably because airlines have such massive big budget items on this like airframe depreciation and fuel that perhaps it pushes the IT percentage down compared to say a bank which spends relatively more in percentage terms on IT because it doesn't have to buy things like A 380s or spend a fortune on fuel.
Banks spend a fortune on salaries and bonuses — around 40% of revenue. Yet they still spend 10% on IT.

The difference is probably that banks face a competitive threat from tech startups. In a cashless economy, deposit-taking banks are potentially redundant (and who would mourn them?), but until one of us invents a teleportation device we'll probably still need physical aircraft.
oblivia is offline  
Old 8th Jun 2017, 05:23
  #584 (permalink)  
 
Join Date: Sep 2009
Location: Haifa
Age: 65
Posts: 18
I remember a lightning strike that killed the UPS and cutover/bypass switch for a large data center. The result was an inability to restart anything until a new switch was installed. Then we discovered that about 20% of disks had failed - they had been spinning for years and many were damaged by the sudden power cut.

This was mid 90's - probably the same technology BA use now.
sabbasolo is offline  
Old 8th Jun 2017, 07:33
  #585 (permalink)  
 
Join Date: Aug 2005
Location: EDLB
Posts: 195
The cost of this single failure exceeds the cost of a same sized brand new data center by a factor between 5 and 10. That tells all about the risk assessment quality of BA IT.
EDLB is offline  
Old 8th Jun 2017, 09:47
  #586 (permalink)  
 
Join Date: Jan 2016
Location: Uk
Posts: 62
to be fair, even if a data centre was wiped out by a power outage, and even if the power came back on haphazardly, and even if a number of devices physically failed due to this - a company of that size dependent on systems to run it's core business should be able to continue. It sounds like this BA have more that one data centre but for whatever reason failing over to it clearly didn't work as planned.

It doesn't matter how many data centres you have, if you can't reliably fail over then you may as well just have one. This smells like a failing in application/system design and testing.

However don't underestimate the complexity of designing systems this way that integrate with both legacy and external systems - it is very difficult to do, to test and is expensive. Every penny I spend on disaster planning is money that ultimately I hope to be wasted - but just sometimes, it is not and it pays itself back 100 fold

Last edited by Snyggapa; 8th Jun 2017 at 09:47. Reason: typo
Snyggapa is offline  
Old 8th Jun 2017, 10:46
  #587 (permalink)  
 
Join Date: Aug 2007
Location: uk
Age: 69
Posts: 82
Can I offer a perspective. I was around a company 12 years ago where someone did a Chernobyl experiment by disconnecting the UPS and trying something. This tripped the circuit breakers and killed the SAN (storage area network) and about 500 servers. Then the company discovered the failover data centre was incorrectly sized and went down as well. The SAN subsequently refused to restart and then the Japanese manufacturer informed the company that Sorry but the SAN was no longer supported as it was an old version. The company paid a lot of money to fly somebody from Japan and it eventually was restarted after a few days. But it cost the company £1m in penalty clause payments to their system users. There were a number of banks and Health systems involved in that but was kept quiet for obvious reasons.
sunnybunny is offline  
Old 8th Jun 2017, 13:48
  #588 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,348
This is the way it always happens.
The actual failure (finger trouble) should have had no effect. BUT
The failover data centre was too small, possibly from inception or because nobody had thought to check the growth against backup. This is an IT management failure of testing and continual monitoring.
The SAN being out of date is again an IT management failure as the continued maintenance and support of special equipment that supports the business should be part of normal IT management checks.
The other management failure shown by both of these is the failure to set flags that 'when the system gets this big' or 'when the system gets this old' a new system will be needed. Those flags should have been set on acceptance of the system and a draft contingency plan for how to replace the system should have been written and maintained.
With a cost cutting management it can prove very difficult to persuade them that the IT system which is still humming along, will need replacing in 18 months time and as the draft replacement plan says it will take 15 months to replace work needs to be paid for now to initiate contracts for replacement.
Ian W is offline  
Old 8th Jun 2017, 16:02
  #589 (permalink)  
 
Join Date: Jan 2010
Location: London
Posts: 379
Why would the contracts need to be placed so early? If anything, I would have thought that the biggest delay would be in getting budget approval past the Finance committee.
Caribbean Boy is offline  
Old 8th Jun 2017, 20:18
  #590 (permalink)  
 
Join Date: Oct 2002
Location: London UK
Posts: 6,434
Originally Posted by EDLB View Post
The cost of this single failure exceeds the cost of a same sized brand new data center by a factor between 5 and 10. That tells all about the risk assessment quality of BA IT.
I understand there isn't any "BA IT" any more - they outsourced it to the bottom bidder.

Was the outsourcing Cruz's idea ?
WHBM is offline  
Old 9th Jun 2017, 09:34
  #591 (permalink)  
 
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,348
Originally Posted by Caribbean Boy View Post
Why would the contracts need to be placed so early? If anything, I would have thought that the biggest delay would be in getting budget approval past the Finance committee.
If you know that your system
1. has just passed one of the indicators that it needs replacement that were declared at its installation. With say system hardware or software becoming unsupported in 2 years or in 2 years after a merger you will need to add a subsidiary airline to the system taking it above its design limits..
2. Installation and acceptance and operational roll out will take 15 months
3. Getting 'finance committee' or board approval will take at least 3 months
4. Contract creation, bid process, selection and formal contract let will take at least 3 months

Then you will have to proceed immediately initiating the replacement project for the system. This is when the biggest battle will be with the bean counters and the CFO and board. It is also where the poorest decisions are made.

Putting head of IT as reporting to the CFO is almost certain to lead to attempts to continue with the old system that the head of IT knows will fail. Head of IT has the choice of making noise to do the right thing that will often result in a pink slip along with the introduction of the outsourcing group who also know the system will fail, but know if it does they control the company as the outsourcing group are now the only people who can provide a solution at their own inflated cost.
Ian W is offline  
Old 9th Jun 2017, 10:03
  #592 (permalink)  
 
Join Date: Nov 2015
Location: BRS/GVA
Posts: 329
Have a read of this https://www.theregister.co.uk/2017/0...path_analysis/
analysis by a leading IT industry news site
hoss183 is offline  
Old 9th Jun 2017, 16:54
  #593 (permalink)  
 
Join Date: Aug 2001
Location: se england
Posts: 1,201
[Hoss
Thanks for that link to a very objective and analytical piece. I am sure that in BA thre is a need for various systems to interact. For example the check in process isnt confined to that action since the operational side of the airline needs to know the pax and baggage weights for fuel and also performance calculations and wight and balance. That could mean there is a need to connect between different systems but as the author points out hats ok if it is between two or four systems but dozens!! It is almost guaranteeign disaster when somethign goes wrong.

Some organsiations deal with this kind of complex interworking by ensuring their key peopel are retained because they built the systems and the interfaces and can fix them. very hard to outsource something very complex and customised . General accounting and sales functions maybe yes since theyare common to many companies. Airline IT systems are in a different elague.

The other point is that made continually on hear that a cost cutting management does not listen to 'this equipment is time expired it has to be replaced. Obviously in an airline aircraft and engines have that kind of condition attached to them but the management don't really have any choice because those issues are regulated but with things like IT systems and UPS there is huge temptation to defer replacements because anew UPS for Cranebank doesnt lower seat mile costs and doesnt produce any more bookings and so its only and immediate effect is on the cost side of the equation, that is of ciurse untiil the wheels come off like in this case.

Management platitudes are common to all companies and not just airlines but airlines are not immune. For abit of amusement I wrote to Easyjet a few years ago and saying that as they had introduced seat reservations Iwas tempted to try them but as they were low cost could I be sure they were safe. Of course I got a lovely PR answer back that safety was their first priority etc etc etc . I wrote back, after a little reserach on the company and said @Yes, i have heard that but if safety is your fist priority why isnt it a responsibility for any member of your board of directors who all seemed to have a fiance or marketing background.

And its the same here no one high enough up who can stand up to the Aer Iberia duo and say , you wont like this boss but our IT system is very very vulnerable and something needs to be done because they know the answer is a P45
pax britanica is offline  
Old 9th Jun 2017, 17:40
  #594 (permalink)  
 
Join Date: May 2011
Location: NEW YORK
Posts: 666
How on earth is WW going to determine who are the 'key people' that he needs to retain?
The system is in continual flux, as new services and new hardware are deployed, so the architecture may have changed as well. I don't think keeping people would have helped avert this disaster, rather BA needs to have some sort of fallback for when it all goes bad.
etudiant is online now  
Old 10th Jun 2017, 00:41
  #595 (permalink)  
 
Join Date: Oct 2002
Location: London UK
Posts: 6,434
Originally Posted by pax britanica View Post

The other point is that made continually on hear that a cost cutting management does not listen to 'this equipment is time expired it has to be replaced. Obviously in an airline aircraft and engines have that kind of condition attached to them but the management don't really have any choice because those issues are regulated
This is a general belief, but within the industry there are those who know differently.

A few years ago the internal state of the BA "European" 767 fleet, at a time when they were still employed on the Moscow route, had got into a very dishevelled state, principally due to engineering having to conform to tight budgets. Over on Flyertalk the Moscow regulars, generally seniors from the banking and oil industries, started to compile a Tech Log, by airframe registration (it was quite a small fleet), listing all the recurrent defects. One aircraft G-BZHC, was known to all as "the Disco" because of defective flashing lights. A number of the issues were indeed safety-related.

Came back on G-BZHC today. What a wreck that plane is.

Flashing disco lights started while still on the ground. Rectified by shutting down the overhead electrics, allowing it to "cool down", and leaving the door open to the IFE equipment for the entire duration of the flight, to stop it overheating and going into a disco light frenzy.

All 3 bulkhead monitors are broken and will never be fixed. In front of 2AB and 2JK completely dead. In front of 1DF, almost pitch black, you might be able to make out something with night vision glasses. Noone in the front row can see the screens, so none of us were able to see the safety demo.

One of the ceiling mounted TVs (above rows 4/5 on LHS) broken. So, a total of 4 out of 7 monitors broken in the forward cabin.

No sound was working in the JK seats throughout the plane.

The bulkhead mounted pocket infront of 1D has almost completely ripped off the wall, leaving the magazines and sickbags dangling into the floor area.

No magazines loaded. No menus loaded.

The oven in the forward galley barely works, so it took an hour to heat the food. Imagine what chicken and rice tastes like after an hour in a warm oven. It was completely hard and came out in lumps, but had to be left in the oven this long to heat through. The outward breakfast was similarly baked for an hour in the mal-functioning oven, and the scrambled eggs came out in one vile lump.
I understand that the thread came to the attention of a BA non-exec, and thus to the main board meeting. There was some action thereafter.
WHBM is offline  
Old 10th Jun 2017, 09:40
  #596 (permalink)  
 
Join Date: Aug 2001
Location: se england
Posts: 1,201
WHBM

You are indeed right, I am myself aware that regulation is not foolproof having grown up on the doorstep to LHR with numerous friends and family friends working at BA and predecessors, indeed a close relative worked in BA Maintrol . I rather deliberately put in an 'official ' way just for comparsion but either way none of this shows BA up ina good light.

As to retaining key people which someone else commented on I am sure WW may not know who the right people to keep are but he should have enough about him to know that if your IT is a rats nest of systems then you lose corporate expertise at your peril. Firstly because the state of the systems/sub systems is clearly not well designed or truly desirable and secondly because in these systems of making do and customised interfaces there is often not a lot written down since that kind of good housekeeping practice doesnt really sit well with a make do and mend strategy does it
pax britanica is offline  
Old 10th Jun 2017, 13:37
  #597 (permalink)  
 
Join Date: Jan 2010
Location: London
Posts: 379
'You talk about it as cost-cutting, I talk about it as efficiency': BA boss Willie Walsh defends airline over IT meltdown - and says criticism has been 'unfair'

BA boss Willie Walsh defends airline over IT meltdown | Daily Mail Online
Caribbean Boy is offline  
Old 11th Jun 2017, 10:02
  #598 (permalink)  
 
Join Date: Feb 2000
Location: London, UK
Age: 61
Posts: 100
He just doesn't get it does he. Efficiency? An efficient airline doesn't close down for a day and a half because of a failure to put in place proper resilience.
ELondonPax is offline  
Old 11th Jun 2017, 11:48
  #599 (permalink)  
 
Join Date: Mar 2015
Location: North by Northwest
Posts: 473
Still charging extra to get seat assignments?

Meanwhile, in conversation with a peer Friday, he related the story that his sister had booked a trip US-Scotland-England and return for July and paid for business class seats. Airline will not give her seat assignments on the outbound leg unless she pays an extra $300 per seat. I remarked that it had to be BA. It is. Still trying to make up cost of IT debacle? Have to wonder what they will charge when she tries to get home.
b1lanc is offline  
Old 11th Jun 2017, 12:01
  #600 (permalink)  
Paxing All Over The World
 
Join Date: May 2001
Location: Hertfordshire, UK.
Age: 63
Posts: 9,141
He just doesn't get it does he.
He certainly does not - and has not from the moment the meltdown started. I was reminded of the Five Stages of Grief that were first proposed by Elisabeth Kübler-Ross in her 1969 book On Death and Dying.
The five stages, denial, anger, bargaining, depression and acceptance are a part of the framework that makes up our learning to live with the one we lost. They are tools to help us frame and identify what we may be feeling. But they are not stops on some linear timeline in grief.
She stated that not everyone will visit these points in this sequence, some points are visited more than once but, before the story is complete, they will have visited them all. In subsequent years, these steps have been identified with divorce, losing your job and (simply) LOSS.

No, he doesn't get it because he has lived all his life with cost cutting and knows nothing else. Having to throw away all that you know and believe in AND that has made you 'top dog' and very rich - is almost impossible.

Another current example is Teresa May.
PAXboy is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service - Do Not Sell My Personal Information

Copyright © 2018 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.