BA delays at LHR - Computer issue
Join Date: Jan 2007
Location: Queensland, Australia
Age: 71
Posts: 105
Likes: 0
Received 0 Likes
on
0 Posts
I know tht SLF are only tolerated here and when admit that, before retirement, I was in charge of engineering for a TV News Agency I suspect there will be muttering under a lot of breaths.
However, because we had to broadcast live a lot of the time, we put in an elaborate back up power system. All critical facilities (in our case, studios, control rooms, edit rooms, computers etc.) ran on a UPS that could keep the whole load going for 30+ minutes. In addition, there were two auto-start generators. Either of these generated enough power to keep all essential services running. If both came on, we would power the whole building--offices and so on--but if one failed for any reason or another, there was automatic load shedding to turn off power to non essentials and keep the important parts going.
On top of that--but very important--we tested the system once a month. Technical management took turns coming in at around midnight, throwing the main power supply off and letting the generators take the load for half an hour or so. The diesel engines running the generators were inspected the next morming.
On top of all that, even before we got to UPS units and generators, we had two separate power supplies into the building and multiple different routes to feed data (and in our case video) out of the building.
It may have been expensive but the accountants were eventually convinced that it was more economical than going off the air.
So it CAN be done.
However, because we had to broadcast live a lot of the time, we put in an elaborate back up power system. All critical facilities (in our case, studios, control rooms, edit rooms, computers etc.) ran on a UPS that could keep the whole load going for 30+ minutes. In addition, there were two auto-start generators. Either of these generated enough power to keep all essential services running. If both came on, we would power the whole building--offices and so on--but if one failed for any reason or another, there was automatic load shedding to turn off power to non essentials and keep the important parts going.
On top of that--but very important--we tested the system once a month. Technical management took turns coming in at around midnight, throwing the main power supply off and letting the generators take the load for half an hour or so. The diesel engines running the generators were inspected the next morming.
On top of all that, even before we got to UPS units and generators, we had two separate power supplies into the building and multiple different routes to feed data (and in our case video) out of the building.
It may have been expensive but the accountants were eventually convinced that it was more economical than going off the air.
So it CAN be done.
Join Date: Sep 2011
Location: Around the place
Age: 43
Posts: 9
Likes: 0
Received 0 Likes
on
0 Posts
Would not be surprised in the slightest if the 'power supply problem' was actually a botched upgrade. It's very easy for the travelling public to understand 'The power cord fell out it stopped working', rather than 'We were updating our database and Little Jonny Tables popped up'.
I rather like todays media comments that BA have advised passengers affected they will get a refund or rebooking . Well of course they will since they sold people tickets that turned out to be unusable and are in breach of contract with a couple of hundred thousand people. How arrogant can they get.
This is an extremely serious incident, if it was JAL or ANA the CEo would be packing his office up this morning but here in the land of 'accountability' not responsibility the blame will fall to the lowest possible credible manager or technician.
It is unthinkable that a power problem at just one site could shut everything down they must surely have a back up or mirror site else where in Uk or even in the US somewhere that would provide some assurance of continuity, aside from everything else it just makes them look stupid as a company and by name association us look stupid as a country
This is an extremely serious incident, if it was JAL or ANA the CEo would be packing his office up this morning but here in the land of 'accountability' not responsibility the blame will fall to the lowest possible credible manager or technician.
It is unthinkable that a power problem at just one site could shut everything down they must surely have a back up or mirror site else where in Uk or even in the US somewhere that would provide some assurance of continuity, aside from everything else it just makes them look stupid as a company and by name association us look stupid as a country
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes
on
0 Posts
Interesting scenario and one that BA or Heathrow management have not learned from : after all the issues with snow and previously high winds: aircraft waiting after landing to offload maybe up to 4 hours ?? incredible. As a result of previous instances and before I left LHR i drafted a contingency OSI for passenger offloading onto taxiways therefore enabling pax to get in the terminal. Aircraft would then taxi to base to park which was always full of available space. This was available for all terminals not just T5 with specific locations specific processes to achieve. During the snow debacle we and offloaded 10 aircraft in 90 minutes that had been waiting for up to 4 hours because of no available stands. This resulted in the draft process being drawn up with ATC and BA agreeing. looks like its now in the bin somewhere !! This is when experience counts with ops guys who can think on their feet and do not have to rely on a black or white procedure!!!!!!
Join Date: Dec 2006
Location: Florida and wherever my laptop is
Posts: 1,350
Likes: 0
Received 0 Likes
on
0 Posts
<<SNIP>>
My fear always is that a single system failure might not be restricted or contained when it is a logical or intrinsic programmer error which with the cold logic of object code propagates through the redundant systems also. The problem in your primary hydraulic system is not actually isolated because the same problem which led to its failure exists on the fallback.
My fear always is that a single system failure might not be restricted or contained when it is a logical or intrinsic programmer error which with the cold logic of object code propagates through the redundant systems also. The problem in your primary hydraulic system is not actually isolated because the same problem which led to its failure exists on the fallback.
Join Date: Jul 2014
Location: kent
Age: 55
Posts: 6
Likes: 0
Received 0 Likes
on
0 Posts
As both SLF (derogatory as that title is) and a highly experienced IT leader (biased towards Infrastructure) and someone who spent nearly 10 hours yesterday in T5 , I feel I have something to contribute.
First off, let's not confuse DR with BCP, although both failed yesterday.
For example while IT wherever were toiling over bringing systems back online , the CW/CE/First queues, that were right out of the terminal, were being "organised" by 2 women who were effectively herding cats. They were on a hiding to nothing as people were joining any one and then losing it when the staff come back round again 20 minutes later telling them to go and join the mega queue at WT. Not enough staff and definitely no sign of Managment at all. This got better during the afternoon, but still no sign of any Senior Staff at all. Even this morning they were trying to get us on a flight as my wife received a text to say it was cancelled but nothing showed on their system. Where were the managers, nowhere to be seen, as they "were in meetings". Maybe those meetings should have been through the night so everyone could be briefed for 0430.
On a more serious note we were told by staff they couldn't find any megaphones to replace the non working PA. I would suggest that these should be easy to find in case of a real emergency.
As for IT, outsourcing is not something I would advocate, but when it has crossed my path, I would never allow a system to go live without:
Rigorous functional testing of system
Rigorous DR Testing
Sign off of all infrastructure designs from someone qualified to do so and counter sign it myself.
The outsourcer should not have unrestricted responsibility for design of something thousands of miles away that isn't theirs. This also makes it easy to swap supplier should they prove to be sub par, which they will.
I guarantee someone within BA has signed that design off as suitable, and that's where heads should roll initially. Then look at your "partner"
Also all the previous posts regarding bean counters are a given as well. Scourge of IT !
On a personal note I'm not actually buying the power excuse but as we don't like to speculate within these halls I'll keep my opinion to myself. I will say however all the systems affected were internet facing.
Anyway, got all that off my chest, and resigned to go back to work on Tuesday instead of enjoying a few cold ones on the Greek coastline !
First off, let's not confuse DR with BCP, although both failed yesterday.
For example while IT wherever were toiling over bringing systems back online , the CW/CE/First queues, that were right out of the terminal, were being "organised" by 2 women who were effectively herding cats. They were on a hiding to nothing as people were joining any one and then losing it when the staff come back round again 20 minutes later telling them to go and join the mega queue at WT. Not enough staff and definitely no sign of Managment at all. This got better during the afternoon, but still no sign of any Senior Staff at all. Even this morning they were trying to get us on a flight as my wife received a text to say it was cancelled but nothing showed on their system. Where were the managers, nowhere to be seen, as they "were in meetings". Maybe those meetings should have been through the night so everyone could be briefed for 0430.
On a more serious note we were told by staff they couldn't find any megaphones to replace the non working PA. I would suggest that these should be easy to find in case of a real emergency.
As for IT, outsourcing is not something I would advocate, but when it has crossed my path, I would never allow a system to go live without:
Rigorous functional testing of system
Rigorous DR Testing
Sign off of all infrastructure designs from someone qualified to do so and counter sign it myself.
The outsourcer should not have unrestricted responsibility for design of something thousands of miles away that isn't theirs. This also makes it easy to swap supplier should they prove to be sub par, which they will.
I guarantee someone within BA has signed that design off as suitable, and that's where heads should roll initially. Then look at your "partner"
Also all the previous posts regarding bean counters are a given as well. Scourge of IT !
On a personal note I'm not actually buying the power excuse but as we don't like to speculate within these halls I'll keep my opinion to myself. I will say however all the systems affected were internet facing.
Anyway, got all that off my chest, and resigned to go back to work on Tuesday instead of enjoying a few cold ones on the Greek coastline !
Join Date: Jul 2007
Location: Hadlow
Age: 60
Posts: 597
Likes: 0
Received 0 Likes
on
0 Posts
Join Date: Sep 2011
Location: Around the place
Age: 43
Posts: 9
Likes: 0
Received 0 Likes
on
0 Posts
Actually, a few viruses exist for the C64:
Com64/BHP | VirusInfo | Fandom powered by Wikia
Com64/BHP | VirusInfo | Fandom powered by Wikia
Join Date: Dec 2013
Location: Aberdeen
Posts: 49
Likes: 0
Received 0 Likes
on
0 Posts
Would not be surprised in the slightest if the 'power supply problem' was actually a botched upgrade. It's very easy for the travelling public to understand 'The power cord fell out it stopped working', rather than 'We were updating our database and Little Jonny Tables popped up'.
Was Mr O' Bama flying BA from Edinburgh ? Damned apostrophes Eh?
None but a blockhead
Join Date: Nov 1999
Location: London, UK
Posts: 535
Likes: 0
Received 0 Likes
on
0 Posts
I'd hope nobody would be rolling out upgrades on one of the busiest weekends of the year.
For what it's worth, and having seen and written about a few major IT omnishambles, I'd be surprised if this omelette wasn't made with swiss cheese. And that the motivation to find and fix the systemic error which allowed the holes to line up won't be sufficient to counter organisational inertia and CYA - but if it is, I'll regain that very scarce state of mind, respect for senior management.
For what it's worth, and having seen and written about a few major IT omnishambles, I'd be surprised if this omelette wasn't made with swiss cheese. And that the motivation to find and fix the systemic error which allowed the holes to line up won't be sufficient to counter organisational inertia and CYA - but if it is, I'll regain that very scarce state of mind, respect for senior management.
Marks and Spencer sandwiches?
I empathise with the frontline staff, and I hope that the passengers remember that it is not the fault of these staff that their journey has been delayed.
I would imagine that the prospect of huge Eu261 claims is probably driving the lack of disclosure as to the exact reason for this debacle.
Free the gates!
I'm somewhat surprised that they are leaving a/c on gates. blocking other flights ... would make more sense to close 5B/5C and start stacking them on the taxiways C/D by now, blocking gates is not the best plan.
Isn't it more what has Alex Cruz taken off the table since he has been CEO, no more meals , cheapskate policy on water for pax etc.
This fiasco is his fault, he is the CEO -he gets the plaudits but should also carry the can He is a slash and burn manager and his philosophy and aggressive cost cutting has obviously been taken taken a step too far here and he has to go for the good of BA
Anyone else agree
This fiasco is his fault, he is the CEO -he gets the plaudits but should also carry the can He is a slash and burn manager and his philosophy and aggressive cost cutting has obviously been taken taken a step too far here and he has to go for the good of BA
Anyone else agree
Join Date: Dec 2016
Location: Yorkshire
Posts: 4
Likes: 0
Received 0 Likes
on
0 Posts
DingerX is spot on, saying "The person who 'saved the airline a fortune in IT costs' has now no doubt been promoted, hired away to a different company and enjoyed a couple more pay raises". Happens right across public sector services too
Join Date: Aug 2002
Location: Europe
Posts: 4
Likes: 0
Received 0 Likes
on
0 Posts
I agree
Isn't it more what has Alex Cruz taken off the table since he has been CEO, no more meals , cheapskate policy on water for pax etc.
This fiasco is his fault, he is the CEO -he gets the plaudits but should also carry the can He is a slash and burn manager and his philosophy and aggressive cost cutting has obviously been taken taken a step too far here and he has to go for the good of BA
Anyone else agree
This fiasco is his fault, he is the CEO -he gets the plaudits but should also carry the can He is a slash and burn manager and his philosophy and aggressive cost cutting has obviously been taken taken a step too far here and he has to go for the good of BA
Anyone else agree