Go Back  PPRuNe Forums > Flight Deck Forums > Rumours & News
Reload this Page >

Complete LH group meltdown

Rumours & News Reporting Points that may affect our jobs or lives as professional pilots. Also, items that may be of interest to professional pilots.

Complete LH group meltdown

Old 15th Feb 2023, 10:47
  #1 (permalink)  
Thread Starter
 
Join Date: Jun 2002
Location: Geneva, Switzerland
Age: 57
Posts: 1,854
Received 2 Likes on 2 Posts
Complete LH group meltdown

"Deutsche Lufthansa AG grounded all of its flights on Wednesday after damage to a set of Deutsche Telekom broadband cables caused widespread IT problems" (link - behind paywall),

If true (ie the root cause, the disruption is very real) it is inexcusable.
atakacs is online now  
Old 15th Feb 2023, 10:54
  #2 (permalink)  
 
Join Date: Feb 2010
Posts: 929
Received 44 Likes on 27 Posts
Pretty bad mess at FRA right now. Flights get diverted elsewhere as the airport is topped off with aircraft after nothing is departing. Mind you, this Friday there are big airport strikes in Germany anyway.
Less Hair is online now  
Old 15th Feb 2023, 12:01
  #3 (permalink)  
 
Join Date: Aug 2005
Location: EDLB
Posts: 292
Received 1 Like on 1 Post
Workers managed to drill into a big fiber trunk 5m under ground near a railroad. Repair ongoing since Tuesday night.
EDLB is offline  
Old 15th Feb 2023, 13:27
  #4 (permalink)  
 
Join Date: Nov 2000
Location: Pewsey, UK
Posts: 1,974
Received 3 Likes on 2 Posts
https://www.theregister.com/2023/02/15/lufthansa/ - non paywalled.
The Nr Fairy is offline  
Old 15th Feb 2023, 14:13
  #5 (permalink)  
Thread Starter
 
Join Date: Jun 2002
Location: Geneva, Switzerland
Age: 57
Posts: 1,854
Received 2 Likes on 2 Posts
Originally Posted by EDLB
Workers managed to drill into a big fiber trunk 5m under ground near a railroad. Repair ongoing since Tuesday night.
Might well be but this type of infra should most definitely be totally redundant - yes it does cost money but this incident will not be cheap either. Even the Frankfurt airport website is MIA...
atakacs is online now  
Old 15th Feb 2023, 15:02
  #6 (permalink)  
 
Join Date: Jul 2003
Location: Manchester
Posts: 881
Likes: 0
Received 0 Likes on 0 Posts
Single point of failure, oh dear !
MAN777 is offline  
Old 15th Feb 2023, 17:30
  #7 (permalink)  
 
Join Date: Aug 2007
Location: england
Posts: 825
Received 0 Likes on 0 Posts
Sounds very British Airways !
hunterboy is offline  
Old 15th Feb 2023, 18:57
  #8 (permalink)  
 
Join Date: Aug 2001
Location: se england
Posts: 1,513
Likes: 0
Received 27 Likes on 11 Posts
As a telecoms guy It is almost beyond belief that all LHs IT could be taken out by one cable cut and on a rail track right of way where it is well known there are always going to be telecom cables ..

Its the airline industry for gods sake redundancy is built into every aspecyt of airliners but it seems not in IT systems.

The CEO of the LH Group should be fired for this
pax britanica is offline  
Old 15th Feb 2023, 19:36
  #9 (permalink)  
 
Join Date: Aug 2007
Location: not where I want to be
Posts: 491
Received 29 Likes on 21 Posts
Originally Posted by pax britanica
... The CEO of the LH Group should be fired for this
While I agree redundancy in such a system is surely a good thing I really don't see why the CEO should be automatically fired as a result of this event?

A person in that position would not normally be expected to know the full down 'n dirty technical minutiae of their system, rather they would rely upon advice they receive from company and external experts. If it transpired they knew about weaknesses in the system, were advised to address these but did nothing, then perhaps they should consider moving on. Otherwise, assuming they're an effective CEO, why not allow them the opportunity to sort the issue out?

To put it another way; should a pilot be automatically 'fired' because an engineer did something that broke their 'plane?

FP.
First_Principal is offline  
Old 15th Feb 2023, 20:18
  #10 (permalink)  
 
Join Date: Nov 2018
Location: back out to Grasse
Posts: 557
Received 28 Likes on 12 Posts
The Piper will have to be paid.

No doubt a serious cost saving measure that turned around and bit them. Single point of failure cannot be tolerated in a failover strategy. Chief Information Officers and Financial Officers would not have adopted this strategy without the full knowledge of the CEO. There will be no hiding place.

IG
Imagegear is offline  
Old 15th Feb 2023, 20:39
  #11 (permalink)  
 
Join Date: Mar 2004
Location: schermoney and left front seat
Age: 57
Posts: 2,427
Received 0 Likes on 0 Posts
Originally Posted by Imagegear
The Piper will have to be paid.

No doubt a serious cost saving measure that turned around and bit them. Single point of failure cannot be tolerated in a failover strategy. Chief Information Officers and Financial Officers would not have adopted this strategy without the full knowledge of the CEO. There will be no hiding place.

IG
My BS detector is right off the scale - do you HONESTLY think, Lufthansa should run their own internet ? Parallel to the infrastructure already installed ? Its not that someone drilled into a cable at the LH HQ.

There is plenty of things I would fire the CEO for (e.g. closing LH Flighttraining down or the downward spiral of T&C for LH Group employees), but this is not one of them.

What this really highlights, is how the (EU driven) shift to privatization of crucial infrastructure allows for these weak points to develop.
His dudeness is offline  
Old 15th Feb 2023, 20:42
  #12 (permalink)  
 
Join Date: Dec 2021
Location: Uk
Posts: 166
Likes: 0
Received 9 Likes on 6 Posts
Originally Posted by Imagegear
The Piper will have to be paid.

No doubt a serious cost saving measure that turned around and bit them. Single point of failure cannot be tolerated in a failover strategy. Chief Information Officers and Financial Officers would not have adopted this strategy without the full knowledge of the CEO. There will be no hiding place.

IG
Adopting a strategy, without approval from the CEO may be one thing. A system in place for years before the CEO took the role? I would cut some slack. I know there is a propensity to think ever CEO of a large corporation is the devil incarnate right now but they can only know what they have been briefed on. To enable them to make a decision. Even then they arenít briefed on every aspect of the organisation.
Flyhighfirst is offline  
Old 16th Feb 2023, 03:04
  #13 (permalink)  
 
Join Date: Aug 2004
Location: US
Posts: 492
Received 4 Likes on 2 Posts
It happened at Newark

Quite a while ago a pile driving crew was driving piles for a new garage at Newark.
They hit a concrete conduit that was not supposed to be there. It was a glancing blow. So they moved over and tried again,.
This time they penetrated the conduit and hit the main power line to various parts of the airport , including the control tower and Customs in Terminal B.
For added measure the back up power line was in the same conduit bank and it was hit as well. !
Nothing moved for about 6 hours and all the inbound international arrivals had to wait for customs to get power back.
Seems the control tower had a back up gen set so they were OK.
20driver is offline  
Old 16th Feb 2023, 04:52
  #14 (permalink)  
 
Join Date: Aug 2005
Location: EDLB
Posts: 292
Received 1 Like on 1 Post
Idiot proofing infrastructure is not easy and expensive. Mainly due to the inventiveness of the idiots.
EDLB is offline  
Old 16th Feb 2023, 05:29
  #15 (permalink)  
 
Join Date: Mar 2001
Location: I wouldn't know.
Posts: 4,489
Likes: 0
Received 0 Likes on 0 Posts
Don’t big companies usually have a CIO (Chief IT Officer) who can be fired for those cases? That said, Lufty probably just paid a lot of money to Telekom to get a redundant fibre connection and Telekom simply put all four fibers in the same conduit below that railway line.

The bigger problem is probably that nobody knows whats down there until you dig into it, since records are only kept within one company. In Berlin they just built the foundation for a new high rise building and afterwards discovered that they nearly destroyed the tunnel of an active underground line which now has to be repaired, stopping traffic in there for months, which is still better than nearly burying a whole commuter train in concrete.
Denti is offline  
Old 16th Feb 2023, 07:35
  #16 (permalink)  
 
Join Date: Nov 2007
Location: malta
Posts: 196
Likes: 0
Received 0 Likes on 0 Posts
I am (constantly) amazed by the calls for firing people without any further information on cause or processes.

Just cause should be applied everywhere, not just in the cockpit.

Somebody made a mistake, let them figure it out, learn from it and than see if somebody is to blame and in what measures.
the_stranger is offline  
Old 16th Feb 2023, 08:42
  #17 (permalink)  
 
Join Date: Jun 2022
Location: Seattle
Posts: 93
Likes: 0
Received 0 Likes on 0 Posts
Replace CEO in all contexts with Captain. Reads a bit differently right?
Boeingdriver999 is offline  
Old 16th Feb 2023, 09:03
  #18 (permalink)  
Pegase Driver
 
Join Date: May 1997
Location: Europe
Age: 73
Posts: 3,492
Likes: 0
Received 0 Likes on 0 Posts
Exactly the same thing occurred 45 yaers ago in my old ( major) Control centre. A worker using a backhoe dig and cut a bunch of cables just outside the centre , severing all telephone and radar lines ( that were all underground at the time ) Lessons were larned to have multiple access, etc.. but one of the root cause still rermains today : sub contracting basic work to the lowest bidder. In our case the guy operating backhoe was not one the regular workers who had a briefing on the cables layout , but the digging crew that morning were hired on the day to do some basic earth removal and they were not made aware of the exact position and sensivity of the cables.
I would not be surprised if the same happenned in Frankfurt yesterday with Deuche Bahn ( the railway company that was contracting the work that cut the cable), How many sub contractors down the line ?
As to the resposnibily of Lufhansa, of course not their direct fault as the incident occured quite far away from the airport, but relying on a single source of communications to function, probably is.
ATC Watcher is offline  
Old 16th Feb 2023, 09:31
  #19 (permalink)  
Tabs please !
 
Join Date: Jun 2004
Location: Biffins Bridge
Posts: 845
Received 57 Likes on 15 Posts
Originally Posted by His dudeness
What this really highlights, is how the (EU driven) shift to privatization of crucial infrastructure allows for these weak points to develop.
I beg to differ. The most robust network I have been involved with was a totally private company. At one point, it carried the BBC broadcasting distribution network so it was a high stakes venture. In 1994, a 737 crashed short of Coventry airport and went straight through the network at Willenhall. I was in the ops centre at the time and the screens lit up like a christmas tree. The BBC transmissions didn't even flicker as the network was fully redundant. All other services including 999 were similarly unaffected.

Separacy and diversity of networks is critical. In a development at locations such as an airport, the incremental costs are negligible compared to the cost of failure. You can plan as much as you like but Murphy is always waiting in the wings. One memorable event involved a Big Telecom company who excavated a duct under a rugby pitch using a horizontal borer. The machine was programmed to dig deeper than other infrastructure which was marked accurately on the diagrams. Nobody noticed that to improve drainage, the height of the pitch had been increased. Ouch ! Another event in Glasgow city centre involved a water main. The excavations did not touch the pipe however it had corroded to the point where the weight of the earth above it was holding it together. The dig reduced the weight and the whole pipe let go in a spectacular fashion.
B Fraser is online now  
Old 16th Feb 2023, 12:07
  #20 (permalink)  
 
Join Date: Aug 2001
Location: se england
Posts: 1,513
Likes: 0
Received 27 Likes on 11 Posts
Most companies of LH size work extremely hard to ensure their networks have adequate diversity.

This often means literlaly going out and walking the paths of critical cables to ensure they reallyare seeprated when they enter your properties and not two cables in one duct.. All kinds of checking for common points of failure are routinely undertaken for mission critical enterprises .

Telecoms plant is often owned or used by more than one company and so buying from company A doesnt mean they use different networks from company B . In this case it is unthinkable that they serve sucha critical facility with just one cable . The reason I said the CEO should get the chop is because he has created or tolerated an obviously slack cunture and attude when it comes to resilience as IT is critical to all major airline ops.. He gets a big bonus when they make a lot of money but he does little or none of the work to do that himself. . In this case of course LH shouldnt run their own telcoms network but theyare a big enought company in Germnay to get T Systems and other suppliers to do exactly what they want .
This is big screw up so someonebig has to be resopnsible
pax britanica is offline  

Thread Tools
Search this Thread

Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service - Do Not Sell or Share My Personal Information

Copyright © 2023 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.