Go Back  PPRuNe Forums > Ground & Other Ops Forums > ATC Issues
Reload this Page >

Problems at Swanwick?

Wikiposts
Search
ATC Issues A place where pilots may enter the 'lions den' that is Air Traffic Control in complete safety and find out the answers to all those obscure topics which you always wanted to know the answer to but were afraid to ask.

Problems at Swanwick?

Thread Tools
 
Search this Thread
 
Old 8th Dec 2013, 08:48
  #81 (permalink)  
 
Join Date: Mar 2001
Location: etha
Posts: 300
Likes: 0
Received 0 Likes on 0 Posts
1066, you have it wrong.
zonoma is offline  
Old 8th Dec 2013, 09:55
  #82 (permalink)  
 
Join Date: Sep 2012
Location: KingsLynn
Age: 37
Posts: 40
Likes: 0
Received 0 Likes on 0 Posts
Thumbs down

when i went for an interview there i was told that there is a training area that could be used for long term contingencies of some of the airpsace [they showed us lots of screens with nothing on them] and also a catastrophic disaster set up there all in the head off building complex so i am confused.this was told to me by a controller from the training school [lady].why is air traffic so confusing>>>>>>>? lol rofl
YouSaidBolt is offline  
Old 8th Dec 2013, 10:22
  #83 (permalink)  
 
Join Date: Dec 2013
Location: N/A
Posts: 14
Likes: 0
Received 0 Likes on 0 Posts
The problem seems to have been they couldn't switch in any extra panels. They had plenty of panels, without the training ones, but just couldn't use them.
Forget millions of lines of code. It only takes on byte, in the wrong place to cause such a problem.
(Finding it , among all the lines of code, that's the fun part).
Fran Tick is offline  
Old 8th Dec 2013, 10:28
  #84 (permalink)  
 
Join Date: Dec 2013
Location: USA
Age: 34
Posts: 1
Likes: 0
Received 0 Likes on 0 Posts
Now that the service has been restored the inquest will be under way.

If I was in charge of NATS I would want to know;

what triggered the initial failure of the telephone system and its backup?

given that a hard reboot and/or a software reload cures the vast majority of system failures why did it take so long to restore this particular critical system?
ADHD is offline  
Old 8th Dec 2013, 11:10
  #85 (permalink)  
 
Join Date: Jan 2002
Location: RWB, UK
Age: 77
Posts: 73
Received 5 Likes on 2 Posts
Thanks zonoma.
1066
1066 is offline  
Old 8th Dec 2013, 12:41
  #86 (permalink)  
 
Join Date: Oct 2009
Location: somerset
Posts: 120
Likes: 0
Received 0 Likes on 0 Posts
YouSaidBolt

I guess you didn't get anywhere after your interview?
possibleconsequences is offline  
Old 8th Dec 2013, 13:46
  #87 (permalink)  
 
Join Date: Jun 2006
Location: UK
Posts: 152
Likes: 0
Received 0 Likes on 0 Posts
Wicked ! But very funny.
Flapping_Madly is offline  
Old 8th Dec 2013, 17:13
  #88 (permalink)  
 
Join Date: Aug 2009
Location: England
Posts: 98
Likes: 0
Received 0 Likes on 0 Posts
OK Who broke the stick ?

flying lid is offline  
Old 8th Dec 2013, 20:02
  #89 (permalink)  
 
Join Date: Jul 2001
Location: Ottawa, Canada
Age: 70
Posts: 213
Likes: 0
Received 0 Likes on 0 Posts
What few people understand about software, is that software is like Swiss cheese. Full of holes, but only the ones that matter get found and fixed , by testing and through use. There will always be a few that matter and only show up later though.
I worked for many years on a voice switch very similar to the one at Swanwick, and software upgrades in the field were always worrying. The new software would first be tested at the manufacturers facility, then we (ops and engineering) would the run our own testing scenarios at our technical centre for a couple of weeks, then we would head off to one of the smaller centres and install the upgrade on a night shift. We would "split" the system and upgrade one side of the switch with the new code, run that for an hour or so and if that worked we'd switch sides and then upgrade the other switch. Next the two systems were put back into full redundancy mode and hopefully would play nicely together. All this would take upwards of 8 hours, and the system HAD to be fully operational by 0600. Even once the system was bak, the next few hours were still critical as controllers had the ability to dig up anomalies that we never could find in testing.

Many 16 hour shifts were spent doing this .... I have the grey hair to prove it!
ex-EGLL is offline  
Old 8th Dec 2013, 20:14
  #90 (permalink)  
ojs
 
Join Date: Jul 2003
Location: London
Posts: 54
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by Fran Tick
Forget millions of lines of code. It only takes on byte, in the wrong place to cause such a problem.
(Finding it , among all the lines of code, that's the fun part).
Yes, agreed... But the change from night to day configuration is carried out by code that must have executed hundreds (if not thousands) of times. So the question is more: what changed?

Agreed, it could be some obscure bug in the code (suddenly reaching that point where a short int no longer works and you need a long one), but personally I think that's unlikely.

My guess:

a) Something broke that wasn't redundant;
b) Somebody implemented a change and it went wrong / had unforeseen consequences (or the backout plan failed / made it worse);
c) There is redundancy in the system but as with all major outages which are a result of a series of unfortunate minor faults that unfortunately occur at the same time, the redundancy was unavailable.
ojs is offline  
Old 8th Dec 2013, 20:19
  #91 (permalink)  
 
Join Date: Apr 2009
Location: Hotel Gypsy
Posts: 2,821
Likes: 0
Received 0 Likes on 0 Posts
I think it has already been said, but ScATCC(Mil), Prestwick closed on Friday night with the task being transferred to Swanwick. One presumes that entailed a new banana or two and a tweak to the VCCS?
Cows getting bigger is offline  
Old 8th Dec 2013, 21:38
  #92 (permalink)  
 
Join Date: Mar 2001
Location: etha
Posts: 300
Likes: 0
Received 0 Likes on 0 Posts
ojs and many others have referred to the "switch from night to day" mode, it isn't like something is different or anything is switched on or pushed, it is just an easy way of letting the general public with no knowledge what so ever try on understand a miniscule speck of the situation. It is no different to standard bandboxing and splitting of sectors that can be happening every minute of every day, just that at night the room is bandboxed almost as much as it possibly could be, and this problem arose as the traffic levels began to pick up and the sectors needed splitting.
zonoma is offline  
Old 9th Dec 2013, 10:45
  #93 (permalink)  
 
Join Date: Jul 2004
Location: On the wireless...
Posts: 1,901
Likes: 0
Received 0 Likes on 0 Posts
Originally Posted by ex-EGLL
I have the grey hair to prove it!
Is it still 'over the collar'…
Talkdownman is offline  
Old 9th Dec 2013, 12:18
  #94 (permalink)  
 
Join Date: Jul 2001
Location: Ottawa, Canada
Age: 70
Posts: 213
Likes: 0
Received 0 Likes on 0 Posts
Is it still 'over the collar'…
Well the capability is still there, but it doesn't reach that length too often these days!
ex-EGLL is offline  
Old 9th Dec 2013, 14:03
  #95 (permalink)  
 
Join Date: Oct 2000
Location: Berkshire, UK
Age: 79
Posts: 8,268
Likes: 0
Received 0 Likes on 0 Posts
As the vicar said to the actress....
HEATHROW DIRECTOR is offline  
Old 9th Dec 2013, 16:36
  #96 (permalink)  
 
Join Date: Feb 2010
Location: farfaraway
Posts: 162
Likes: 0
Received 0 Likes on 0 Posts
Having watched various TV channels and newspaper reports relating to the comms. hiccup at Swannwick is it not about time the top neddies at NATS finally come to realise that NATS does translate as National Air Traffic Services and rebrand the company accordingly.
obwan is offline  
Old 9th Dec 2013, 20:00
  #97 (permalink)  
 
Join Date: Dec 2013
Location: N/A
Posts: 14
Likes: 0
Received 0 Likes on 0 Posts
Changes introduced at earlier minor upgrades can sometimes not kick in until after a subsequent restart, sometimes triggered by eg. a minor hardware fault sometime later. Had such a problem elsewhere only a week before and a total (power/off ) reload of the system didn't clear it (would only have precipitated it in fact - if it hadn't already shown up by then) . Realising your last resort has failed is scary.

Only a backup/redundant system with total autonomy from the main protected system can necessarily avoid running into this mess. Several mission critical operations have them. Can only wonder why this one didn't, or did they realise what redundancy means ?
Fran Tick is offline  
Old 9th Dec 2013, 21:23
  #98 (permalink)  

More than just an ATCO
 
Join Date: Jul 1999
Location: Up someone's nose
Age: 75
Posts: 1,768
Likes: 0
Received 0 Likes on 0 Posts
Can only wonder why this one didn't
Lon More is offline  
Old 10th Dec 2013, 06:46
  #99 (permalink)  
 
Join Date: Dec 1999
Location: LHR/EGLL
Age: 45
Posts: 4,392
Likes: 0
Received 0 Likes on 0 Posts
So you want a completely redundant VCCS function, to enable 100% capacity ops when the current one is degraded? You want to develop and maintain a new set of hardware, from a different manufacturer, with a completely independent update/upgrade cycle? A new software set up with another million lines of code? You want to design and develop new interfaces to all the related systems in adjacent FIRs?

Really?
Gonzo is offline  
Old 10th Dec 2013, 07:09
  #100 (permalink)  
 
Join Date: Oct 2000
Location: Berkshire, UK
Age: 79
Posts: 8,268
Likes: 0
Received 0 Likes on 0 Posts
Strikes me that it's often technology for technology's sake. We were told that when we moved to the new approach room at Heathrow when simple tasks like entering a 4 digit transponder code on the radar took about a dozen steps witj on screen menus, etc. The old clockwork system did it in seconds.

Don't know if Gonzo remembers the old 6th floor approach room? On the wall at the north end was a bank of telephone keyboards which were copies of all the other keyboards in the room. They were selector panels and when a position was opened or closed one simply selected or deselected the telephone lines one required using the "copy" of the keyboards on the operational postions. It never went wrong. Come back clockwork!
HEATHROW DIRECTOR is offline  


Contact Us - Archive - Advertising - Cookie Policy - Privacy Statement - Terms of Service

Copyright © 2024 MH Sub I, LLC dba Internet Brands. All rights reserved. Use of this site indicates your consent to the Terms of Use.