Swanwick's gone tits up!!!!

[atco-matic]

Just had a call from crew friend sitting on the ground at heathrow wanting to know why they were going nowhere amid rumours of computer failure at Swanwick.... .. .And here is what the CFMU says:. .<a href="http://www.cfmu.eurocontrol.be/chmi_public/ciahome.jsp?serv1=aim&aim=00044713" target="_blank">http://www.cfmu.eurocontrol.be/chmi_public/ciahome.jsp?serv1=aim&aim=00044713</a>. .<a href="http://www.cfmu.eurocontrol.be/chmi_public/ciahome.jsp?serv1=aim&aim=00044720" target="_blank">http://www.cfmu.eurocontrol.be/chmi_public/ciahome.jsp?serv1=aim&aim=00044720</a>. .. .Apparently they had problems with the fones this morning again... don't know if its related.. .. .What a larf!

[Loki]

Impossible, don`t believe it, was told during OCT it would never happen. So there.

[atco-matic]

Heres the latest from CFMU.... .. .<a href="http://www.cfmu.eurocontrol.be/chmi_public/ciahome.jsp?serv1=aim&aim=00044722" target="_blank">http://www.cfmu.eurocontrol.be/chmi_public/ciahome.jsp?serv1=aim&aim=00044722</a>. .. .seems they are having problems with ''associated equipment'' so maybe it was the fones? Anybody coming home from work on an early go can tell us whats been going on?

[eyeinthesky]

'Early go'?.. On a failure day: unlikely!

[BDiONU]

On holiday so no idea why there's a problem. However it does say that LACC has gone manual, which leads me to suspect either a NAS link failure, or NAS failure which took a while to sort out. Effectively if NAS is down for more than 6 minutes then the ATC Watch Supervisor has to decide on whether to go manual or not. Going manual means that they can eventually move more traffic once its established, but initially there will have been a huge impact on flow.. .Recovering from a manual to an electronic operation means that traffic levels have to be massively reduced as LACC re-connects to NAS and has to update all the flight data. It is therefore better for all concerned to remain manual until the traffic reduces sufficiently to allow them to go electronic.. .I am only postulating a possible reason why they went manual but the rest of it is how it works.

[Carnage Matey!]

Did anything go wrong at Manc Control? Where I was sitting we were initially told it was their problem and a few minutes later we were told it was now affecting NERC.

[411A]

You see CM, the guys on the ground NEED your help and attention...and sympathy. Give 'em a break!

[StoneyBridge Radar]

Even if you'd been in work niteflite, I'm not sure how the Sim would have had any effect on the crash....... .. . <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> . <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> . <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> . <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> . <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> . <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> .

[Exel]

And here is the news at 10 - BONG BONG BONG <img border="0" title="" alt="" src="biggrin.gif" /> . .. . [b]CHAOS AT AIRPORTS AS NEW UK ATC SYSTEM FAILS !!. .. .I Knew today would be a good day to have a days leave !!!! <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> !!!. .. .Good luck guys - Wonder what the official management drivel will be ???????? or have i just read it on CEEFAX ?

[BEagle]

Ah-hah - that explains it. Came back from a North Sea AARA flight today needing to do a Strumble HMU run for an equipment check and the ATC chaps advised us that London Control's 'computers had crashed'. Nevertheless we managed to achieve what we needed with a very, very slight delay and the sound, unflappable ATC from LATCC was exemplary. Thanks a lot, chaps - and I hope that the ZX-80s recover soon!!

[Bright-Ling]

This is the latest on the CFMU website:. .. .LONDON COMPUTER FAILURE UPDATE. .Valid from 27 Mar 2002 to 27 Mar 2002 (released 27 Mar 2002 at 21:32:22). .. .GROUND STOP FOR ALL DEPARTURES FROM THE UK. ... .NO FLIGHT IS TO GET AIRBOURNE FROM ANY UK AIRFIELD UNTIL FURTHER . .NOTICE BECAUSE WE ARE UNABLE TO MANAGE THE PLANNED LEVEL OFDEMAND . .WITHIN THE LONDON FIR.. ... .THE STOP WILL LAST FOR AT LEAST 1 HOUR.. ... .FMD BRUSSEL ON BEHALF OF LONDON FMP

[niknak]

And guess what the response to the computer failure was.......... .Issue everyone with a knee jerk 2100 departure slot, regardless of where they were going, no consideration was given to the fact that anything north of Birmingham was unaffected by the failure of NATS management to ensure safe and proper procedures were in place.. .Bunch of t**sers.

[LimaPapaLima2A]

Ahhhh niteflite01 - has your cover just been blown? <img border="0" title="" alt="[Big Grin]" src="biggrin.gif" /> . .. .Are you a collegue of mine in the TS? I wonder which one? . .. .Has the great unit-wide hunt for you just ended? . .. .Will Danny give you your money back for your lovely blue moniker?. .. .See you tomorrow!. .. .From your young scouse ATSA buddy in training <img border="0" title="" alt="[Wink]" src="wink.gif" />. . . . <small>[ 27 March 2002, 23:01: Message edited by: LimaPapaLima2A ]</small>

[atco-matic]

story on ceefax says the problem was at ''west drayton'' not at swanwick... hmmm interesting, is NATS to embarassed to admit to the general public that the problem was at swanwick? . .. .Is nobody home from the afternoon watch yet to tell us what went on?

[PEAWEE]

Spoke to a college who had the pleasure of todays madness, its ok as management came into the ops room and purvayed the chaos, yet did nothing about the madness that was about to ensue on the north bank. <img border="0" title="" alt="[Mad]" src="mad.gif" />

[5milesbaby]

niknak, Many things north of Birmingham would have been affected, as long as they were requesting FL280+, and changing to FL200+ over Birmingham cruising south. . .. .Still believe it was the link that failed so technically was at West Drayton as that is where the computer rumbles, however, for non-ATC types, it affected LACC (or Swanwick, or Nerc) only.

[Big Nose1]

If it was a NAS link failure, then it could have been at West Drayton end, the Swanwick end or somewhere in between. The fact that only Swanwick was affected would suggest a problem at that end.

[atco-matic]

meridian news just had a story on about the failure, but absolutely no mention was made of swanwick whatsoever!!!

[TrafficTraffic]

</font><blockquote><font size="1" face="Verdana, Arial, Helvetica">quote:</font><hr /><font size="2" face="Verdana, Arial, Helvetica"> NO FLIGHT IS TO GET AIRBOURNE FROM ANY UK AIRFIELD UNTIL FURTHER . .NOTICE BECAUSE WE ARE UNABLE TO MANAGE THE PLANNED LEVEL OF DEMAND . .WITHIN THE LONDON FIR. </font><hr /></blockquote><font size="2" face="Verdana, Arial, Helvetica">Sounds like a normal day in London to me.... <img border="0" title="" alt="[Eek!]" src="eek.gif" /> <img border="0" title="" alt="[Eek!]" src="eek.gif" /> . .. .Sorry Bex...did they make the Font even smaller? . .. .Dont you hate it when you say "it will never work, it will never work...." and no one listens and then, guess what - it doesnt.

[Flight Plan Fixer]

To put everyone out of their confusion; here are the facts re: yesterday's delays (I'll put my comments on the NATS forum).... .. .NAS - the flight processing system feeding all of UK airspace and beyond - at West Drayton had a set of six CNN aborts at approx 1620 yesterday. The OLDI links - and of course, the link to Swanwick - were thrown out by the problems.. .. .The system had to be restarted to restore stability and cure the FLOP, and was brought back online quickly - the operational engineers and FDP at LTCC did a great job. . .. .All of the OLDI links were brought back straight afterwards, with the exception of the SFS link to Swanwick...procedures dictate that that link can only be brought back at the request of the ATC Watch Manager at LACC.. .. .Swanwick obviously went manual straight away, but even a few minutes outage at that time of day causes massive delays and demands flow - again, ATC at both West Drayton and Swanwick handled the situation as best they could.. .. .The link to Swanwick stayed down for several hours. Pre O-date, the link would NOT have been the biggest factor and normal ops would have been resumed within half an hour.. .. .The aborts are still being investigated - rumours of rogue flight plans and Irish Gap problems are going around, but I couldn't possibly comment...i.e. I don't know!. .. .As this was the first time NAS has FLOPped since O date, the whole problem was a learning curve for all concerned - I'm sure there are lessons to be learned and if there is a next time, it'll be handled better and faster. Well done to all ATC involved around the country.. .. .Can anyone from Swanwick ATC comment on why the NASSFS link stayed down for so long?. . . . <small>[ 28 March 2002, 07:02: Message edited by: Flight Plan Fixer ]</small>

[BEXIL160]

Can't comment on why the Restart at Swanwick took SO LONG (0100 this am I'm told).. .. .It's worth pointing out the FACT that Swanwick still relies on NAS and cannot function without it. So we have a "new" ATC centre that cost £XXX million, opened late, has all sorts of faults, and yet still needs what is basically a souped up IBM9020D of late 1960's vinatge to function.(or at least software from the 9020) . .. .Worse still NERC is very vunerable to NAS "hiccups", a regular occurance at what was LATCC, with largely manual system that was very tollerant of errors.. .. .I believe I'm right in saying that had yesterdays (brief) failure occured at LATCC, there would have been very little impact. Progress? . .. .To my dear old colleague Traffic... yep, I've had enough. "I told you so" doesn't make me feel any better. I will continue to go in, do the best job I can given the available tools, but contribute anything else, NO. Can't be bothered anymore.... No thanks from management, no admissions that they were wrong, no constructive dialogue on how to put things right.. .. .It's a nice day today. Think I might go for a sail.... .. .BEX

[BDiONU]

Yes unfortunately LACC still requires NAS, like the other NATS units. It was asked for right at the beginning of the project, NOT to use NAS but the team were lumbered with it.. .FlightPlanFixer:. .See my previous post for the answer to your question. Basically, following a NAS FLOP, LACC enters SWIMM (Swanwick In Manual Mode) and the NAS-SFS link will not be recovered until traffic levels are low enough (very low!) to allow the sector staff to match the electronics with the manual operation. Generally that won't be until late at night, in this case 1am sounds right as there will have been some delayed flights to try and sort out before restoring the link.

[Hooligan Bill]

Carnage Matey,. .. .One of the problems those of us at airports and indeed at least one en-route centre had was actually trying to get some positive information as to the problem. Obviously the main priority was trying to get a system up and running and achieve the maximum flow rate possible. However, the information we got was somewhat patchy and was not helped by the fact that the situation was changing by the minute. I know it does not help you sitting on the ramp with an aircraft full of passengers for hours and not being able to tell them anything. Believe me though, a lot of people put in a lot of effort behind the scenes to try and get everybody away with the minimum of delay.

[BDiONU]

Flight Plan Fixer:. .Sorry, I obviously wasn't clear enough. Once NAS is up and running the LACC NAS-SFS link can be restored when the Watch Manager chooses. However as soon as you do then the OLDI ACT's get sent and they get the LAM's, but LACC is still manual and will still pass manual estimates. Also other NAS units will get ACT's. But the main confusion will arise within LACC as ATCO's will be presented with some aircraft whuch have all the electronic indications and some which don't. Busy sectors will take a long time to recover electronically whilst they are also running a manual operation. Hence the need to wait for VERY low traffic levels as you can move a lot more traffic manually rather than impose draconian flow measures to get the traffic levels low enough to recover electronically, then let everything get airborne.. .I am not neccesarily 'loyal' to the NERC system, just trying to explain how things work. Believe me the ATC Systems Development team had pointed all these things out to management long before 'O' date. Apart from loss of node a NAS FLOP is one of the worst fallbacks to experience. However we are assured that NAS is becoming much more reliable! Yeah right!!

[jackmorgan]

landed snn at 20:10 launched again at :057 !!!! we counted 3 other aircraft in london control before being launched to mastrict...felt sort of hard done by !!!!

[Mr Chips]

I would like to leap to the defence of NAS....we have been friends for quite some years now, and she/it/he serves well if you treat her/it/him right.. .. .The problem came from an...interesting...input as I understand, regading the Irish gap (as someone said to me today, "we keep making the hole smaller") but NAS was back up and running within a reasonable time (I know more than 30 seconds outage is unreasonable, but you know what i mean) - it was the SFS re-synch that caused the problem. .. .Leave NAS alone!!!!!

[Scott Voigt]

Mr. Chips;. .. .I would have to disagree with you. You can NOT proceed forward with modernization until you get RID of NAS. The system is old and clunky. You can't do anything even close to modern with it. It is time to dump Jovial and BAL and move on.. .. .regards

[Flight Plan Fixer]

Startover and Take 3 Call 5.... .. .Thanks for your detailed and accurate reasoning re: the NASSFS link being down for nine hours. After asking around at work, and spending some time doing the LACC Method of Operations CBT exercises, I can now see how complex and labour intensive catching the workstations up to the dataflow when the link is restored is.. .. .That, coupled with the usual lack of air traffic staff at LACC that day, made the link restoration possible but not advisable from a safety and delay point of view. See what you mean now, guys! Glad the aborts didn't happen at 8 in the morning.... .. .Have to agree with Scott; either NAS has to be replaced, or relocated to Swanwick and the link subsumed into the rest of the LACC token rings.. .. .Will promise to post with my brain in future and give my ar$e a rest!

[POMPI]

More on the service ring(s) !, the Prime ring is probably going full tilt already.

[Mr Chips]

But Scott - it wasn't NAS that caused the problem...it was Swanwick. NAS was back up in 1/2 hour, Swanwick in 9......

[BEXIL160]

True.... . .. .But, NAS, old and relatively reliable is way, way past the limits of what it was designed to do. Although we're all fairly comfortable with NAS, it's limitations and it's quirks, it's not possible to develop it any further. Hence the need for something completely new.. .. .What we have got, however, is the Swanwick system. For all it's pluses and minuses, it still relies on good ol' NAS. It's bit like having a nice new Jaguar with all its modern creature comforts, but with the ignition system from a Model T Ford.. .. .Until we step away from NAS, this sort of failure will happen again. It's impact in the pre NERC days was limited and relatively easy to rectify quickly. Post NERC, lose NAS even for a short time and you say goodbye to NERC electronics and hello to manual (slow) ops until either the traffic is reduced by draconian flow (STOP ALL), or until the early hours of the morning when there is little traffic anyway. Only then is it safe to bring NERC back.. .. .Until the next time..... .. .Rgds BEX

[sony backhander]

completely agree with you bex, management WERE told i believe along time ago that relying on NAS was a BIG mistake-its only a matter of time before some poor sod at lacc has a nasty -hope its not too nasty is all i can say.... .so we have:. .crap screens . .crap r/t. .NAS being crap-. .bargain for £600m really wasn't it

[BEXIL160]

Well, not EVERYTHING is Crap.... .. .NAS ain't perfect, but normally does do the job it was designed to do, and a lot more besides. I have little criticism of it in that respect, and a whole lot of admiration for those that maintain it. Well done Tels! . .. .However it's expecting an awful lot of system designed in the 1960's to be 100% reliable today, which is why I (and many others) wonder why Swanwick was designed in such way as to rely on it.. .. .The RT at NERC is being FIXED. New microphones, headsets, and (possibly) less digital/analogue interfaces.. .. .The NERC screens? I agree, A FUNDEMENTAL PROBLEM. One that HSE have highlighted. Reasons in writing required, with a Timescale for fixes. Hopefully HSE will not back down.. .. .Anyway the point is clear. Swanwick relies on NAS. It shouldn't have to, but that's the way it is. I personally think this is/was a mistake. Whilst Swanwick continues to rely on NAS the events of Wednesday last WILL happen again, and again, and again....... . .. .BEX

[BEXIL160]

Oooops...Apologies. To whoever does do such a good job maintaining NAS then. Sorry for my ignorance!. .. .Bex