Problems at Swanwick
 

Problems at Swanwick this morning again, after the shutdown last night. Long delays expected. Richard Everitt apologises on Radio 4.
Oh well, at least it was delayed from last night!

More details please.

Oh well, another busy afternoon looms. Saves us getting bored.. and keeps us out of that damned hot sun!

May be more "interesting" than you think, HD. My Wx forecast says lots of Cbs later... have fun ;)

Best rgds
BEX

Up to 4 hour delays into some airports in the south - LGW/LHR.
I believe LCY is closed to inbounds at the moment, can anyone confirm?
BRAL ops on the ball for a change, filing lower levels and avoiding some slots.

Have fun in the SE this afternoon chaps!

Feel sorry for the SLF stuck in the terminals with only overpriced tea/coffee and stale sandwiches for company!:p

With the operation stuck in the TDU, and daytime staff numbers it could be that for the first time Swanwick is actually overstaffed!:eek:

The BBC have picked up on the fact that the DD&C performed last night should have happened on Wednesday.

They are also saying that it was delayed until last night due to the number of extra flights connected with the European Controllers Cup..... (I wish!), no, sorry, I mean the European Cup Final.

No mention of the real reason for the delay until Thursday night, which was not enough staff at Swanwick. As it was several sectors were closed on Wednesday night.

Until VERY late in the day, Swanwick management were INSISTING that the DD&C go ahead on Wednesday.

As to todays problem. I wonder when they discovered it wasn't possible to "spilt" the sectors? Not a nice feeling when the traffic is relentlessly coming off the Ocean and you're getting buried.

Getting Swanwick back up will be a challenge. Will a complete shutdown (of Swanwick) be required before rebooting the whole thing, a bit like DD&C in reverse? If so when? A tough decision.

All this makes me very sad.

Rgds BEX

Well, something's up and running again -- loads of delays have just melted away completely. I was watching an ETD on a flight from EDI -- nominally 20:15, it was at 00:16 for most of the morning. Then it went back to 22: something, and now it's back on schedule.

How good are those rollback procedures?

R

Calling all controllers,I got a rubbish roster this afternoon,SOU to DUB,DUB to SOU,SOU to EDI,EDI to SOU then SOU to IOM,any help in actually getting any of the above completed before midnight would br great,yea and the wx looks rubbish too!!

Well, at least it will all be sorted by the summer(?)
Not noticed anyone mention the mysterious workstation freezes that happened a while back- as far as i know systems never found out what happened they just cured themselves.....
You're right Bex it IS sad (especially when we all expected this)

N H, suggest you pack plenty of reading material and extra sarnies. Speak to you this afternoon, evening and probably night...

Does any body out there know if the operation is SWIMMING or fully electronic but in the TDU?

If the former then I suspect that the only way back, because of NAS link problems, is for the airspace to become very empty before an attempted reboot.

Z

No, I think they came out of the TDU after the DD&C, went electronic and THEN when it came time to split for the morning, they couldn't OPEN SECTORS.

BEX

P.S. The BBC is reporting the system now fixed, but LONG delays are inevitable

Thanks Bex, that should mean a rapid return to normal once they can split again. Unfortunately the back log will take a while to clear.

If any SLF is reading this thread please accept the Swanwick ATCO's and ATSA's apologies for any inconvenience which is not of their making. NATS management were warned that this sort of thing could happen and will continue to happen for the forseeable future.

Z

ok, what's an "SLF"?

SLF=Self Loading Freight..... err.. Passengers

BEX

I was in for the DD&C last night and it all went OK, no problems at all! Bandboxed back in Ops at 0130Z and checked out all the Workstations, no problems. Left the control staff to it, we were off home by 2Z. Don't know why the day watch appeared to be unable to split out, possible bug in the software drop?

"possible bug in the software drop" Poosible. Wouldn't be the first time and won't be the last, either.

The NAS AIRAC update went perfectly, no probs with the NASSFS link either, NAS shutdown between 0009 and 0039 and the beast arose from its slumbers first time.

Possible instability with Swanwick software drop? Did a rogue flight plan do some damage within the workstation token ring?

Condolences to the morning watch (Air Traffic AND Tels) and passengers involved.

Token ring, now there's an expression I haven't heard for a long time!

What's it used for at Swanwick?

Absolute pain of a morning shift, loads of flight plans coming out but few getting airborne, London City fills up, lots of level capping and bizarre reroutes to try & avoid NERC sectors, then Pease radar has a wobble, saturation of SSRs due to the vast number of a/c still on the ground at EGLL, NERC preparing to go to SwIMM mode, was talk of going to SwIMM at 3pm local.

As usual, operational staff at the coal face kept the show running but we're all getting a bit fed up of this now. Pax, airlines & NATS own staff are being let down bigstyle every time these problems occur, I dread to think the reputation NATS is getting with the flying public.

1261

All the LAN's at LACC use the IBM token ring system.


Iron City:
You're quite right, neither the first nor last time. No matter how exhaustive your testing, with a system as complex and large as LACC there are BOUND to be bugs which only surface when you drop onto the operational system and let the 'customers' lose on it.
We do our best but we are human and not perfect.

Just looking at the easyJet Website, it talks about:

"Following a significant failure in the UK's air traffic control system this morning, there has been a subsequent failure in the Brussels Air Traffic Slot Co-ordination Management System. "

I presume they mean CFMU - any details please?:(

The problem was due to 1 ATC workstation computer.

When it started up after the DDC, it had problems communicating with other computers in the system.
When it came to split out the sectors from this workstation, it couldn't copy it's Flight Data to the new workstation. So when ATC tried to use the new workstation, it received it first flight data update and goes "I don't know about this flight, I'd better restart".

Only the sectors on that sectors on the 1 problem workstation were affected, but nobody could tell what was the problem from the Monitor System. So they didn't split the other sectors.

This is a problem which has been around for a while, but has been deemed my management to be not urgent, and isn't due to be fixed for another year.

I blame September 11th! :D

Management made these problems; let management fix them!

I hope they don't fix it. More days like today (i.e. having time to read The Herald cover-to-cover) would be much appreciated. Thanks, Richard!

1261:

Its not management who made the problem, its just a consequence of a very complex system. No chance of 'them' fixing it, thats down to the engineers and programmers at the coal face who take considerable pride in their work and take it personally when things 'fail' to work as they envisaged.

DD&C
 

Looks like this system problem could have been avoided totally if the day support engineers had been in - but that costs money - at least £1000 total and heh, they don't even work for Swanwick anymore because NATS management don't need 'experts' for COTS systems - and yes, the bit that broke is COTS.

TAG be warned. :eek:

Take3Call5,

I think that Tanglefoot has hit the nail on the head. I know that it's the guys at the coalface who will always end up digging us out of whatever hole we're in, be it an engineering or ATC one!

I was referring to management metaphorically "fixing it" by allocating the resources required (both in terms of cash and manpower) to do the job properly with a new (and unproven, at least operationally) system.

The first step on the road to recovery is, after all, admitting that you're sick :)

Swanwick problems- The Times online
 

WHAO THERE....

Lets see if I've got this straight. Todays chaos was caused by a KNOWN problem that can occur on a a SINGLE workstation that our beloved management KNEW ABOUT but deemed "not urgent".

Is this TRUE?

BEX :mad:

What frightens me is that this is the third time that I can recall in recent months where total chaos has been caused by systems failures in ATC.

The cost in money terms to all airlines but in particular to my own is astronomical.

That is before we look at the loss of goodwill from passengers.

Where will it end?

No criticism of the ATCOs caught up in it, you must be as frustrated as we pilots are.

Once again the system fails and the ops staff (atsa, atco and atce) are left to clear up the mess!

Funny thing though, for the first time that I have seen, the TV news said that the SWANWICK computer had a problem, where usually they blame the one at WEST DRAYTON.

Change in attitude from the management? They are usually so proud of their very expensive system in Hampshire, that I am surprised they didn't blame West Drayton again.

Or is that just the same computer?

Now I'm confused.

It's all semantics to cover up for a system which seems to be teetering on the edge of stability.

Everett is correct in that this is the first failure which is attributable solely to the NERC system which has brought the place to a halt during the day. What he skates over, however, is the fact that the first NAS failure was corrected relatively quickly, but the NERC system had to remain off for the REST OF THE DAY because of the perceived difficulty in getting it back on line again. That in my book is a failure of the NERC system as it failed to do the job for which it was designed: move air traffic more efficiently than we did at LATCC. Had we still been at LATCC on that day we would have been up and running again in about an hour.

You might not also be aware that at about 1230 local today the NAS link failed for about 2-3 minutes and we all prepared to go manual. This was shortly after we had heard that the TACT computer at FMP Brussels had also failed. Delays peaked at about 178000 minutes total, I think. That's 123.6 days!!

All of this points to a system which is inherently unstable, and cannot cope with inputs which it doesn't like. The computer upon which I am typing this has protection in it which stops me doing things which will make it crash. It cost just under £1000. Why can't a multi-million pound system do the same???:confused:

Where were the airline updates?
 

Firstly thanks for all the work that ATC and airline personnel do during these difficult times, it does however lead me to ask one question.

As a corporate travel agent, I have spent today fielding calls (until I finished at 2030bst) from travellers stranded at Heathrow, and other european airports, asking what is happening to their flight. When I check Flifo in the BA and BD systems there are no updates, not even to say the flight was cancelled, and I then have to start searching through the Net for airport screens (which mostly only show arrivals) to try and find out what is going on.

In this day of supposed technical automation I always feel at a loss as to why, when the data is most required, it is not available. The public service voices always seem to be saying that travellers should contact their airlines, however it is then normally impossible to get through on the phone lines, and their agents - who SLF expect to know all - are stuck as well.

Oh well, back to the drawing board and another week that will probably see us sorely tried by more distruption (only joking) somewhere in the world that will cause havoc - whether it be air, rail or any other means of transport.

Have a good weekend all and keep on smiling!

Scottie Dog

I have a feeling the Swanwick ATCOs which turned up for their morning duty this morning were thinking exactly the same thing!!!! :(

Didn't quite appreciate how the wording might have looked, but a very true and poinent comment.

Trust it will soon be resolved - well maybe this century!

Scottie Dog

******.....
 

And I thought that I was having a bad day!

Hope you all manage to sort this one out and just to let you know that one of the SLF appreciates what you are doing/going through.

Thanks guys - have a beer on me.:D

How many times can this happen again until something really nasty happens? We`re just about to enter our busiest period of the year - even after September 11th this summer will be AS busy as last year, if not busier! We need to have more confidence in the system and know that it will not let us down - we just can`t do that at present. I expect all the boys and girls did an excellent job today (like they ALWAYS do in extreme circumstances) with picking up the pieces and, knowing what they`re like, no doubt injecting a bit of humour into the situation! Well done all!!! Whose Watch`s turn is it to cop it next???

Spare a thought for the unlucky Paddys that have to deal with the resultant flak every time there is a problem with the London computer. This is the third time in as many months that there has been an explosion of traffic through the Dublin and Shannon FIRs.

Mind you, its not just when the computers refuses to work at Swanwick that these problems occur. Its a daily occurance, every morning with the eastbound NAT London impose gallons of reroutes on the Irish so that they get what they want coming into English airspace.

In effect, Ireland has taken over the complexity of the airspace so that London don't have to deal with it. How much longer can NATS struggle on before the whole thing sits down?