LA(T)CC Shutdowns - Pilot Question

[DrSyn]

I operated a LGW - MIR (Monastir) - LGW last night (Wed 20 Feb), STD MIR 2015. Having arrived at MIR on schedule, we were told there were no restrictions for departure. Just as the first pax bus reached the aircraft for boarding, a 2115 slot materialised. This is virtually unheard of on a mid-winter, mid-week night flight and was quite a surprise. Planned inbound route was via BEXIL.

Our Ops were advised by Brussells that the restriction was Dover-Lydd (15-17) and gallantly attempted a more westerly re-route via Hurn (19-21) which they were assured was unrestricted. This resulted in a revised slot of 2208!! The flow restrictions then appeared to follow a "domino" pattern from East to West across southern England. Ops were told that the reason given by ATC was the ". . . monthly software update of AIRAC data." Apparently, other operators were experiencing delays of up to 180 mins.

Eventually, the best we could get was a 2200 slot on the original route, reaching LGW 1:35 late at 0045.

I did not want to become embroiled in the lengthening Nerc . . . the story so far thread, which is why I started this one. As a sympathetic but nonetheless concerned end-user, I would be grateful if someone could enlighten me here. It would seem from interpolating recent posts by St.Louis and zzz that the problem was caused by a planned shutdown of the system for remedial action and that a further 12 (?) are planned between now and 18 March.

I have been on the phone to my Ops, since returning home, and confirmed that no information was promulgated prior to this event, nor indeed on any of the other intended shutdowns which are allegedly imminent. They will no doubt be pursuing the matter through more official, though undoubtedly slower, channels than PPRuNe! In the absence of qualified comment, it does appear at first view that the information being disseminated in respect of tonight's event was, shall we say, somewhat disingenuous - especially as we are one of the new shareholders!

I have every sympathy with the ladies and gents who do such a splendid job of preventing us from bumping in to one another, and realise that whatever is going on here is not their fault. I am sure that most of my colleagues in the industry appreciate the unexpected glitches which crop up when introducing new (sic) technology. However, if it is indeed true that these disruptions are known about in advance, it would be very helpful to us aviators, our Ops staff, and ultimately our valuable customers if we could be pre-notified.

Any comments from LA(T)CC management or Intelligence from those at the sharp end of ATC would be thankfully received by the fliers and their support teams. Nil illigitimae. . . . . etc <img src="smile.gif" border="0">

[BDiONU]

I was involved in last nights planned shutdown. It is the normal one for every AIRAC cycle which comes around every 28 days and was the first 'real' one at LACC. However the actual shutdown doesn't occur until 0030 so your delay was NOT caused by the shutdown. It was caused as a direct result of an engineering upgrade to the voice comms system during the afternoon.. .Not clear on the exact sequence of events, whether the engineers told the watch manager and he told no-one else or if engineering told no-one (unlikely) or if the watch on duty never understood the implications of a phone upgrade. In any event the result was that a sector tried to bandbox/split out during the upgrade, which is a system no no and caused chaos. Result was 4000 minutes of delays.

On the subject of shutdowns:. .NAS is what actually gets shut down (or to be correct the Host Computer System). This is the computer system which the whole of NATS uses for Flight Data Processing etc. etc. It usually has a shutdown every Wednesday, with a new software drop on the AIRAC days. Generally all planned for 0030, when traffic levels are low and controllers operate manually.. .There have been quite a few problems caused at LACC by NAS (incorrect strip outfall to name just one) and these are being fixed rapidly by the engineers. Hence there being more NAS shutdowns, to allow the patches to be dropped onto HCS and fix the trouble. So there are shutdowns now on Sundays and Wednesdays to enable the fix's to go on.

I hope that helps <img src="smile.gif" border="0">

[Arkady]

Last nights shut down had the added complication of being the first requiring the LACC ops room to be closed and operations to be conducted from the Training suites in an adjacent room. The telephone problems earlier in the afternoon were caused by overly keen engineers configuring the training room to accept us well in advance of the move but not appreciating the consequences. Shouldn't happen again. The LACC ops room needs to be shutdown at least once a month, as opposed to NAS shutdowns that occur once or twice a week. I would expect either type of planned shutdown to impact on LACCs ability to handle traffic, until all the watches (engineering and ATC) are happy with the process. Unplanned shutdowns don't bear thinking about.

[Great Unmanaged]

Almost correct. I think you may find that many of the strip problems are the fault of the Swanwick systems and what they do with the data after it comes through the wall.

Some are NAS related though - so I was told today anyway.

[ghost-rider]

Must admit last night caught us out as well. 5 flights boarding pax and unrestricted for the return UK sectors, suddenly receive 3hr slot delays out of nowhere ! <img src="eek.gif" border="0"> Luckily we had a cunning plan to get round the restrictions ! <img src="cool.gif" border="0"> ( although my boss will freak at the fuel bills !! <img src="frown.gif" border="0"> )

The only question I have is surely this must have been planned by the grownups ( management ) in advance ? <img src="rolleyes.gif" border="0">

Why oh why were LACC flow equally caught out by it ?

And talking of which, many thanks to Lon FMP desk for their assistance last night.

[ 21 February 2002: Message edited by: ghost-rider ]</p>

[atco-matic]

Dr Syn,

''...the information being disseminated in respect of tonight's event was, shall we say, somewhat disingenuous - especially as we are one of the new shareholders! ''

hmmm, we atco's are all shareholders too, you know, but do we ever get treated differently when travelling as staff? (''...you work for who??'') So why should you get preferential treatment then?

[DrSyn]

atco-matic, I will try to make better use of smilies instead of standard punctuation - but not today. However, the point was about quality of infromation. Also, you are always welcome on my flight deck.

Thank you to those who have posted explanations here. There appears to have been a combination of plannned and unforseen circumstances coming together at the same time. The real fault lies in the lack of communication (ironic) to the airspace users prior to the event and subsequent to the additional problems. No one at our end had an informed picture of what on earth was happening. As ghost-rider implies, these things cost us money, but we have a very resillient system available to cope with them if we are kept in the loop - preferably in advance.

I don't want to get involved in the general discussion of NERC and its "new technology", as you guys seem to have done a pretty thorough job of it on this forum and in leaks to Private Eye. However, I could not help a wry smile at the info from Take3Call5 that this brand spanking new system required a phone upgrade!

The over-time, over-budget aspects of this system are already well publicised but it does not inspire confidence to hear that even strip outfall is suffering problems! Indeed, reading the ATCO posts on this forum, one could be forgiven for thinking that the whole project has been distinctly lacking in inspired leadership.

Bless you all for keeping our blips apart!

[Gonzo]

DrSyn,

&lt;&lt;and confirmed that no information was promulgated prior to this event, nor indeed on any of the other intended shutdowns which are allegedly imminent.&gt;&gt;

You don't have to feel left out, we'weren't told about it either. The first we knew of it was when slots like 0300 and 0450 started to appear on the slot computer in the tower at Heathrow for a/c that usually depart just after 2200. Result was much running about a la headless chickens trying to figure out why, and if BAA were going to actually allow all the flights to depart after the night noise ban comes into force at 2330.

All in all a bit of a shambles.

Gonzo.

[BDiONU]

DrSyn. .When LACC was connected up it was using transition phone lines which all went through LATCC. Now we can connect directly and the upgrades, in part, are to remove the old transition lines. Also the operational users sometimes have identified extra lines which they require on individual sectors. These are added in as upgrades.. .The only real upgrade we hope to have in future is the replacement of the old fashioned style (yes OK the centre just opened, but most of the kit has been here for years) digital telephone handsets. These will replace the bleepers currently used to recall control staff when required, and the walkabout phones that supervisors have.

HTH!

[Lon More]

Arkady, &lt;&lt; The LACC ops.room needs to be shutdown at least once a month&gt;&gt; and shutting down the computer to update AIRAC info.. .Why? is there no possibility to reconfigure the sectors in the "huge" ops room in the first case or to work on a backup computer in the second.. .There is more thn one mainframe computer available, I presume. It's just that nobody else in Europe that I'm aware of has to go to the extreme of shutting the system down to do this.. .It was rumoured some years ago that NATS was hiring out the computer to do salary calculations;Any base in fact?

[NERC Dweller]

Lon More - We have to shutdown the entire system at one time for a couple of reasons

1) The Sector Workstations have a lot of functionality which makes use of AIRAC type data, so they have to be restarted to use the new data

2) NERC does not use a mainframe (apart from NAS) , it is a collection of servers performing different roles. These servers use the same adaptation as the workstations so have to be restarted at the same time as the workstations

[ 22 February 2002: Message edited by: NERC Dweller ]</p>

[Scott Voigt]

In the US when we do a chart date change or a new software change we shut down the primary system and run on a backup with no automation between facilities. Like NERC or whatever you are calling it now &lt;G&gt;, we run on a LAN for our displays, but all the flight data processing is still on the HOCSR mainframe...

regards

[POMPI]

A general observation and not aimed at any particular part of this post.

You can shut down various parts of NERC and not have to shut the whole lot down to reload the sectors. The CGW's for example don't need to go down nor do the RDP's. I assume you are not changing the clocks as this used to take up a lot of time.. .In any event it is possible to reload the whole of NERC ie, go from running to stopped and reload and started in about 30 minutes. I know, I used to do it and on my own, involved a lot of running up and down stairs and a bit of cordination, but can it be done.

[iss7002]

It would appear that this is a necessary evil of the technology. The Eurocat 2000 which we use in Aust. requires the workstation to be run in a degraded mode during any upgrade, and then rebooted at some point as each workstation is capable of running in a degraded stand alone mode. Generally the disconnection from the FDP is the major problem as the processing is suspended. After four years the Airac updates are handled OK as we have become more familiar with the requirements of the outage. Where real problems have arisen is during software upgrades which usually require more downtime, the ultimate being where the new software has crashed and the old software has to be reloaded.

[ghost-rider]

It happened again on Wednesday night <img border="0" title="" alt="[Eek!]" src="eek.gif" /> ... is this going to be every month or is it now sorted ? <img border="0" title="" alt="[Confused]" src="confused.gif" /> . .. .Thanks. . . . <small>[ 22 March 2002, 08:41: Message edited by: ghost-rider ]</small>

[BDiONU]

Ghost-Rider:. .Every month, well, to be precise, every 28 days. Thats in line with the AIRAC cycles. Although it will not always necessitate a shutdown of the LACC computer system, depends on whether there's a new or updated software build to be dropped on. The AIRAC's are for NAS (the Host Computer System) and affect all of the NATS units connected to NAS.. .That said there currently isn't one next month, the next one is 516 (16th May).. .If you were concerned about the delays, that was a hangover from the afternoon watch who's controllers didn't have enough validations to fully operate a couple of sectors. Manpower shortage is now the limiting factor for moving traffic through LACC airspace. <img border="0" title="" alt="[Roll Eyes]" src="rolleyes.gif" />

[ghost-rider]

Take3Call5,. .. .Thanks for that. . .. .Re May 16th, should we expect 3-4hr slot delays as per the last two AIRAC updates and re-route well in advance ? Or should it be less painfull next time. <img border="0" title="" alt="[Roll Eyes]" src="rolleyes.gif" /> . .. .A warning on the CFMU Daily Network News would be a good idea IMHO.. .. .Out of interest - how come the updates are hammering UK airspace since Swanwick, as opposed to the seemless updates at LATCC ? <img border="0" title="" alt="[Confused]" src="confused.gif" /> . .. .Point taken about staff shortages. <img border="0" title="" alt="[Frown]" src="frown.gif" /> . .. .Rgds.. . . . <small>[ 22 March 2002, 10:06: Message edited by: ghost-rider ]</small>

[BDiONU]

Hi Ghost-Rider:. .Impossible to say if 16th May will have big problems. Mainly its down to having staff on both afternoon and night shifts with the correct validations. If there are no hold-ups during the day then things will be easier at night. That said there will be some delay, the reason for that answers one of your questions.. .. .LACC is different to LATCC in that at LATCC all the co-ordination between sectors (apart from standing agreements) was done by phone. At LACC its done electronically using the NERC system. Think of a NERC software update as being like upgrading your PC OS from Windows 95 to Windows 98. You have to format the drive and clear all the old stuff off or have the upgrade disk which will 'convert' Win95 to Win98. Just the same at LACC, so you cannot operate the NERC system with that going on, you cannot use the workstations at all. What happens is that LACC is disconnected from NAS (the HCS) which means we go manual, passing estimates by phone etc, telephone co-ordination as well (like ye goode olde days). Then the whole operation is moved out of the main Ops room into the Training & Development Unit (our simulator effectively). Then the Ops room is handed over to the Enjineers to fiddle about and change the wiggly amp bits. When they've done all that, reloaded the new software and ensured its working we then check it out. When we're happy we move the whole operation back into the main Ops room and reconnect to NAS so we can become electronic again.. .Because of moving rooms etc. flow is imposed so that we're not handling too much traffic during the moves. Last Wednesday we got into the TDU at 2315 and were back in Ops at about 0200, which is the fastest we've done it (my team has been training for this over the past 18 months).. .. .I don't know why Flow don't publish a warning, flow's a bit of a black art to me! Possibly because its all new to the staff and they haven't yet taken on board the full implications.. .. .As to the staffing issue, no idea when things will get better. Currently it looks like there will be more people leaving (early retirement etc.) than there are replacements available! There are a lot of ex-cadets from the college who've completed their ATCO courses and have been at LACC for months & months waiting to train on sectors. Obviously with the move here and unfamiliarity with the kit etc. there has been no On the Job Training (OJT)for months. When OJT will commence and how many people will actually validate is a matter for the Gods <img border="0" title="" alt="[Frown]" src="frown.gif" /> So although Target Sector Flow rates are almost all back to previous levels, over the summer you can expect more delays. <img border="0" title="" alt="[Frown]" src="frown.gif" /> <img border="0" title="" alt="[Frown]" src="frown.gif" /> . .Forewarned is forearmed I suppose. <img border="0" title="" alt="[Frown]" src="frown.gif" />

[ghost-rider]

Take3Call5,. .. .Many thanks for the informative answer. . .. .Scary stuff ! <img border="0" title="" alt="[Eek!]" src="eek.gif" /> You'd think there was a less painfull way of doing these things this day and age ! <img border="0" title="" alt="[Frown]" src="frown.gif" /> . .. .Think I'd better warn the grown-ups about this ! <img border="0" title="" alt="[Roll Eyes]" src="rolleyes.gif" /> . .. .Thanks again and good luck !

[ZIP250]

The planned system (several years ago) was to have four different methods of changing software. They are known as "cutovers" and had various degrees of transparency to users. In the interests of expediency (yes there was some during the protracted design of Swanwick) the three cutovers below the "shut it all down and reload" one were all delayed until after "O" date. At that time we (the end users) believed that such improvements would happen as the system developed. With new owners who are apparently skint it seems unlikely that there will be any changes which are not already paid for unless an overwhelming financial case can be made.. .. .So guys over to you. Tell NATS that this is unacceptable on a monthly basis and hopefully the wedge will arrive. It could be the case that other levels of cutover are already planned and paid for but, call me a cynic, I have me doubts. Perhaps one of those in the know could enlighten us all.. .. .Z

[JuicyLucy]

There was a warning (NOTAM A3308 from memory) issued at @13:00 that day - but not only did it not give the possible delays incurred, only traffic restrictions MAY be applied, but it also did not seem to arrive at many units!. .Even the flow message at 20:30 did not tell us that massive restrictions were to start at 21:00..... .Seems the biggest lesson to be learnt is to tell the users whats going on in advance and let us plan around it.

[5milesbaby]

Now believe next DD&C (????) AIRAC update is 17th April <img border="0" title="" alt="[Frown]" src="frown.gif" /> <img border="0" title="" alt="[Frown]" src="frown.gif" /> <img border="0" title="" alt="[Frown]" src="frown.gif" /> <img border="0" title="" alt="[Frown]" src="frown.gif" />

[Check your speed with....]

For anybody flying domestic routes via the manchester area at such times, I would suggest re-filing with a cruising level below FL200. This will keep you out of NERC's airspace and therefore you won't be subject to those particular flow restictions. It will make Manchester and TC a bit busier, but we should be able to cope!. . . . <small>[ 25 March 2002, 20:41: Message edited by: Check your speed with.... ]</small>