Chaos in Scotland [Archive]

Davey Clark

14th Dec 2006, 09:15

This was a planned power test that went horribly wrong!! The power to the Scottish Domestic operations room (and probably the Shanwick Oceanic Ops room) was completely lost, which resulted in loss of all main systems for about twenty minutes.

Only thing working was the secondary standby R/T system, which saved the day. Radar, phones, R/T and standby R/T all out for the duration. Not the most relaxing place to be at 2115 last night!!

Do airlines ever do power systems tests on aircraft when they are in flight with fare-paying passengers onboard????? - I think not!!! :ugh:

Wings level, Ball in the puddle!

MancBoy

14th Dec 2006, 09:34

Davey, you're brave announcing that to the world!!

Plus, chaos in Scotland, should be titled chaos over the whole of central England caused by some muppets decision somewhere!

How many supplies does the Scottish centre have anyway and what about the backup batteries?

HEATHROW DIRECTOR

14th Dec 2006, 10:05

Electronic Flight Progress Strips anyone.............................?

London Mil

14th Dec 2006, 10:26

This was a planned power test that went horribly wrong!!

Now when I was a young sprog, some hairy ar$ed flight lieutenant taught me a valuable lesson:

a. Never agree to a standby power supply test on your watch.

b. If you have no choice, insist that it takes place at 0300 on a Sunday morning.

rab-k

14th Dec 2006, 13:26

insist that it takes place at 0300 on a Sunday morning.

Bu**er off! Some of us are getting rather busy at 0300, on any given morning :E

London Mil

14th Dec 2006, 13:53

Ok, apart from those who are playing around with the North Atlantic. You can have you power run just after breakfast. ;)

Radarspod

14th Dec 2006, 16:04

It's been kept rather quite down South....nothing on Natsnet, no gossip, nothing.....took a phone call from a narked ScATCC systems engineer to even know something had happened.

MancBoy

14th Dec 2006, 16:25

Probably kept quiet because it was a cock up as Davey announced above.

BDiONU

14th Dec 2006, 18:16

Probably kept quiet because it was a cock up as Davey announced above.
Must have been a humungeous huge cock up to have switched off the mains, the UPS and the generators all at the same time :rolleyes:

BD

vintage ATCO

14th Dec 2006, 18:19

Electronic Flight Progress Strips anyone.............................?

Yes please, they work fine where I am . . . ;) :)

Sudden Stop

14th Dec 2006, 18:21

a. Never agree to a standby power supply test on your watch.

They still teach this at college. :ok:

A I

14th Dec 2006, 18:38

I saw the Scottish GM at CTC yesterday! I bet he went north sharpish!

This is the failure which is supposed never to happen. Because the OES cannot be relied upon, all NATS critical systems are supplied from uniterruptible power supplies. ie the OES keeps the batteries charged and they feed inverters which power the equipment for a certain length of time. This time is determined as being sufficient to safely clear the skies of all traffic in the event that the generators fail to start which would keep the batteries charged until the OES comes back.

Obviously something didn't work very well!

A I :confused: :confused: :{

Nogbad the Bad

14th Dec 2006, 19:56

Wait for it....

"A NATS spokesperson said safety was not compromised "

Coming to a comic strip near you, soon :}

Davey Clark

14th Dec 2006, 21:23

I think that radarspod has hit the nail on the head. The silence is deafening up north of the border. Nobody is even talking about it above shop floor level. Nothing on NATS intranet site, no "Well done chaps, back for tea and medals", nothing at all. How strange. Least said soonest mended??? :mad: Once again, I think not!!

Almost as quiet as the airwaves last evening!!

Wings level, ball in the puddle.

SM4 Pirate

14th Dec 2006, 22:43

Do airlines ever do power systems tests on aircraft when they are in flight with fare-paying passengers onboard????? - I think not!!! :ugh:
Fair point, but when exactly is the exact moment when the whole centre/facility doesn't have something airborne... The modern world makes these things more concerning; before 'consolidation' a failure in one unit was a small but significant event, now it's a catsorophy as it is so wide spread.

As for the electronic strips thing, upon failure here we have dumped the Flight plan data base onto A4 paper, distributed relevent copies to each and every console and had our own A4 "strips"... This of course assumes the power to the printers/photo copiers and flight plan box is working.

I think the more that software gets a hold in our world, every 'upgrade' is considered a very hi risk; cause you never really know what the end result will be until it's switched back on.

London Mil

15th Dec 2006, 05:52

No point in having any sort of flight strip if your radios aren't working. :hmm:

threemiles

15th Dec 2006, 06:55

safety compromised => file a report

HEATHROW DIRECTOR

15th Dec 2006, 07:03

<<No point in having any sort of flight strip if your radios aren't working. >>

Yep.... I expected that sort of response from a bloke in a blue suit.

BDiONU

15th Dec 2006, 07:05

safety compromised => file a report
Yes, gotta file. Undoubtedly several different errors involved in this one, yea olde 'chain of events'. Possibly a failure in the Risk assessment which no one who reviewed and signed it off noticed, maybe an additional equipment failure, possibly some one did something out of sequence which caused the whole thing to collapse (although that should have been identified and mitigated in the RA). In any case there will be a major engineering investigation but NATS is a 'no blame' company. Hhhhhmmm.

BD

London Mil

15th Dec 2006, 07:10

<<No point in having any sort of flight strip if your radios aren't working. >>
Yep.... I expected that sort of response from a bloke in a blue suit.

Yes, we tend to think on simple terms. ;)

PS. As a customer, do we get any money back because NATS failed to meet the terms of our Contract or would such an outage be covered in any fault allowance?

BDiONU

15th Dec 2006, 07:23

As a customer, do we get any money back because NATS failed to meet the terms of our Contract or would such an outage be covered in any fault allowance?
MOD have always received money back for outages and the new contract is the same. In simple terms the amount depends on what it is (a single frequency would not 'earn' as much as a radar or a workstation) and how long the outage lasted. The longer the outage the more the costs (to NATS) ratchet (increase exponentially) i.e lets say a frequency was out for 5 minutes and you got £10 a minute. For the first 5 mins you'd get £50, for the 2nd 5 mins you'd get £100 (plus the 1st 5 mins £50), for the 3rd 5 mins you'd get £200 (plus £100 and £50) etc. Thats a very simplistic answer as the ratchet mechanism is detailed over pages and pages in the contract. As is how its paid etc. etc.

BD

TATC

15th Dec 2006, 20:30

just read that the outage caused less than 2000 minutes of delay - wow less than 33 hours. I know that this is total delays but I wonder how many flights are handled during the night in the scottish TMA and had such an incident happened inthe morning how much delay would have been caused.

MrJones

15th Dec 2006, 21:17

This was a serious event. NATS spends a small fortune on making their power supplies uninterruptible and for such a failure to occur is alarming.

Sure it was planned maintenance and at a fairly quiet time but imagine if it had been a real power failure at LACC or LTCC on a Monday morning and the back-up hadn't worked.

I don't think it's a question of back to the drawing board but no matter what it costs it has to be sorted.

Davey Clark

16th Dec 2006, 09:56

I have to agree with Mr Jones.This was a very serious event, particularly for the operational staff who had the misfortune to be on duty at the time, and again, particularly for the ATCOs who were on radar when it all went blank.

It appears to me that NATS seem to be trying to dumb-down this incident and sweep it under the carpet. (Maybe tarnish their shiny new logo:) )

The NATS intranet site finally admitted to the incident two days after the event, and described it as a power interuption where a few systems were lost - no mention that these were live operational systems.

Also worthy of note was that the report only contained a few lines, and that the page counter showed that less than 100 people had viewed the page. Another (different) page also released onto the site that day had well over 1000 hits. So at what time on Friday was this page put up onto the intranet??? Maybe after most staff had gone home for the weekend?

Worse than Tony B and his "good day to hide bad news" methinks, or am I just TOO cynical?

Wings level, ball in the puddle

PH-UKU

16th Dec 2006, 11:02

I have to agree with Mr Jones.This was a very serious event, particularly for the operational staff who had the misfortune to be on duty at the time, and again, particularly for the ATCOs who were on radar when it all went blank.
... snip snip snip ....#

Of course it could NEVER happen. Funny, is that not what was said the last time it 'never happened' .... but my memory is failing me ....... "nurse.. Nurse ... NURSE !! ... is it time for my pills again .. ?"

Well done to everyone who recovered from what might have been a very very bad situation. I think they should all get a huge Christmas bonus, a couple of days off in Lieu, and a public "thank you ". Thank goodness there was no outage of NAS planned ....

However, I would love to know the background of whoever put up the intranet statement that said "safety was never compromised".......... not that I am suggesting it was, but it is a rather pointless statement particularly if read by someone who has just heard the air conditioning go quiet ...

Hmmmm - if that's the case ... why don't we just get rid of radar and serviceable frequencies ... I mean do we REALLY need them, as they cost soooooooo much ..... ?

Until you've been there and had it all go 'plink' .. you have no idea ....

Eric T Cartman

16th Dec 2006, 11:59

Way back in 1973 when I did my Area Procedural Rating oral board exam, the last question I was asked by the Chairman (the famous Malcolm Lucas) was " why do we bother teaching this stuff when we're going for an all radar setup at the Centres ?" I said something along the lines of "because nobody believes Tels Engineers who say it'll never fail !" which caused some mirth on the Board.
As the French say (rough translation) " the more things change, the more they stay the same!" :)

Minesapint

16th Dec 2006, 17:12

Anyone remember the power failure at LATCC (yes - LATCC) in 91 or 92? That was a cracker, the room was only lit by the emergency comms panels and the glow of panicking ATCO's. Took hours to return to normal.

supersicroly

17th Dec 2006, 09:52

I said something along the lines of "because nobody believes Tels Engineers who say it'll never fail !" which caused some mirth on the Board.

No Tels Engineer would ever say such a thing!

Everything fails -eventually.

Eric T Cartman

17th Dec 2006, 19:52

Quote:
No Tels Engineer would ever say such a thing!

Wrong - I guess you never met the engineer known as "Tandy Man" ! ;)

danceswithsheep

25th Dec 2006, 21:29

In Scottish 2 maybe 3 years ago when the same thing happened.......switched off mains to test standby..no problem..........switched off standby....oopps missed the page that said switch mains back on!!!!!!!!!
Could it be the engineers have those nasty channels back on the tele and have been making a mess of the manuals??????:=
ps.......sorry i missed that one, no power problems in the desert:ok:

Talla Radar

26th Dec 2006, 18:15

danceswithsheep

26th Dec 2006, 21:12

Then tell me what really happened....but for sure.....we were left on a saturday morning with nothing but fresh air and Taff trying to solve the problem in the back ground!!!!!!!

Scott Voigt

27th Dec 2006, 00:04

Why can't I get the visual out of my head, the engineer at Scotish looking like Marty Feldman in Young Frankenstien at the power lever <G>???

AGEDMIL

8th Jan 2007, 03:56

It was a very serious event. Historically I do remember a JCB digging through the BT lines near Kilmarnock when ScACC/ScATCC(Mil) lost all radars and frequencies because BT failed to route the 2 feeds through different BT lines! :eek: In this instance the change from mains to standby power failed due to a faulty switch - which triggered a shutdown (a very non-technical statement!). The resultant loss of all radars and R/T can only be termed a catastrophic failure. When D and D were approached for possible use of 121.5 MHz to pass urgent messages to aircraft - D and D stated that all the emergency frequencies were also lost. As stated by someone earlier - if this had happened during the busiest part of the day, the results could have been a lot worse. It should always be the case that safety is paramount, and the availability of an assured source of power to radars and frequencies must be guaranteed.:ugh: