Statement from AIS re site outage
Phil Bate, Manager UK AIS sent me the following email on Friday evening (11 Oct). Unfortunately it was to my office email so I didn't get to see it until this morning.
Mike
Mike/Russell,
You will no doubt be aware that the web has been off-line today. This began
last night (Thurs) when the BAA had a programmed power interruption to the
Control Tower. This of course should not have caused any difficulties as
absolutely everything here is protected by back-up generators and UPS.
Although we have a suspicion that an individual UPS may have allowed a 'grey
spike' to encroach, we do not as yet know the exact problem but suffice to
say that there was a difficulty with the 'Replica Server' which as you may
recall lies between the ADIMS database and the web servers, protected by a
firewall on both sides. Having struggled locally for some time to get
things going, the supplier was informed and asked to provide support under
the warranty agreement. They have done this remotely for most of the day
with no success (partly due to the efficiency of the firewalls) and decided
a few minutes ago to call it a day. They are flying a team over early
tomorrow morning and we are also going to be in attendance. NATS has made
it very clear that we cannot run a 24/7 service with this level of
instability and that we must at all costs achieve a stable environment from
now on (which in software term we thought we had done up to last night).
Please can I ask you to post something on the various sites explaining that
we really are trying 110% to sort this out and we apologise for the
interruption of service but we are now in the hands of our supplier (who to
be fair is also showing the fullest commitment). We will be staffed to
provide support to those requiring a briefing on the usual number (020 8745
3464 - If busy try 3450 or Fax. 3453) and ask that pilots also try their
'local' international airport flight briefing unit, who should be able to
supply information from the bulletins (A1/A8 etc. - I've heard of them
before!). We will also be faxing and e-mailing pre-prepared briefs to
clubs/aerodromes requesting them.
I can't tell you just how devastating a blow this is to my staff who were
just beginning to look forward to some stability, a reduction in pressure
and the next phase of development. Instead they are now having to bear some
serious individual abuse, which of course is terribly unfair but I guess
understandable. They will endeavour to maintain their cool and will as
always give the fullest support to our customers. We do record the
telephone lines and will use this if we need to in extreme cases of abuse.
I believe that I have the highest quality staff and I have unfortunately
just recently had to accept a resignation and I don't need more of that as
they are almost impossible to replace.
We will be sending an advisory notice via AFTN and FAX.
On other topics, when I get a minute I will talk to you both about the way
forward as I think that I have identified a route which might interest you
and others, also with the support of our Regulator and SRG.
Update to above, sent on Friday at 21:17
Mike
Mike/Russell,
I have just been informed that our engineers here have found the problem and
fixed it. The Web is stable, updating and operating successfully and we
have put the French team on hold. You might still want to place something
on the forums for me though. I am very grateful to the NATS staff involved
in this, who have demonstrated exceptional commitment and ability. We WILL
get there in the end despite some very testing times and our promise to
provide the best possible service is not a hollow one.
Thanks for your help in this.
Regards,
Phil