U.K. NATS Systems Failure
Alba Gu Brath
Join Date: Oct 2000
Location: Merseyside
Age: 55
Posts: 738
Likes: 0
Received 0 Likes
on
0 Posts
Could I just briefly drift this thread, and ask how a computer system can cost 100 million - or even 10 million for that matter? The costs blow my mind, but I am curious, rather than being Mr Angry of Crawley here.
Can anybody with industrial IT experience give us rough idea of the itemised costs of hardware, software, programming, development, testing etc ? And also the process that is followed in designing and commissioning a new system?
Can anybody with industrial IT experience give us rough idea of the itemised costs of hardware, software, programming, development, testing etc ? And also the process that is followed in designing and commissioning a new system?
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
Take a look at the UKRI Future of Flight Competition for some really good IP generation in this area.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
It’s the ‘£90m out of £100m that is being wasted’ which shows you how dire the situation is; there is literally no accountability (as the Board ‘can’t even spell IT’!) so it just degenerates from there.
Join Date: Oct 2004
Location: Southern England
Posts: 481
Likes: 0
Received 0 Likes
on
0 Posts
And there are few fewer military controllers world wide than there were then.
Join Date: Oct 2004
Location: Southern England
Posts: 481
Likes: 0
Received 0 Likes
on
0 Posts
It’s not the £10m-100m price tag the incumbents need to ‘get out bed’, that’s being spent every year as has been said.
It’s the ‘£90m out of £100m that is being wasted’ which shows you how dire the situation is; there is literally no accountability (as the Board ‘can’t even spell IT’!) so it just degenerates from there.
It’s the ‘£90m out of £100m that is being wasted’ which shows you how dire the situation is; there is literally no accountability (as the Board ‘can’t even spell IT’!) so it just degenerates from there.
It's actually difficult to know how much is "wasted", there are 4 main ANSP groupings buying replacement systems at the moment and the systems being developed appear to be costing about the same with the same very protracted timescales. You would expect NATS to need to spend more than the other ANSPs in its grouping as it missed out at least 2 upgrade cycles since the 1990s and the change is much bigger but the application software will cost about the same.. A few ANSPs including NATS need the bells and whistles in the latest iteration to handle the levels of traffic and complexity. If you can cope with the previous generation of controller tools etc then you will pay much less, even as much as 95% less.
If all the suppliers say that the price is £100M then that's the price. If the suppliers know how much you have to spend, a flaw of the existing "transparency", it is hardly a coincidence if the price is set accordingly.
NATS has tried a few times to escape the clutches of those suppliers but at the onset of the pandemic was more reliant on them that at any time before. That situation will return if it hasn't already.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
About 150 posts ago I made a comment that it's difficult to assess the effectiveness of the NATS Investment but it's kept a lot of people employed. Most of those weren't in NATS but generating profit for a handful of large companies.
It's actually difficult to know how much is "wasted", there are 4 main ANSP groupings buying replacement systems at the moment and the systems being developed appear to be costing about the same with the same very protracted timescales. You would expect NATS to need to spend more than the other ANSPs in its grouping as it missed out at least 2 upgrade cycles since the 1990s and the change is much bigger but the application software will cost about the same.. A few ANSPs including NATS need the bells and whistles in the latest iteration to handle the levels of traffic and complexity. If you can cope with the previous generation of controller tools etc then you will pay much less, even as much as 95% less.
If all the suppliers say that the price is £100M then that's the price. If the suppliers know how much you have to spend, a flaw of the existing "transparency", it is hardly a coincidence if the price is set accordingly.
NATS has tried a few times to escape the clutches of those suppliers but at the onset of the pandemic was more reliant on them that at any time before. That situation will return if it hasn't already.
It's actually difficult to know how much is "wasted", there are 4 main ANSP groupings buying replacement systems at the moment and the systems being developed appear to be costing about the same with the same very protracted timescales. You would expect NATS to need to spend more than the other ANSPs in its grouping as it missed out at least 2 upgrade cycles since the 1990s and the change is much bigger but the application software will cost about the same.. A few ANSPs including NATS need the bells and whistles in the latest iteration to handle the levels of traffic and complexity. If you can cope with the previous generation of controller tools etc then you will pay much less, even as much as 95% less.
If all the suppliers say that the price is £100M then that's the price. If the suppliers know how much you have to spend, a flaw of the existing "transparency", it is hardly a coincidence if the price is set accordingly.
NATS has tried a few times to escape the clutches of those suppliers but at the onset of the pandemic was more reliant on them that at any time before. That situation will return if it hasn't already.
Four suppliers with a quarter of the market each is a technical monopoly (iaw the CMA, but not on NATS 100% scale, so they’re unlikely to point it out for fear of allegations of hypocrisy). But it doesn’t matter if several of them say the price tag’s £XXm (that’s easy to set up offline), the correct question is what’s the better way of architecting the solution - but as you say, the sector will refuse to have that conversation for fear of having to do something they haven’t done numerous times already.
Join Date: Oct 2004
Location: Southern England
Posts: 481
Likes: 0
Received 0 Likes
on
0 Posts
That’s another brilliant spin, but entirely wrong; 90% of the project in question didn’t need to be done and NATS was advised accordingly, but it went ahead anyway because of one employee’s KIP and bonus.
Four suppliers with a quarter of the market each is a technical monopoly (iaw the CMA, but not on NATS 100% scale, so they’re unlikely to point it out for fear of allegations of hypocrisy). But it doesn’t matter if several of them say the price tag’s £XXm (that’s easy to set up offline), the correct question is what’s the better way of architecting the solution - but as you say, the sector will refuse to have that conversation for fear of having to do something they haven’t done numerous times already.
Four suppliers with a quarter of the market each is a technical monopoly (iaw the CMA, but not on NATS 100% scale, so they’re unlikely to point it out for fear of allegations of hypocrisy). But it doesn’t matter if several of them say the price tag’s £XXm (that’s easy to set up offline), the correct question is what’s the better way of architecting the solution - but as you say, the sector will refuse to have that conversation for fear of having to do something they haven’t done numerous times already.
You are correct that the regulatory regime encourages spending but at some point you need to deliver and start depreciating that cost and if your investment doesn't deliver the benefit promised there will be a storm at the next licence renewal. Potentially you could end up with a big revenue outgoing that isn't funded by the revenue stream.
ATM System supply may be a near monopoly but it's one that has arisen from withdrawal from the market rather than dominant companies buying up competitors so it's hard to see how the CMA or anybody else can act to stop it. NATS at least had a choice because its previous supplier isn't in the market, if you are already bought into a family, think Apple vs Android (NATS was on Blackberry), there is less of a free choice.
I think I have agreed with your last point, we probably only disagree on the practicality of changing that.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
I was talking in general terms. You may have personal knowledge of a particular project, I can probably throw in a few myself. Maybe you can apply that to the entire capital investment plan but I'm not so sure.
You are correct that the regulatory regime encourages spending but at some point you need to deliver and start depreciating that cost and if your investment doesn't deliver the benefit promised there will be a storm at the next licence renewal. Potentially you could end up with a big revenue outgoing that isn't funded by the revenue stream.
ATM System supply may be a near monopoly but it's one that has arisen from withdrawal from the market rather than dominant companies buying up competitors so it's hard to see how the CMA or anybody else can act to stop it. NATS at least had a choice because its previous supplier isn't in the market, if you are already bought into a family, think Apple vs Android (NATS was on Blackberry), there is less of a free choice.
I think I have agreed with your last point, we probably only disagree on the practicality of changing that.
You are correct that the regulatory regime encourages spending but at some point you need to deliver and start depreciating that cost and if your investment doesn't deliver the benefit promised there will be a storm at the next licence renewal. Potentially you could end up with a big revenue outgoing that isn't funded by the revenue stream.
ATM System supply may be a near monopoly but it's one that has arisen from withdrawal from the market rather than dominant companies buying up competitors so it's hard to see how the CMA or anybody else can act to stop it. NATS at least had a choice because its previous supplier isn't in the market, if you are already bought into a family, think Apple vs Android (NATS was on Blackberry), there is less of a free choice.
I think I have agreed with your last point, we probably only disagree on the practicality of changing that.
Join Date: Oct 2004
Location: Southern England
Posts: 481
Likes: 0
Received 0 Likes
on
0 Posts
It certainly happens on the edges. Indra have hoovered up all sorts of small suppliers, but the main ATM system suppliers were all big beasts not small independents and some have just withdrawn.
So, can anybody explain the proportions of IT costs, hardware, software, etc.?
If companies tendering for the IT contract cannot provide an explanation of costs, that would seem to be a problem right there - how can managers select the best tender if all they have to go on is one single number?
Surely there mist be a tendering process, with submissions including detailed plans and itemised costs?
If companies tendering for the IT contract cannot provide an explanation of costs, that would seem to be a problem right there - how can managers select the best tender if all they have to go on is one single number?
Surely there mist be a tendering process, with submissions including detailed plans and itemised costs?
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
So, can anybody explain the proportions of IT costs, hardware, software, etc.?
If companies tendering for the IT contract cannot provide an explanation of costs, that would seem to be a problem right there - how can managers select the best tender if all they have to go on is one single number?
Surely there mist be a tendering process, with submissions including detailed plans and itemised costs?
If companies tendering for the IT contract cannot provide an explanation of costs, that would seem to be a problem right there - how can managers select the best tender if all they have to go on is one single number?
Surely there mist be a tendering process, with submissions including detailed plans and itemised costs?
if you ask ‘how much does a radar cost’ the answer is likely to be ‘what’s your budget?’. Moreover, if you say £XXm you can expect an invoice for £XXm, then half way through the project a ‘change order’ for £Xm for this, £Xm for that (because you didn’t say explicitly that you wanted monitors to track the aircraft on or logistic support or some such nonsense). (In reality it’s a bit more nuanced than that, but this is exactly the game the suppliers play, and also why you HAVE to be an intelligent customer’ (not just a ‘supply chain manager’ who chooses suppliers according to how much they wine and dine you).
And that’s (relatively easy) hardware, because a) you should know roughly what you need - or ask your ATCOs/engineers, and b) just talk to other ANSPs ‘on the circuit’ and ask what they’re paying.
They say ‘hardware’s hard’ (that means to make btw, not to buy), but software’s considerably harder. You can’t see it, really, you don’t know if it’s missing vital components like fallbacks (as has been the problem with this ANSP - and do you believe the salesman even when he says it does?!), and if you try to specify everything you’re back into the £Xm change request trap again, (even if you’ve only been ‘got’ because systems have updated, names have changed, or new functionality has to be covered etc).
Grim, isn’t it?
Especially when you hear that some of the big players have backed out of the market (not enough customers and deals take too long), and when you get into DD it sometimes seems like each supplier is as bad as each other.
The only one route out is to have trusted advisers and suppliers, often operating on semi open book/agreed margin bases. But if you have (incompetent) buyers you can’t trust you’ll get (and deserve!) suppliers you can’t trust, and that is where we are. And btw, don’t believe all the nonsense about everyone struggling on both sides at the top end of this market - take a look at their cars (yachts and mansion houses!), no one wants to break up this cosy little arrangement.
Join Date: Oct 2004
Location: Southern England
Posts: 481
Likes: 0
Received 0 Likes
on
0 Posts
Just for the avoidance of doubt the system that failed last week was not a £100M system. The budget will have been much more modest than that.
I don't know the tendering arrangements. As a technology refresh of an existing system single tender might have been appropriate, or it might have been widely tendered.
They may have been living the high life at NATS expense but as they were bought out of receivership around that time I think it's unlikely.
The hardware requirements for these systems are quite modest & it's possible they were a separate tender. Even pre Brexit buying hardware from the UK shipping it to another country for integration & then shipping it back was a nightmare. I shudder to think what it's like now.
The amount of fallback wouldn't have been hidden from NATS. Everybody would be very aware that there was no software redundancy. We will need to wait for the next report to find out why that was thought to be Ok.
I don't know the tendering arrangements. As a technology refresh of an existing system single tender might have been appropriate, or it might have been widely tendered.
They may have been living the high life at NATS expense but as they were bought out of receivership around that time I think it's unlikely.
The hardware requirements for these systems are quite modest & it's possible they were a separate tender. Even pre Brexit buying hardware from the UK shipping it to another country for integration & then shipping it back was a nightmare. I shudder to think what it's like now.
The amount of fallback wouldn't have been hidden from NATS. Everybody would be very aware that there was no software redundancy. We will need to wait for the next report to find out why that was thought to be Ok.
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
Just for the avoidance of doubt the system that failed last week was not a £100M system. The budget will have been much more modest than that.
I don't know the tendering arrangements. As a technology refresh of an existing system single tender might have been appropriate, or it might have been widely tendered.
They may have been living the high life at NATS expense but as they were bought out of receivership around that time I think it's unlikely.
The hardware requirements for these systems are quite modest & it's possible they were a separate tender. Even pre Brexit buying hardware from the UK shipping it to another country for integration & then shipping it back was a nightmare. I shudder to think what it's like now.
The amount of fallback wouldn't have been hidden from NATS. Everybody would be very aware that there was no software redundancy. We will need to wait for the next report to find out why that was thought to be Ok.
I don't know the tendering arrangements. As a technology refresh of an existing system single tender might have been appropriate, or it might have been widely tendered.
They may have been living the high life at NATS expense but as they were bought out of receivership around that time I think it's unlikely.
The hardware requirements for these systems are quite modest & it's possible they were a separate tender. Even pre Brexit buying hardware from the UK shipping it to another country for integration & then shipping it back was a nightmare. I shudder to think what it's like now.
The amount of fallback wouldn't have been hidden from NATS. Everybody would be very aware that there was no software redundancy. We will need to wait for the next report to find out why that was thought to be Ok.
Who went into receivership? I was talking about NATS seniors and suppliers generally, you surely know the joke about why they chose Whiteley for NATS?
Join Date: Nov 2018
Location: UK
Posts: 82
Likes: 0
Received 0 Likes
on
0 Posts
Join Date: Jul 2014
Location: UK
Posts: 116
Likes: 0
Received 0 Likes
on
0 Posts
I'm increasingly baffled by the 'duplicate waypoint name' issue. This suggests - it's surely impossible - that waypoints are known to NATS as strings of characters. Surely each waypoint has a globally unique identifier or key? The flightplan has to mean something to the personnel, so they can select 'INGOR, ANNET, NAKID...' or whatever, but behind the scenes each of those should be a unique id. What am I missing?
I also have no idea why the system was allowed to get into a state where it decided it was utterly untrustworthy and collapsed. Planes have multiple software paths so that a rogue path can be outvoted by the other two. Why didn't the NATS system look at its own performance against all the other flights it was handling, and make at least an interim choice to keep going while it flagged up the anomaly?
I also have no idea why the system was allowed to get into a state where it decided it was utterly untrustworthy and collapsed. Planes have multiple software paths so that a rogue path can be outvoted by the other two. Why didn't the NATS system look at its own performance against all the other flights it was handling, and make at least an interim choice to keep going while it flagged up the anomaly?
Join Date: Oct 2004
Location: Southern England
Posts: 481
Likes: 0
Received 0 Likes
on
0 Posts
With regard to Swanwick that isn't totally a joke. The chosen location had to be acceptable to the ATCO workforce. Several of the candidate locations definitely weren't. It's fair to say similar considerations did not apply to the Technical workforce when Whiteley was chosen.
Join Date: Oct 2004
Location: Southern England
Posts: 481
Likes: 0
Received 0 Likes
on
0 Posts
I'm increasingly baffled by the 'duplicate waypoint name' issue. This suggests - it's surely impossible - that waypoints are known to NATS as strings of characters. Surely each waypoint has a globally unique identifier or key? The flightplan has to mean something to the personnel, so they can select 'INGOR, ANNET, NAKID...' or whatever, but behind the scenes each of those should be a unique id. What am I missing?
I also have no idea why the system was allowed to get into a state where it decided it was utterly untrustworthy and collapsed. Planes have multiple software paths so that a rogue path can be outvoted by the other two. Why didn't the NATS system look at its own performance against all the other flights it was handling, and make at least an interim choice to keep going while it flagged up the anomaly?
I also have no idea why the system was allowed to get into a state where it decided it was utterly untrustworthy and collapsed. Planes have multiple software paths so that a rogue path can be outvoted by the other two. Why didn't the NATS system look at its own performance against all the other flights it was handling, and make at least an interim choice to keep going while it flagged up the anomaly?
What it did then was fortunately safe but I, and I suspect most FDP staff, would not expect an issue with a single plan to result in an exception that brought the whole system down. We don't know whether that was a design decision, an oversight in design for an error not foreseen, or a coding error that meant it didn't do what the designer intended. We are going to have to wait a while to get the answer to that. If it was intended then I would hope they expected to resolve that issue much quicker than they did. Again we are going to have to wait a while for answers in that area.