My two pennyworth ...... its no one organisation that is at fault.
I recall (maybe an urban legend) that the Chairman of BAE in the late 90's / early noughties, told a Parliamentary committee that if MOD knew the true cost of what they had asked for, they would never be able to afford anything.
Sadly, as a country we are just rubbish at defining and purchasing things. This is not unique to defence as rail, transport, government are just as bad.
No real insight into why this may be, but just opinions.
Personally, I think it comes down to an overweening British desire to reinvent the wheel every time rather than make do and have 90% of what we want, now. Instead we try and beat the Americans capabilities with something that will only be available in a decades time, might not work and will cost billions more than the original estimate. This desire is not unique to the armed forces.
And then be obsolete when/if it is ever finished.
Is it any wonder that the Bowman system because known as Better Off With Map And Nokia
