Chinook - Still Hitting Back 3 (Merged)
Join Date: Nov 2005
Location: Norfolk England
Posts: 247
Likes: 0
Received 0 Likes
on
0 Posts
FADEC Software
AA,
As another reminder it was not just Boscombe Down - in 1993 MOD asked EDS-Scicon as an independent and expert company for an evaluation of the FADEC software, which had been substantially re-written after the Wimington incident (in which you may recall an engine runaway up occurred whch took the rotors to over 140%) - their conclusion was that the "new" software was "poorly developed code [which] is significantly more difficult to analyse than code written and documented to a well-defined standard." These comments were made after EDS had apparently found 485 anomalies in the first 18% of the code they analysed - at which point work was stopped. In their final report dated July 1993 these anomalies were broken down as follows:
Primary Lane Cat 1 (real error in the code or a discrepancy between the code and documentation) 13; Cat 2 (poor code or poor correspondence between code and documentation but it is likely that it performs the intended function) 111; Cat 3 (obvious documentation errors) 42: Cat 4 (poor coding style) 81. The figures for the Reversionary Lane were Cat 1 - 8; Cat 2 - 43; Cat 3 - 75; Cat 4 - 20. The figures for Documentation Tracebility were Cat 1 - 35; Cat 2 - 39; Cat 3 - 75; Cat 4 - 3.
AD/HP1 met Textron Lycoming (T/L) on 21 Jan 1994 to "discuss the way forward with Chinook HC MKII FADEC software". A secondary aim of the meeting was to "discuss the longer term options for revisions to the software which would enable the UK to satisfactorily complete the CA Release process". Several significant actions were agreed at the meeting, but despite the obvious problems there is a footnote which says "At the end of the day the T/L meeting did not alter DHP's current view of the FADEC, which is that we have no positive eveidence of a software safety problem that would critically affect airworthiness, but that it would be prudent to observe the restrictions recommended by A&AEE in their Interim CA Release report until our present investigations and additional studies are concluded."
Like everyone else concerned with the injustice of this verdict I make no claims that FADEC software failings definitely led to this accident, but they, along with quite a few other airworthiness related issues, could have done. Given that actions agreed at the end of January did not change the FADEC software in use in June, what is sure is that the BoI should have looked at all these issues, and especially whether DHP's confidence was misplaced - and it certainly looks to me that it was. But then, as has been said many times before, the word "airworthiness" does not figure in their deliberations or theirs and the ROs' findings.
JB
As another reminder it was not just Boscombe Down - in 1993 MOD asked EDS-Scicon as an independent and expert company for an evaluation of the FADEC software, which had been substantially re-written after the Wimington incident (in which you may recall an engine runaway up occurred whch took the rotors to over 140%) - their conclusion was that the "new" software was "poorly developed code [which] is significantly more difficult to analyse than code written and documented to a well-defined standard." These comments were made after EDS had apparently found 485 anomalies in the first 18% of the code they analysed - at which point work was stopped. In their final report dated July 1993 these anomalies were broken down as follows:
Primary Lane Cat 1 (real error in the code or a discrepancy between the code and documentation) 13; Cat 2 (poor code or poor correspondence between code and documentation but it is likely that it performs the intended function) 111; Cat 3 (obvious documentation errors) 42: Cat 4 (poor coding style) 81. The figures for the Reversionary Lane were Cat 1 - 8; Cat 2 - 43; Cat 3 - 75; Cat 4 - 20. The figures for Documentation Tracebility were Cat 1 - 35; Cat 2 - 39; Cat 3 - 75; Cat 4 - 3.
AD/HP1 met Textron Lycoming (T/L) on 21 Jan 1994 to "discuss the way forward with Chinook HC MKII FADEC software". A secondary aim of the meeting was to "discuss the longer term options for revisions to the software which would enable the UK to satisfactorily complete the CA Release process". Several significant actions were agreed at the meeting, but despite the obvious problems there is a footnote which says "At the end of the day the T/L meeting did not alter DHP's current view of the FADEC, which is that we have no positive eveidence of a software safety problem that would critically affect airworthiness, but that it would be prudent to observe the restrictions recommended by A&AEE in their Interim CA Release report until our present investigations and additional studies are concluded."
Like everyone else concerned with the injustice of this verdict I make no claims that FADEC software failings definitely led to this accident, but they, along with quite a few other airworthiness related issues, could have done. Given that actions agreed at the end of January did not change the FADEC software in use in June, what is sure is that the BoI should have looked at all these issues, and especially whether DHP's confidence was misplaced - and it certainly looks to me that it was. But then, as has been said many times before, the word "airworthiness" does not figure in their deliberations or theirs and the ROs' findings.
JB
As Thor correctly quoted, Boscombe recommended a rewrite, having declared the implementation "positively dangerous".
ACAS signed his RTS less than a month later, in doing so declaring that all regulations had been complied with.
14 months later the AAIB stated the software in ZD576 was at its original issue (i.e. at the standard assessed and rejected by Boscombe).
This rather begs the question how the Certificate of Design for Safety Critical Software System/Subsystem (if one even exists) could be remotely valid given the Designer had to sign the Certificate to the effect it (software and host system) complied with laid down standards and that the "Requirement, Hazard Analysis, Design, methods employed and standard of work carried out, with a specific regard to safety, meet the satisfaction of the Independent Safety Auditor". The latter then had to sign the Certificate to confirm this. The PM then had to sign his acceptance. (See JB's post above - I can't imagine EDS-Scicon signing the Certificate). No Certificate, the equipment does not get in the PM's aircraft. End of.
Another good question is just who the Design Authority for the engine/FADEC was. This is not straight forward because at the time, for older engines, MoD (D/Eng) acted as DA, not the contractor. I wish I knew this detail because therein lies the answer to many questions on this subject. I have a distinct feeling this system changed during FADEC / new engine variant development (mid-1993 at a guess) and would not be surprised if the answer was confusion reigned supreme. But the stability and continuity was provided by Boscombe, which was all the more reason to listen to their advice.
So - Who signed the Certificate and who accepted the design, knowing Boscombe had condemned it as "positively dangerous" and recommended a complete re-write? It makes you wonder what pressure was applied.
All this may seem a bit of a bore, bureaucratic and unnecessary to some. Certainly, the staffs of the RAF Chief Engineer in 1991-94 thought it a complete waste of money. But to the guy with the signature, very often a relatively low grade, implementing these regulations properly is money in the bank. Verbal agreements or discussions do not stand the test when the chips are down, such as following an accident when certain types, who do not sign anything but have the "authority" to instruct you to ignore regulations, disappear into the woodwork with denials and different interpretations.
ACAS signed his RTS less than a month later, in doing so declaring that all regulations had been complied with.
14 months later the AAIB stated the software in ZD576 was at its original issue (i.e. at the standard assessed and rejected by Boscombe).
This rather begs the question how the Certificate of Design for Safety Critical Software System/Subsystem (if one even exists) could be remotely valid given the Designer had to sign the Certificate to the effect it (software and host system) complied with laid down standards and that the "Requirement, Hazard Analysis, Design, methods employed and standard of work carried out, with a specific regard to safety, meet the satisfaction of the Independent Safety Auditor". The latter then had to sign the Certificate to confirm this. The PM then had to sign his acceptance. (See JB's post above - I can't imagine EDS-Scicon signing the Certificate). No Certificate, the equipment does not get in the PM's aircraft. End of.
Another good question is just who the Design Authority for the engine/FADEC was. This is not straight forward because at the time, for older engines, MoD (D/Eng) acted as DA, not the contractor. I wish I knew this detail because therein lies the answer to many questions on this subject. I have a distinct feeling this system changed during FADEC / new engine variant development (mid-1993 at a guess) and would not be surprised if the answer was confusion reigned supreme. But the stability and continuity was provided by Boscombe, which was all the more reason to listen to their advice.
So - Who signed the Certificate and who accepted the design, knowing Boscombe had condemned it as "positively dangerous" and recommended a complete re-write? It makes you wonder what pressure was applied.
All this may seem a bit of a bore, bureaucratic and unnecessary to some. Certainly, the staffs of the RAF Chief Engineer in 1991-94 thought it a complete waste of money. But to the guy with the signature, very often a relatively low grade, implementing these regulations properly is money in the bank. Verbal agreements or discussions do not stand the test when the chips are down, such as following an accident when certain types, who do not sign anything but have the "authority" to instruct you to ignore regulations, disappear into the woodwork with denials and different interpretations.
Chinook 240
I'm afraid you're using the same rationale as Cazatou did earlier, in turn reflecting MoD's stance. That is, justifying safety decisions after the event.
"Don't waste money now, let's wait to see if there is an accident before deciding to do anything" is precisely what led to numerous accidents. This is a systemic failure and not just applicable to ZD576 or Chinook. It is what the Haddon-Cave report is all about.
Any Safety Management System, including MoD's, expressly forbids this method of justification so, in this case, Boscombe could only assess safety based on what was put in front of them. At a higher level, even if they had given it the nod "on the bench" they could not test it in a representative Chinook HC Mk2, because they did not have one. (An oft-ignored fact, largely because most don't understand the linkage between Build Standard, Safety Case and RTS). That is, for the RTS to be valid, it must be based on a valid Safety Case which reflects the Build Standard(s). It follows the Build Standard must be known at all times and maintained. THAT is the single biggest airworthiness failing in MoD over the last 25 years - as outlined by the Inspectorate of Flight Safety in his Chinook Airworthiness report of August 1992. As I said earlier, every single technical recommendation could be resolved by maintaining the Build Standard.
In the case of Chinook FADEC, there was no "read across" available from another user as our version was unique. (Despite Dr Reid implying otherwise). The regulations stated what had to be done, and they were ignored. Worse, MoD then lied for many years, claiming the software did not meet the criteria for Safety Critical, when it clearly did. They even lied about the definition of SCS.
What part of these regulations would you be willing to waive if YOU were required to sign for the airworthiness?
There have been two upgrades. I don't know when they were embodied, but neither was in ZD576 (AAIB statement).
I'm afraid you're using the same rationale as Cazatou did earlier, in turn reflecting MoD's stance. That is, justifying safety decisions after the event.
"Don't waste money now, let's wait to see if there is an accident before deciding to do anything" is precisely what led to numerous accidents. This is a systemic failure and not just applicable to ZD576 or Chinook. It is what the Haddon-Cave report is all about.
Any Safety Management System, including MoD's, expressly forbids this method of justification so, in this case, Boscombe could only assess safety based on what was put in front of them. At a higher level, even if they had given it the nod "on the bench" they could not test it in a representative Chinook HC Mk2, because they did not have one. (An oft-ignored fact, largely because most don't understand the linkage between Build Standard, Safety Case and RTS). That is, for the RTS to be valid, it must be based on a valid Safety Case which reflects the Build Standard(s). It follows the Build Standard must be known at all times and maintained. THAT is the single biggest airworthiness failing in MoD over the last 25 years - as outlined by the Inspectorate of Flight Safety in his Chinook Airworthiness report of August 1992. As I said earlier, every single technical recommendation could be resolved by maintaining the Build Standard.
In the case of Chinook FADEC, there was no "read across" available from another user as our version was unique. (Despite Dr Reid implying otherwise). The regulations stated what had to be done, and they were ignored. Worse, MoD then lied for many years, claiming the software did not meet the criteria for Safety Critical, when it clearly did. They even lied about the definition of SCS.
What part of these regulations would you be willing to waive if YOU were required to sign for the airworthiness?
There have been two upgrades. I don't know when they were embodied, but neither was in ZD576 (AAIB statement).
I'm afraid you're using the same rationale as Cazatou did earlier, in turn reflecting MoD's stance. That is, justifying safety decisions after the event.
My question is simple, the code hasn't been rewritten, if it was grossly unairworthy then, surely it must be now, you can't have it both ways.
We're still flying Block 1 software, admittedly not the Jun 94 standard, but not completely rewritten.
chinook240:
If the code being used has not been written, checked and cleared in accordance with the UK Military Airworthiness Regulations, then yes it is unairworthy. Even if gross errors that were in it at RTS, causing uncommanded power ups, downs and engine shut downs have been resolved, thus ensuring your 221,000 hours of presumably (?) no airworthiness related incidents, it would still be unairworthy. As tuc says, no accidents doesn't mean it is airworthy.
Is it still grossly unairworthy?
C240
You used the argument about time in service, in the same way MoD justify all their decisions with "There haven't been any more Chinook accidents, so it must have been safe in 1994". If that is not what you meant, it is better to avoid confusion and look at the situation as it was at the time. That is, the Chief Engineer and ACAS were sitting on a report telling them at least 5 crashes had been caused by airworthiness failings. THAT is the important fact and what ACAS had to consider in Nov 1993. These officers need to be asked what they did about it. (CDP already answered that for the major criticism, in 1999 - nothing).
Like I said, there were two upgrades. If you say they were never embodied, that is a slightly different thing and I of course accept your word. However, what later evidence demonstrated was that the major (FADEC) failings perceived at the time by Boscombe and RAF pilots were partly resolved as they came to understand their nature. Such failures during testing demand you stop and try to work them out. If you cannot explain them, then you cannot prepare, for example, AMs and FRCs that are fit for purpose; or determine what limitations to set out in the RTS. That is, the technical maturity, which includes understanding, is insufficient. That alone prevents a Certificate of Design being issued because you cannot declare the product compliant with the spec when it cannot be verified; but the project manager may decide to sign an Interim CoD to allow flight testing to proceed.
Anyone familiar with this process can see that, on 2nd June 1994, the Chinook HC Mk2 did not meet the mandated maturity levels (based on recommendations from the Chief Scientific Advisor and endorsed by SoS). On that day, the HC Mk2 was not airworthy. The programme status was that of an immature design in early testing, with Safety Critical Software not yet validated or verified, and no Comms or Nav systems with Full or even Limited clearance. (Largely because representative testing could only commence once the FADEC software issue had been resolved).
But, it can be seen that with no further modifications it may have become airworthy, as understanding matured. From papers I've read, this declaration was made in January 1996 with Issue 6 of the RTS. I hope that answers your question.
You used the argument about time in service, in the same way MoD justify all their decisions with "There haven't been any more Chinook accidents, so it must have been safe in 1994". If that is not what you meant, it is better to avoid confusion and look at the situation as it was at the time. That is, the Chief Engineer and ACAS were sitting on a report telling them at least 5 crashes had been caused by airworthiness failings. THAT is the important fact and what ACAS had to consider in Nov 1993. These officers need to be asked what they did about it. (CDP already answered that for the major criticism, in 1999 - nothing).
Like I said, there were two upgrades. If you say they were never embodied, that is a slightly different thing and I of course accept your word. However, what later evidence demonstrated was that the major (FADEC) failings perceived at the time by Boscombe and RAF pilots were partly resolved as they came to understand their nature. Such failures during testing demand you stop and try to work them out. If you cannot explain them, then you cannot prepare, for example, AMs and FRCs that are fit for purpose; or determine what limitations to set out in the RTS. That is, the technical maturity, which includes understanding, is insufficient. That alone prevents a Certificate of Design being issued because you cannot declare the product compliant with the spec when it cannot be verified; but the project manager may decide to sign an Interim CoD to allow flight testing to proceed.
Anyone familiar with this process can see that, on 2nd June 1994, the Chinook HC Mk2 did not meet the mandated maturity levels (based on recommendations from the Chief Scientific Advisor and endorsed by SoS). On that day, the HC Mk2 was not airworthy. The programme status was that of an immature design in early testing, with Safety Critical Software not yet validated or verified, and no Comms or Nav systems with Full or even Limited clearance. (Largely because representative testing could only commence once the FADEC software issue had been resolved).
But, it can be seen that with no further modifications it may have become airworthy, as understanding matured. From papers I've read, this declaration was made in January 1996 with Issue 6 of the RTS. I hope that answers your question.
Tuc and Chug,
Thank you both for the clarification. You will understand I am not using the benefit of hindsight to justify the decisions taken in 1994, sorry if that caused confusion. If you were referring to the Block 2 upgrade, then I don't believe they were.
Many early FADEC malfunctions were the definately result of incorrect procedures, which we now understand better and have amended our drills to compensate. But the BD recommendation to rewrite did not happen and no further verification work took place, as far as I'm aware.
As simple aircrew, the idea that understanding the design can change the status of an aircraft from unairworthy to airworthy seems counter-intuitive. Or am I confusing the terms 'airworthy' and 'safe'?
Thank you both for the clarification. You will understand I am not using the benefit of hindsight to justify the decisions taken in 1994, sorry if that caused confusion. If you were referring to the Block 2 upgrade, then I don't believe they were.
Many early FADEC malfunctions were the definately result of incorrect procedures, which we now understand better and have amended our drills to compensate. But the BD recommendation to rewrite did not happen and no further verification work took place, as far as I'm aware.
As simple aircrew, the idea that understanding the design can change the status of an aircraft from unairworthy to airworthy seems counter-intuitive. Or am I confusing the terms 'airworthy' and 'safe'?
Join Date: Oct 2000
Location: Retired to Bisley from the small African nation
Age: 67
Posts: 461
Likes: 0
Received 0 Likes
on
0 Posts
As ex-simple-aircrew, but with a legal education, I suggest that this is a bit like the difference between "not guilty" and "innocent". The law gives the benefit of the doubt to the accused (Chinook Mk2 safety). So it was "not guilty" because it could not be demonstrated that an airworthiness issue was the problem (so something/somebody else must be "guilty" - false logic but all too often used). As time progressed evidence assembles to the point that to a certain standard (the definition of which we can argue over) it is demonstrated that the accused is not at fault. Hence "innocent" or "airworthy" (equally invalid for reciprocal reasons).
Obviously, I do not believe this stands up. One of the big problems with safety critical software is precisely proving the negative - that there is nothing wrong with it. And aviation standards of engineering proof - 1 x 10-9 or whatever - are way beyond the legal standard of "reasonable doubt". But, critically in this case, they do not reach the required standard for the verdict of "no doubt whatsoever".
And so the Air Marshals' opinion fails.
Iain
Obviously, I do not believe this stands up. One of the big problems with safety critical software is precisely proving the negative - that there is nothing wrong with it. And aviation standards of engineering proof - 1 x 10-9 or whatever - are way beyond the legal standard of "reasonable doubt". But, critically in this case, they do not reach the required standard for the verdict of "no doubt whatsoever".
And so the Air Marshals' opinion fails.
Iain
Join Date: Jul 2000
Location: Nova
Posts: 1,242
Likes: 0
Received 0 Likes
on
0 Posts
Olive Oil
Interesting question (why do you ask...)
The real answer is simple. Since the RAF CHOSE not to fit a cockpit voice recorder, (despite a recommendation 8yrs earlier, after another largely unexplained chinook fatal accident!) how can we possibly know what their interaction was like on this particular flight...
That would simply be speculation wouldn't it...
what are we to do... Make assumptions... Would that be helpful...
But then the RAF already expect 'the boys' to bear the responsibility for the decision not to fit that equipment! So let's not cast aspersions eh!
Edited to add: Of course the same, previous BoI also recommended the fitment of DATA recorders. Which would have allowed us to KNOW exactly how this machine (utterly inadequate at that time!) performed in those vital seconds. The RAF CHOSE to deny Tapper and Cook that 'testimony' too, simply replacing 'fact' with 'convenient assumptions'! But in their final determination made absolutely no allowance for that omission either!
(Though to be fair, Wg Cdr Pulford and his two colleagues did, clearly stating it had been impossible to recreate the sequence of events leading up to the accident!)
Not long now. Let right be done.
While we are on the crew subject, would anyone who worked with them on the unit care to confirm the level of co-operation between the pilots involved? In other words, did they get on, were they mates, or was their relationship in the cockpit "difficult"?
The real answer is simple. Since the RAF CHOSE not to fit a cockpit voice recorder, (despite a recommendation 8yrs earlier, after another largely unexplained chinook fatal accident!) how can we possibly know what their interaction was like on this particular flight...
That would simply be speculation wouldn't it...
what are we to do... Make assumptions... Would that be helpful...
But then the RAF already expect 'the boys' to bear the responsibility for the decision not to fit that equipment! So let's not cast aspersions eh!
Edited to add: Of course the same, previous BoI also recommended the fitment of DATA recorders. Which would have allowed us to KNOW exactly how this machine (utterly inadequate at that time!) performed in those vital seconds. The RAF CHOSE to deny Tapper and Cook that 'testimony' too, simply replacing 'fact' with 'convenient assumptions'! But in their final determination made absolutely no allowance for that omission either!
(Though to be fair, Wg Cdr Pulford and his two colleagues did, clearly stating it had been impossible to recreate the sequence of events leading up to the accident!)
Not long now. Let right be done.
Last edited by Tandemrotor; 9th Jun 2011 at 11:23.
C240
Generally speaking, MoD standards are very good and, if adhered to, ensure an adequate level of safety, which allows the RTS to be signed.
I think the problem was, and remains, much more basic. Refusal to implement, primarily to save time and money. We’ve been discussing Safety Critical Software, and it is a simple exercise to create a compliance matrix of the FADEC programme Vs Def Stan 00-55 Pts 1 & 2. At the time “The Procurement of Safety Critical Software in Defence Equipment” (renamed “Requirements for Safety Related Software” in 1997).
That matrix would reveal key requirements were completely ignored. One could write a book, but the most obvious example on FADEC is the Boscombe recommendation to re-write the software. Over the years, MoD has sought to present this as Boscombe having a hissy fit.
But the reality was that the regulations required a Safety Plan which incorporated a Code of Design Practice. That had to include an agreed acceptance criteria and “the procedures for dealing with unacceptable components” (00-55, Part 2, para 20k).
If I may quote the next para;
“The acceptance criteria for SCS should distinguish between errors in Formal Arguments and discrepancies found during dynamic testing. The criteria should include the maximum number of errors found by the V&V team in Formal Arguments or during Static Path Analysis beyond which the item will be redeveloped from scratch”.
Given that last, and the sheer number of anomalies uncovered during testing (see JB’s #7866), Boscombe’s recommendation seems entirely reasonable and in line with mandated policy. What we don’t know, however, is what the criteria were. We do know that Boscombe offered an opinion as to what number of anomalies would be acceptable, which tends to indicate no formal criteria was laid down in the Safety Plan. THAT is a major failure on the part of both MoD and Design Authority.
In time, such failures could be corrected. The issue I have here is one of timescale and unseemly haste. FADEC had been in development for years. Yet, a mere 3 weeks before ACAS signed the RTS, Boscombe were still flagging serious deficiencies in the Safety Critical Software. What happened in that 3 week period to persuade ACAS that Boscombe and CA were wrong? MoD’s justification was cobbled together after the event, for the benefit of the Fatal Accident Inquiry. As I’ve said before, they tried to dismiss the issue by saying the software was not Safety Critical; when it clearly was, as defined by MoD policy.
The entire programme is littered with similar major failures and deceit. This is not being wise after the event. On airworthiness, prior warning had been given to both MoD(PE) and AMSO senior staffs between 1988 and 1993; and the 1992 CHART report is effectively a consolidation of those reports as applied to Chinook HC Mk1 and Mk2 (plus Puma and Wessex).
The above doesn’t seek to explain the crash, but illustrate the Organisational Faults that make it impossible to attribute sole blame to the pilots. I sympathise to a degree with those charged with delivering the SCS. At the time, and ever since, MoD’s attitude toward safety amounted to criminal negligence. If a project manager flagged a safety problem, he risked disciplinary action, especially from the RAF Chief Engineer’s non-technical staffs. What is really despicable is the deliberate lies to cover up these failings. The emergence (into the public domain) of the CHART report is vital to understanding all this. To their credit, current MoD staffs have declined to comment, perhaps realising how utterly damning the report is and how implementation would have probably prevented the need for Haddon-Cave. But the reaction of retired officers is revealing; one in particular who delights in denigrating the pilots in the press but will now hopefully be called to account for the failure to implement the report’s recommendations.
Tandemrotor - good post.
Generally speaking, MoD standards are very good and, if adhered to, ensure an adequate level of safety, which allows the RTS to be signed.
I think the problem was, and remains, much more basic. Refusal to implement, primarily to save time and money. We’ve been discussing Safety Critical Software, and it is a simple exercise to create a compliance matrix of the FADEC programme Vs Def Stan 00-55 Pts 1 & 2. At the time “The Procurement of Safety Critical Software in Defence Equipment” (renamed “Requirements for Safety Related Software” in 1997).
That matrix would reveal key requirements were completely ignored. One could write a book, but the most obvious example on FADEC is the Boscombe recommendation to re-write the software. Over the years, MoD has sought to present this as Boscombe having a hissy fit.
But the reality was that the regulations required a Safety Plan which incorporated a Code of Design Practice. That had to include an agreed acceptance criteria and “the procedures for dealing with unacceptable components” (00-55, Part 2, para 20k).
If I may quote the next para;
“The acceptance criteria for SCS should distinguish between errors in Formal Arguments and discrepancies found during dynamic testing. The criteria should include the maximum number of errors found by the V&V team in Formal Arguments or during Static Path Analysis beyond which the item will be redeveloped from scratch”.
Given that last, and the sheer number of anomalies uncovered during testing (see JB’s #7866), Boscombe’s recommendation seems entirely reasonable and in line with mandated policy. What we don’t know, however, is what the criteria were. We do know that Boscombe offered an opinion as to what number of anomalies would be acceptable, which tends to indicate no formal criteria was laid down in the Safety Plan. THAT is a major failure on the part of both MoD and Design Authority.
In time, such failures could be corrected. The issue I have here is one of timescale and unseemly haste. FADEC had been in development for years. Yet, a mere 3 weeks before ACAS signed the RTS, Boscombe were still flagging serious deficiencies in the Safety Critical Software. What happened in that 3 week period to persuade ACAS that Boscombe and CA were wrong? MoD’s justification was cobbled together after the event, for the benefit of the Fatal Accident Inquiry. As I’ve said before, they tried to dismiss the issue by saying the software was not Safety Critical; when it clearly was, as defined by MoD policy.
The entire programme is littered with similar major failures and deceit. This is not being wise after the event. On airworthiness, prior warning had been given to both MoD(PE) and AMSO senior staffs between 1988 and 1993; and the 1992 CHART report is effectively a consolidation of those reports as applied to Chinook HC Mk1 and Mk2 (plus Puma and Wessex).
The above doesn’t seek to explain the crash, but illustrate the Organisational Faults that make it impossible to attribute sole blame to the pilots. I sympathise to a degree with those charged with delivering the SCS. At the time, and ever since, MoD’s attitude toward safety amounted to criminal negligence. If a project manager flagged a safety problem, he risked disciplinary action, especially from the RAF Chief Engineer’s non-technical staffs. What is really despicable is the deliberate lies to cover up these failings. The emergence (into the public domain) of the CHART report is vital to understanding all this. To their credit, current MoD staffs have declined to comment, perhaps realising how utterly damning the report is and how implementation would have probably prevented the need for Haddon-Cave. But the reaction of retired officers is revealing; one in particular who delights in denigrating the pilots in the press but will now hopefully be called to account for the failure to implement the report’s recommendations.
Tandemrotor - good post.
Join Date: Sep 2003
Location: Perth, Western Australia
Posts: 786
Likes: 0
Received 0 Likes
on
0 Posts
I recall that some time ago the question arose of how many hours the software was used for without change and without serious incident.
On that occasion there was much waffling and evasiveness – I do not recall a definitive answer.
C240 now gives us that answer and the expected bluster and obfuscation is disgorged from the airworthiness flock.
To that enormous number of hours I add that not only was there no evidence of a FADEC problem in this crash but, as I have described previously, that snapshot state preserved by the impact showed definitively that the engine management system had been performing normally.
On that occasion there was much waffling and evasiveness – I do not recall a definitive answer.
C240 now gives us that answer and the expected bluster and obfuscation is disgorged from the airworthiness flock.
To that enormous number of hours I add that not only was there no evidence of a FADEC problem in this crash but, as I have described previously, that snapshot state preserved by the impact showed definitively that the engine management system had been performing normally.
Join Date: Apr 2008
Location: UK
Posts: 49
Likes: 0
Received 0 Likes
on
0 Posts
Airborne Aircrew wrote:
I think that's a very odd characterisation of how "safe" code is arrived at. In particular, it seems to imply that the software that is first flown can not be safe until it has been flown for long enough to allow all the amendments. Bit of a bummer for the poor sods who have to fly the software in the early days! It's also peculiar to compare with Windows since it is generally considered to be unsuitable for use in a safety-related context. Indeed the constant amendments to it are part of the problem!
This is all too common, at least regarding release, and still happens. You might say that MoD should never let a contract without ensuring access to any safety-related source code and I wouldn't necessarily disagree but it does still happen (sometimes, the source code is profoundly difficult to get hold of due to, e.g., ITAR regulation).
I agree with Thor Nogson that ZH875 gave a plausible description of the situation with the FADEC software (though it does seem that Boscombe actually made a positive assertion that the software was unsafe). I think ZH875 is just describing Airborne Aircrew's fourth option:
I have seen lots of examples of this. I.e., appallingly written safety-related code that is just a spaghetti morass that is essentially unanalysable and untestable. I remember one particular awful example where, working from the spec., I could see thirty (it might have been one or two more, I don't remember) scenarios that needed to be dealt with. So the spec. could have been implemented as a thirty-way case statement and would have been very easy to comprehend accordingly. Instead, the code was a mass of nested and overlapping "if" statements that had well over 60 million distinct paths. Completely impossible to verify by analysis or test.
Thor Nogsen:
That would prove the software is unreliable, not that it is unsafe. It also requires that the spec. exists and is appropriate and complete. So, now's a good time to mention that I have also seen lots of safety-related software variously:
tucumseh says that source code access was catered for explicitly in MoD regulations in the early-mid 90s. I'm a bit surprised to hear that. It was a bugger's job getting hold of the source code for C-130J in the mid-90s and it would have been much easier if it had been contracted in the way tucumseh says regulation demanded. Perhaps the regulation was there but routinely ignored. There's no such regulation these days.
chinook240 wrote:
I'm not sure what chinook240 is getting at since (s)he seems to concede the code may have been amended (presumably in response to some of the deficiencies identified in it). It's perfectly possible to turn unairworthy (unsafe) code into airworthy (safe) code by amending it to remove deficiencies.
I'll also note that 221,000 problem-free flying hours is not enough to demonstrate the airworthiness (safety) of code as critical as the Chinook FADEC software (in the absence of any other forms of evidence). You'd want an absolute minimum of a million hours and very probably a lot more. (It would depend on how "safe" it had to be and how much confidence in its safety you required).
chinook240 again:
Maybe. I think airworthiness is an inherent property of an aircraft because it is concerned with the potential for it to be operated safely. Safety is a property of an aircraft (or some other equipment, system, etc.) in the context of how it is used and maintained. An aircraft can be airworthy but operated unsafely because, for example, it is flown upside-down or flown by untrained aircrew or ...
I don't think airworthiness or safty changes just because one understands the aircraft design. But what will almost certainly change drastically is the ability to demonstrate airworthiness. It is not enough for an aircraft to be airworthy (or safe, for that matter). It must be demonstrated airworthy (or safe, for that matter). And if the FADEC software was unverifiable by Boscombe because they did not understand it and/or the context of its use in the aircraft, it is conceivable it was airworthy (safe) but without their verification evidence no one could (legitimately) claim to have demonstrated it airworthy (or safe).
So when chinook240 writes:
If the code genuinely has not been amended in any way then (imo):
1. it's airworthiness status can not have changed.
2. it's safety status may have changed drasticly becasue it may be used in a different way (chinook240 alludes to changed procedures in another post).
3. the ability to demonstrate the software airworthy or safe may have changed drasticly (when tucumseh writes, "it can be seen that with no further modifications it may have become airworthy, as understanding matured", I disagree but I do think that improved understanding could well have allowed demonstration of airworthiness).
(Oh, and strictly, I disagree with Chugalug2 when (s)he suggests that if regulation is not followed, it necessarily means the software is unairworthy. However, I do believe that if regulation is not followed then it is very unlikely that the software could be satsifactorily demonstrated airworthy (as required) - a subtle distinction.)
Code is tested, retested ad infinitum and released. After release, it is tested in the real world. That's where the things programmers can't see, test for, think of come to light. Then amendments, (updates), are made to the original code and disseminated. Until that process has occurred through numerous cycles it cannot be considered "safe".
I have read several times in this thread that the FADEC code is "un-auditable". There are just two reasons why the code could be in such a state:-
- The company/contractor who wrote the initial code doesn't have/will not release the source code. [...]
I agree with Thor Nogson that ZH875 gave a plausible description of the situation with the FADEC software (though it does seem that Boscombe actually made a positive assertion that the software was unsafe). I think ZH875 is just describing Airborne Aircrew's fourth option:
4. Those charged with auditing the code find such an unholy mess they throw up their hands and give up.
Thor Nogsen:
It is possible however, to prove that [software] is unsafe - If the code doesn't match the specification for example.
That would prove the software is unreliable, not that it is unsafe. It also requires that the spec. exists and is appropriate and complete. So, now's a good time to mention that I have also seen lots of safety-related software variously:
- without any sort of decent specification
- with a specification that has clearly been reverse-engineered from the software
- with a blatently wrong specification
tucumseh says that source code access was catered for explicitly in MoD regulations in the early-mid 90s. I'm a bit surprised to hear that. It was a bugger's job getting hold of the source code for C-130J in the mid-90s and it would have been much easier if it had been contracted in the way tucumseh says regulation demanded. Perhaps the regulation was there but routinely ignored. There's no such regulation these days.
chinook240 wrote:
Approx 221,000 flying hrs later (average AFT OF 13,000 X 17 years) the code has not been rewritten. Is it still grossly unairworthy
I'll also note that 221,000 problem-free flying hours is not enough to demonstrate the airworthiness (safety) of code as critical as the Chinook FADEC software (in the absence of any other forms of evidence). You'd want an absolute minimum of a million hours and very probably a lot more. (It would depend on how "safe" it had to be and how much confidence in its safety you required).
chinook240 again:
As simple aircrew, the idea that understanding the design can change the status of an aircraft from unairworthy to airworthy seems counter-intuitive. Or am I confusing the terms 'airworthy' and 'safe'?
I don't think airworthiness or safty changes just because one understands the aircraft design. But what will almost certainly change drastically is the ability to demonstrate airworthiness. It is not enough for an aircraft to be airworthy (or safe, for that matter). It must be demonstrated airworthy (or safe, for that matter). And if the FADEC software was unverifiable by Boscombe because they did not understand it and/or the context of its use in the aircraft, it is conceivable it was airworthy (safe) but without their verification evidence no one could (legitimately) claim to have demonstrated it airworthy (or safe).
So when chinook240 writes:
My question is simple, the code hasn't been rewritten, if it was grossly unairworthy then, surely it must be now, you can't have it both ways.
1. it's airworthiness status can not have changed.
2. it's safety status may have changed drasticly becasue it may be used in a different way (chinook240 alludes to changed procedures in another post).
3. the ability to demonstrate the software airworthy or safe may have changed drasticly (when tucumseh writes, "it can be seen that with no further modifications it may have become airworthy, as understanding matured", I disagree but I do think that improved understanding could well have allowed demonstration of airworthiness).
(Oh, and strictly, I disagree with Chugalug2 when (s)he suggests that if regulation is not followed, it necessarily means the software is unairworthy. However, I do believe that if regulation is not followed then it is very unlikely that the software could be satsifactorily demonstrated airworthy (as required) - a subtle distinction.)
Walter Kennedy
As far as I can see all that is being said here is the aircraft did not meet the airworthiness standards on the day of the crash and those saying this have offered pretty conclusive proof. I agree that this failing is sufficient to cast doubt on the verdict and can only lend weight to the other main argument that the legal test in AP3207 wasn't satisfied.
It really does take a special kind of imbecile to come on an aviation forum and openly state he doesn't agree with the need to have airworthy aircraft.
bluster and obfuscation is disgorged from the airworthiness flock
It really does take a special kind of imbecile to come on an aviation forum and openly state he doesn't agree with the need to have airworthy aircraft.
Squidlord
Even though you challenge a statement I made, I agree entirely with your post. The difference is that I look at it from a perspective of attaining airworthiness in the first place, whereas I think many here (e.g. aircrew) only see a small part of the maintaining airworthiness process. That is not a criticism; in the same way I cannot begin to imagine what they think and how they go about their job. I think most aircrew would agree that they assume the aircraft is airworthy if it arrives at the squadron door and they are told to fly it. They rely entirely on everyone before them in the chain having done their job properly. But when was that ever enforced in MoD?
Thus, when I talk about “understanding” I mean that in the sense that technology and system integration maturity must be demonstrated to defined levels before approval can be granted to (a) enter development and (b) production. If you don’t achieve that understanding at each stage, then the question of signing an airworthiness statement doesn’t even arise. May I suggest this process is largely invisible to most on this forum. (Thank God say all).
A system such as Safety Critical Software > DECU > FADEC > Chinook cannot be said to be mature if there are hundreds of anomalies in the SCS and the possible effects are not understood. If you do not understand, then how can you implement a Limitations based Release to Service? That is what Boscombe were saying when they recommended a re-write and refused to make a recommendation that the Mk2 be released to service. (A recommendation that the RTS ignores). For a system of “lesser” criticality, one may accept some higher degree of risk, but not for SCS. But even then, if you haven’t demonstrated the required maturity, you simply don’t get approval to proceed to the next stage. This is mandated, but very poorly implemented. A Chinook 2 Star in MoD(PE) / DPA is on record as saying this entire process (e.g. Design Reviews) may be waived to save money.
So, from that perspective, the lack of understanding is one reason why the aircraft could not be declared airworthy. As I said, Boscombe’s position made it unlikely there was a valid Certificate of Design, which the regulations state precludes fitting to an aircraft. If you can’t fit it, as you rightly say, you can’t demonstrate compliance at that stage.
What reveals this lack of maturity in a way everyone here can understand is that there was no Full or Limited clearance for any Nav or Comms system on 2.6.94.
This fact was withheld from aircrew and MoD lied about it when asked by MoD Legal Services before the FAI. If you haven’t got clearance to rely on ANY Nav system, then my instinct tells me flight in IMC is out of the question. And that gets to the nub of MoD’s case.
Even though you challenge a statement I made, I agree entirely with your post. The difference is that I look at it from a perspective of attaining airworthiness in the first place, whereas I think many here (e.g. aircrew) only see a small part of the maintaining airworthiness process. That is not a criticism; in the same way I cannot begin to imagine what they think and how they go about their job. I think most aircrew would agree that they assume the aircraft is airworthy if it arrives at the squadron door and they are told to fly it. They rely entirely on everyone before them in the chain having done their job properly. But when was that ever enforced in MoD?
Thus, when I talk about “understanding” I mean that in the sense that technology and system integration maturity must be demonstrated to defined levels before approval can be granted to (a) enter development and (b) production. If you don’t achieve that understanding at each stage, then the question of signing an airworthiness statement doesn’t even arise. May I suggest this process is largely invisible to most on this forum. (Thank God say all).
A system such as Safety Critical Software > DECU > FADEC > Chinook cannot be said to be mature if there are hundreds of anomalies in the SCS and the possible effects are not understood. If you do not understand, then how can you implement a Limitations based Release to Service? That is what Boscombe were saying when they recommended a re-write and refused to make a recommendation that the Mk2 be released to service. (A recommendation that the RTS ignores). For a system of “lesser” criticality, one may accept some higher degree of risk, but not for SCS. But even then, if you haven’t demonstrated the required maturity, you simply don’t get approval to proceed to the next stage. This is mandated, but very poorly implemented. A Chinook 2 Star in MoD(PE) / DPA is on record as saying this entire process (e.g. Design Reviews) may be waived to save money.
So, from that perspective, the lack of understanding is one reason why the aircraft could not be declared airworthy. As I said, Boscombe’s position made it unlikely there was a valid Certificate of Design, which the regulations state precludes fitting to an aircraft. If you can’t fit it, as you rightly say, you can’t demonstrate compliance at that stage.
What reveals this lack of maturity in a way everyone here can understand is that there was no Full or Limited clearance for any Nav or Comms system on 2.6.94.
This fact was withheld from aircrew and MoD lied about it when asked by MoD Legal Services before the FAI. If you haven’t got clearance to rely on ANY Nav system, then my instinct tells me flight in IMC is out of the question. And that gets to the nub of MoD’s case.
tucumseh says that source code access was catered for explicitly in MoD regulations in the early-mid 90s. I'm a bit surprised to hear that. It was a bugger's job getting hold of the source code for C-130J in the mid-90s and it would have been much easier if it had been contracted in the way tucumseh says regulation demanded. Perhaps the regulation was there but routinely ignored. There's no such regulation these days.
Policy promulgated by Deputy Under Secretary of State for Defence Procurement, December 1987. His successor, J M Stewart, reiterated this in Issue 2 of the policy, 14th December 1989.
If validation procedures are to be carried out by an independent agency (in this case, Boscombe) MoD should secure adequate rights of access and use to design information and code for itself and its agents, while protecting the rights of the originators.
Thus, the policy was extant throughout the FADEC development period and, to my knowledge, has never been rescinded. (Which doesn't mean it is implemented!).
Join Date: Jul 2000
Location: Nova
Posts: 1,242
Likes: 0
Received 0 Likes
on
0 Posts
Hi tuc, I hope you are well.
I would be very grateful if you could put a little more meat on the bones of that statement. As you know, I was involved at precisely the time of preparations for the FAI, and I would very much welcome the concept of people spending time 'at Her Majesty's Pleasure' as a result of 'non disclosure'! I have the means to follow this up.
Very many thanks.
Andy
This fact was withheld from aircrew and MoD lied about it when asked by MoD Legal Services before the FAI. If you haven’t got clearance to rely on ANY Nav system, then my instinct tells me flight in IMC is out of the question. And that gets to the nub of MoD’s case.
Very many thanks.
Andy
Last edited by Tandemrotor; 9th Jun 2011 at 22:15.
Join Date: Nov 2005
Location: Norfolk England
Posts: 247
Likes: 0
Received 0 Likes
on
0 Posts
FAI Disclosures
Tandemrotor,
Have you ever seen the "brief" for the MOD legal team from DHP - quite a few items we have since found out about seem to be "missing" eg the BD letters. If you have not seen it I will happily copy it to you if you PM me the contact details.
JB
Have you ever seen the "brief" for the MOD legal team from DHP - quite a few items we have since found out about seem to be "missing" eg the BD letters. If you have not seen it I will happily copy it to you if you PM me the contact details.
JB
Join Date: Jul 2000
Location: Nova
Posts: 1,242
Likes: 0
Received 0 Likes
on
0 Posts
Hi tuc and JB
Thank you for your kind offers. I will pm you with my email address.
Not long now.
Olive oil. Do you have a suggestion for us to discuss... I'm not really following what point you are trying to make...
Thank you for your kind offers. I will pm you with my email address.
Not long now.
Olive oil. Do you have a suggestion for us to discuss... I'm not really following what point you are trying to make...