‘Data Driven’ or good judgement.
Thread Starter
‘Data Driven’ or good judgement.
From the Senate hearings as reported by WSJ, 29 Oct, criticising the FAA’s role in the 737 Max.
After Lion, the FAA’s analytics showed a good chance that ‘the same malfunction would crop up again’, - presumably AoA failure, but neither the data source or method of analysis was stated (‘a rigid and well defined process’). Thence the reliance on checklist and crew action (Boeing) - the FAA, waits for the report, for more data.
After Ethiopian, ‘the FAA began a fresh risk analysis’ - ‘seeking to quantify the likelihood of a third such emergency’. ‘The FAA maintained … that the specifics (data) were to unclear to merit decisive action’. Thence becoming the last authority to ground the Max.
“We (FAA) have said all along that we are a data-driven organisation.”
Our industry appears to becoming data driven. Do we rely too much on data - statistical analysis, risk management, critical safety calls.
Being data-driven depends on the availability and suitability of data, and choice of analytical technique, both of which involves understanding and judgement.
The final analysis and decision reviewing the output and choosing a course of action is, or should be, an entirely human judgement.
Are we choosing to be driven by data, derogating judgement to a machine, or driven by data as a consequence of modern operations, or not really thinking about the invidious use of ‘data’ and its potential to bias safety action ?
After Lion, the FAA’s analytics showed a good chance that ‘the same malfunction would crop up again’, - presumably AoA failure, but neither the data source or method of analysis was stated (‘a rigid and well defined process’). Thence the reliance on checklist and crew action (Boeing) - the FAA, waits for the report, for more data.
After Ethiopian, ‘the FAA began a fresh risk analysis’ - ‘seeking to quantify the likelihood of a third such emergency’. ‘The FAA maintained … that the specifics (data) were to unclear to merit decisive action’. Thence becoming the last authority to ground the Max.
“We (FAA) have said all along that we are a data-driven organisation.”
Our industry appears to becoming data driven. Do we rely too much on data - statistical analysis, risk management, critical safety calls.
Being data-driven depends on the availability and suitability of data, and choice of analytical technique, both of which involves understanding and judgement.
The final analysis and decision reviewing the output and choosing a course of action is, or should be, an entirely human judgement.
Are we choosing to be driven by data, derogating judgement to a machine, or driven by data as a consequence of modern operations, or not really thinking about the invidious use of ‘data’ and its potential to bias safety action ?
Paxing All Over The World
Your business is not driven by data but by money. All the data is formulated to look AT the money. Since more money = good, then follow the data. Simples!!
This change over from putting the customer first to putting the shareholders and the bonus for the Directors first - started in real earnest some 35 years ago. The UK signed up enthusiastically to the USA game plan and here we are. There is the classic injunction to: "Follow the money" (Watergate)
Forget about 'Seat of the Pants' - even though that is what usually saves the day. Someone, somewhere looks at a bunch of financial results - or the instruments on the panels and thinks, "Something's not right here ..."
This change over from putting the customer first to putting the shareholders and the bonus for the Directors first - started in real earnest some 35 years ago. The UK signed up enthusiastically to the USA game plan and here we are. There is the classic injunction to: "Follow the money" (Watergate)
Forget about 'Seat of the Pants' - even though that is what usually saves the day. Someone, somewhere looks at a bunch of financial results - or the instruments on the panels and thinks, "Something's not right here ..."
Thread Starter
PAX, The OP question relates to safety activity.
The debate re safety is the business (‘safety always our first priority’) is covered extensively at https://skybrary.aero/bookshelf/books/5377.pdf
The debate re safety is the business (‘safety always our first priority’) is covered extensively at https://skybrary.aero/bookshelf/books/5377.pdf
Join Date: Nov 2015
Location: Paisley, Florida USA
Posts: 289
Likes: 0
Received 0 Likes
on
0 Posts
Data trumps Good Judgement
"The FAA and OEMs should re-evaluate their assumptions for what constitutes an average flight crew’s basic skill and what level of systems knowledge a ‘properly trained average flight crew’ has when encountering failures.
Therefore, KNKT recommends that Boeing include a larger tolerance in the design is (sic) required to allow operability by a larger population of flight-rated pilots."
Therefore, KNKT recommends that Boeing include a larger tolerance in the design is (sic) required to allow operability by a larger population of flight-rated pilots."
Cheers,
Grog
Join Date: Jun 2019
Location: VA
Posts: 210
Likes: 0
Received 0 Likes
on
0 Posts
I think "data driven" is one of those buzz words that sound good but don't really get at the underlying philosophy. For example, one could take the position to assume something is fine until "the data" tells them it is not. On the other hand, one could assume something is suspect until "the data" says that it is okay. Both philosophies are "data driven", but they differ radically in how that data is used.
Case in point - I asked someone in our training department why Runaway Stab Trim (a memory item) wasn't a training spot in the sim for initial/transition courses. I was told that because "the data" indicated that this was a very rare event on the 737 and thus not a good use of sim time. This person was technically correct on the probability, but was this the proper way to use the data? I think not.
Case in point - I asked someone in our training department why Runaway Stab Trim (a memory item) wasn't a training spot in the sim for initial/transition courses. I was told that because "the data" indicated that this was a very rare event on the 737 and thus not a good use of sim time. This person was technically correct on the probability, but was this the proper way to use the data? I think not.
Join Date: Oct 2017
Location: Vienna
Posts: 143
Likes: 0
Received 0 Likes
on
0 Posts
FAA's "data driven" in this case is nothing but smoke and mirrors, a PR fad to cover up the real reason of their inaction.
Further, "data driven" does not apply to fatal airplane incidents. See, to be "data driven", you must first collect data about events. Thing like "market basket analysis" where you analyse millions of "shopping baskets" to figure out what people who bought A are also inclined to buy, and make sure they are exposed to it so the likelyhod of sell is higher.
Now, applying "data driven" to flight safety would mean you let airplanes crash and don't bother to investigate until a certain type has statistically significant higher probability of haul loss than other types, and only then start to investigate why.
Well, I guess I was wrong in my opening sentence then... clearly FAA was in fact "data driven" in this case!
Further, "data driven" does not apply to fatal airplane incidents. See, to be "data driven", you must first collect data about events. Thing like "market basket analysis" where you analyse millions of "shopping baskets" to figure out what people who bought A are also inclined to buy, and make sure they are exposed to it so the likelyhod of sell is higher.
Now, applying "data driven" to flight safety would mean you let airplanes crash and don't bother to investigate until a certain type has statistically significant higher probability of haul loss than other types, and only then start to investigate why.
Well, I guess I was wrong in my opening sentence then... clearly FAA was in fact "data driven" in this case!
Join Date: May 2010
Location: Boston
Age: 73
Posts: 443
Likes: 0
Received 0 Likes
on
0 Posts
I think "data driven" is one of those buzz words that sound good but don't really get at the underlying philosophy. For example, one could take the position to assume something is fine until "the data" tells them it is not. On the other hand, one could assume something is suspect until "the data" says that it is okay. Both philosophies are "data driven", but they differ radically in how that data is used.
Case in point - I asked someone in our training department why Runaway Stab Trim (a memory item) wasn't a training spot in the sim for initial/transition courses. I was told that because "the data" indicated that this was a very rare event on the 737 and thus not a good use of sim time. This person was technically correct on the probability, but was this the proper way to use the data? I think not.
Case in point - I asked someone in our training department why Runaway Stab Trim (a memory item) wasn't a training spot in the sim for initial/transition courses. I was told that because "the data" indicated that this was a very rare event on the 737 and thus not a good use of sim time. This person was technically correct on the probability, but was this the proper way to use the data? I think not.
The basic mantra is requirement for 'proof of effectiveness' which can cause major issues since many procedures were developed and became widely practiced without formal studies.
There have been a few instances of accepted practice not leading to best outcomes when studied which further muddies the issue.
Point being that "data driven' is a rather blunt instrument that can be (like most statistics) twisted to support predetermined conclusions. Of course the insurance companies are not eager to step up and help pay for studies of existing unstudied procedures.
My take on the medical field is "lack of proof is not proof of lack (of effectiveness)
Join Date: Oct 2017
Location: Vienna
Posts: 143
Likes: 0
Received 0 Likes
on
0 Posts
Yes, it is! But did you know that when we don't like the results of data analysis we call it bias, and manually "fix" the models to be "unbiased"?
https://www.kdnuggets.com/2018/09/si...-penalize.html
https://www.kdnuggets.com/2018/09/si...-penalize.html
Join Date: May 2010
Location: Boston
Age: 73
Posts: 443
Likes: 0
Received 0 Likes
on
0 Posts
My cynical view is just that the insurance companies are twisting things to their advantage, hardly a surprise I guess.
Now back to cynical discussion of original thread topic
Plastic PPRuNer
'This change over from putting the customer first to putting the shareholders and the bonus for the Directors first - started in real earnest some 35 years ago."
"There is a parallel problem in medical practice, driven in large part by insurance companies trying to reduce costs for expensive procedures."
Sadly both true. We have moved away from 'ethics driven ethics' toward 'legally driven ethics".
The question now is not, "What is best for the patient?", but "What procedure can I do that is least-likely to get me sued if someone is unhappy?"
The world has become rather a nasty money driven-place, where doing the right thing can often get you fired.
Mac :-(
"There is a parallel problem in medical practice, driven in large part by insurance companies trying to reduce costs for expensive procedures."
Sadly both true. We have moved away from 'ethics driven ethics' toward 'legally driven ethics".
The question now is not, "What is best for the patient?", but "What procedure can I do that is least-likely to get me sued if someone is unhappy?"
The world has become rather a nasty money driven-place, where doing the right thing can often get you fired.
Mac :-(
Thread Starter
Dave, data may be a fact, but alone it does not provide understanding required for decision making.
More data is more facts which might improve understanding (how to ask better questions), but this depends how the ‘facts’ are presented and interpreted - how we judge the value of accumulated data.
Understanding is not just joining the facts, it’s knowing what counts as a fact in the first place; what is the context of the enquiry. Simple views might only consider poor outcomes - fatalities, but in a highly reliable industry we require the more complex view - knowledge of factual relevance and how contributions interact. Without this knowledge or ability to identify interactions, all that data-driven might mean is there are many facts but without any clear understanding of what they mean.
derjodel, good link and follow-on links for those people wishing to be ‘data educated’.
Mac,
loma, good judgement can be improved with expertise (for the less ignorant).
Choosing to be data-driven indicates the lack of expertise - even unwillingness or impossibility to become an expert in the modern world.
This ‘unwillingness’ is an attitude that someone else (or some machine) will apply judgement; regulator trades judgement for data, or down-loads the task to the manufacturer, who similarly down-loads the task to the operator (“give us the data”), and where the operator just generates data without knowledge of how it should / could be used.
Our regulation, airworthiness, safety, all depend on data assumption. That some one / some machine will manage these data. This is a continuing problem and often appears as the inability to manage the complexities involving human interaction, thus solutions revert to the more tangible ‘use automation’, which overlook that this ‘solution’ too involves human interaction. - grog.
If the answer to #1 is yes, then what is the next question; what might be done, what future for aviation safety.
More data is more facts which might improve understanding (how to ask better questions), but this depends how the ‘facts’ are presented and interpreted - how we judge the value of accumulated data.
Understanding is not just joining the facts, it’s knowing what counts as a fact in the first place; what is the context of the enquiry. Simple views might only consider poor outcomes - fatalities, but in a highly reliable industry we require the more complex view - knowledge of factual relevance and how contributions interact. Without this knowledge or ability to identify interactions, all that data-driven might mean is there are many facts but without any clear understanding of what they mean.
derjodel, good link and follow-on links for those people wishing to be ‘data educated’.
Mac,
loma, good judgement can be improved with expertise (for the less ignorant).
Choosing to be data-driven indicates the lack of expertise - even unwillingness or impossibility to become an expert in the modern world.
This ‘unwillingness’ is an attitude that someone else (or some machine) will apply judgement; regulator trades judgement for data, or down-loads the task to the manufacturer, who similarly down-loads the task to the operator (“give us the data”), and where the operator just generates data without knowledge of how it should / could be used.
Our regulation, airworthiness, safety, all depend on data assumption. That some one / some machine will manage these data. This is a continuing problem and often appears as the inability to manage the complexities involving human interaction, thus solutions revert to the more tangible ‘use automation’, which overlook that this ‘solution’ too involves human interaction. - grog.
If the answer to #1 is yes, then what is the next question; what might be done, what future for aviation safety.
Join Date: May 2010
Location: Boston
Age: 73
Posts: 443
Likes: 0
Received 0 Likes
on
0 Posts
From the Senate hearings as reported by WSJ, 29 Oct, criticising the FAA’s role in the 737 Max.
After Lion, the FAA’s analytics showed a good chance that ‘the same malfunction would crop up again’, - presumably AoA failure, but neither the data source or method of analysis was stated (‘a rigid and well defined process’). Thence the reliance on checklist and crew action (Boeing) - the FAA, waits for the report, for more data.
After Ethiopian, ‘the FAA began a fresh risk analysis’ - ‘seeking to quantify the likelihood of a third such emergency’. ‘The FAA maintained … that the specifics (data) were to unclear to merit decisive action’. Thence becoming the last authority to ground the Max.
“We (FAA) have said all along that we are a data-driven organisation.”
After Lion, the FAA’s analytics showed a good chance that ‘the same malfunction would crop up again’, - presumably AoA failure, but neither the data source or method of analysis was stated (‘a rigid and well defined process’). Thence the reliance on checklist and crew action (Boeing) - the FAA, waits for the report, for more data.
After Ethiopian, ‘the FAA began a fresh risk analysis’ - ‘seeking to quantify the likelihood of a third such emergency’. ‘The FAA maintained … that the specifics (data) were to unclear to merit decisive action’. Thence becoming the last authority to ground the Max.
“We (FAA) have said all along that we are a data-driven organisation.”
Same data can be used in totally opposite ways.
Echoes of the Challenger accident where engineers were challenged to 'prove it was not safe to fly' after raising doubts about the O rings operating outside designed temperature range.
Data, judgement, skill
Salute!
As an unwilling IP for most of my time in USAF, I feel qualified to add my two cents here. 3 jets, single-seat, one engine for two of them.
I checked out several folks that I called "technicians". They had procedures nailed down cold, knew all about the plane but were basically automatons. When it came to "feeling" the plane and where it was in the envelope, they were in the lower third of all the folks I helped learn. But their judgement was great and they knew their own limits and the "published" airplane limits. They went on to fruitful careers. But the folks we encountered with pee poor judgement, next to no "feeling" for the plane and lack of system knowledge got sent on to something less demanding. Sorry, but no room for "equal opportunity" single-seat pilots.
So my point is that good pilots have a good ratio of 1) knowledge of data concerning the plane and basic aero principles, 2) know their own limits and those of the plane/crew/support folks, and then 3) their skill.
The integration of those factors, and a small amount of luck involving catastrophic equipment failure or enemy fire, result in several of us here entering our 5th decade of flying and posting here.
Gums opines....
As an unwilling IP for most of my time in USAF, I feel qualified to add my two cents here. 3 jets, single-seat, one engine for two of them.
I checked out several folks that I called "technicians". They had procedures nailed down cold, knew all about the plane but were basically automatons. When it came to "feeling" the plane and where it was in the envelope, they were in the lower third of all the folks I helped learn. But their judgement was great and they knew their own limits and the "published" airplane limits. They went on to fruitful careers. But the folks we encountered with pee poor judgement, next to no "feeling" for the plane and lack of system knowledge got sent on to something less demanding. Sorry, but no room for "equal opportunity" single-seat pilots.
So my point is that good pilots have a good ratio of 1) knowledge of data concerning the plane and basic aero principles, 2) know their own limits and those of the plane/crew/support folks, and then 3) their skill.
The integration of those factors, and a small amount of luck involving catastrophic equipment failure or enemy fire, result in several of us here entering our 5th decade of flying and posting here.
Gums opines....
Paxing All Over The World
safteypee
Indeed, which is why I mentioned money. We all know that insurers have a checklist of how much each human limb and life is worth. The book The Tombstone Imperative ~ The Truth About Air Safety by Andrew Weir said it all 20 years ago.
PAX, The OP question relates to safety activity.