‘Data Driven’ or good judgement.

[safetypee]

From the Senate hearings as reported by WSJ, 29 Oct, criticising the FAA’s role in the 737 Max.

After Lion, the FAA’s analytics showed a good chance that ‘the same malfunction would crop up again’, - presumably AoA failure, but neither the data source or method of analysis was stated (‘a rigid and well defined process’). Thence the reliance on checklist and crew action (Boeing) - the FAA, waits for the report, for more data.

After Ethiopian, ‘the FAA began a fresh risk analysis’ - ‘seeking to quantify the likelihood of a third such emergency’. ‘The FAA maintained … that the specifics (data) were to unclear to merit decisive action’. Thence becoming the last authority to ground the Max.

“We (FAA) have said all along that we are a data-driven organisation.”

Our industry appears to becoming data driven. Do we rely too much on data - statistical analysis, risk management, critical safety calls.
Being data-driven depends on the availability and suitability of data, and choice of analytical technique, both of which involves understanding and judgement.
The final analysis and decision reviewing the output and choosing a course of action is, or should be, an entirely human judgement.

Are we choosing to be driven by data, derogating judgement to a machine, or driven by data as a consequence of modern operations, or not really thinking about the invidious use of ‘data’ and its potential to bias safety action ?

[PAXboy]

Your business is not driven by data but by money. All the data is formulated to look AT the money. Since more money = good, then follow the data. Simples!!

This change over from putting the customer first to putting the shareholders and the bonus for the Directors first - started in real earnest some 35 years ago. The UK signed up enthusiastically to the USA game plan and here we are. There is the classic injunction to: "Follow the money" (Watergate)

Forget about 'Seat of the Pants' - even though that is what usually saves the day. Someone, somewhere looks at a bunch of financial results - or the instruments on the panels and thinks, "Something's not right here ..."

[safetypee]

PAX, The OP question relates to safety activity.

The debate re safety is the business (‘safety always our first priority’) is covered extensively at https://skybrary.aero/bookshelf/books/5377.pdf

[capngrog]

Are we choosing to be driven by data, derogating judgement to a machine, or driven by data as a consequence of modern operations, or not really thinking about the invidious use of ‘data’ and its potential to bias safety action ? I'm afraid that the answer to your question is Yes. One only has to read the extensive discussions on this Forum concerning the growing reliance on transport category aircraft automation and the decreasing reliance upon pilot skills. This seems to have been confirmed in the Indonesian National Transportation Safety Committee (KNKT) Final Report on the crash of Lion Air Flight LNI610. Among many Safety Recommendations included in the Report is Number 04.M-2018.12, from which I quote:
"The FAA and OEMs should re-evaluate their assumptions for what constitutes an average flight crew’s basic skill and what level of systems knowledge a ‘properly trained average flight crew’ has when encountering failures.

Therefore, KNKT recommends that Boeing include a larger tolerance in the design is (sic) required to allow operability by a larger population of flight-rated pilots." I interpret the above to mean that more dependence should be placed on automation, rather than flight crew competence, so as to allow a population of less qualified flight-rated pilots to have access to the cockpits of modern transport category aircraft. How do we define "qualified" or "competent" with respect to piloting transport category aircraft? Is there a widely recognized standard of competence?

Cheers,
Grog

[Tomaski]

I think "data driven" is one of those buzz words that sound good but don't really get at the underlying philosophy. For example, one could take the position to assume something is fine until "the data" tells them it is not. On the other hand, one could assume something is suspect until "the data" says that it is okay. Both philosophies are "data driven", but they differ radically in how that data is used.

Case in point - I asked someone in our training department why Runaway Stab Trim (a memory item) wasn't a training spot in the sim for initial/transition courses. I was told that because "the data" indicated that this was a very rare event on the 737 and thus not a good use of sim time. This person was technically correct on the probability, but was this the proper way to use the data? I think not.

[derjodel]

FAA's "data driven" in this case is nothing but smoke and mirrors, a PR fad to cover up the real reason of their inaction.

Further, "data driven" does not apply to fatal airplane incidents. See, to be "data driven", you must first collect data about events. Thing like "market basket analysis" where you analyse millions of "shopping baskets" to figure out what people who bought A are also inclined to buy, and make sure they are exposed to it so the likelyhod of sell is higher.

Now, applying "data driven" to flight safety would mean you let airplanes crash and don't bother to investigate until a certain type has statistically significant higher probability of haul loss than other types, and only then start to investigate why.

Well, I guess I was wrong in my opening sentence then... clearly FAA was in fact "data driven" in this case!

[MurphyWasRight]

I think "data driven" is one of those buzz words that sound good but don't really get at the underlying philosophy. For example, one could take the position to assume something is fine until "the data" tells them it is not. On the other hand, one could assume something is suspect until "the data" says that it is okay. Both philosophies are "data driven", but they differ radically in how that data is used.

Case in point - I asked someone in our training department why Runaway Stab Trim (a memory item) wasn't a training spot in the sim for initial/transition courses. I was told that because "the data" indicated that this was a very rare event on the 737 and thus not a good use of sim time. This person was technically correct on the probability, but was this the proper way to use the data? I think not.

There is a parallel problem in medical practice, driven in large part by insurance companies trying to reduce costs for expensive procedures.

The basic mantra is requirement for 'proof of effectiveness' which can cause major issues since many procedures were developed and became widely practiced without formal studies.
There have been a few instances of accepted practice not leading to best outcomes when studied which further muddies the issue.

Point being that "data driven' is a rather blunt instrument that can be (like most statistics) twisted to support predetermined conclusions. Of course the insurance companies are not eager to step up and help pay for studies of existing unstudied procedures.

My take on the medical field is "lack of proof is not proof of lack (of effectiveness)

[cattletruck]

ICD-10 and DSM-5 are data driven models for very good epidemiological reasons, not perfect, but otherwise it would just be quackery.

[andrasz]

I can only respond with an oft quoted classic: "There are lies, damned lies and statistics"

[DaveReidUK]

"Data" is simply a synonym for "the facts". Clearly we should never let those get in the way of a good story, or of good decision-making. :O

[derjodel]

"Data" is simply a synonym for "the facts".

Yes, it is! But did you know that when we don't like the results of data analysis we call it bias, and manually "fix" the models to be "unbiased"?

https://www.kdnuggets.com/2018/09/siegel-when-algorithms-explicitly-penalize.html

[MurphyWasRight]

ICD-10 and DSM-5 are data driven models for very good epidemiological reasons, not perfect, but otherwise it would just be quackery.
Totally agree, as I mentioned not all "established practice" is found to be optimal (or even not harmful) when tested.
My cynical view is just that the insurance companies are twisting things to their advantage, hardly a surprise I guess.
Now back to cynical discussion of original thread topic :)

[Mac the Knife]

'This change over from putting the customer first to putting the shareholders and the bonus for the Directors first - started in real earnest some 35 years ago."

"There is a parallel problem in medical practice, driven in large part by insurance companies trying to reduce costs for expensive procedures."

Sadly both true. We have moved away from 'ethics driven ethics' toward 'legally driven ethics".
The question now is not, "What is best for the patient?", but "What procedure can I do that is least-likely to get me sued if someone is unhappy?"

The world has become rather a nasty money driven-place, where doing the right thing can often get you fired.

Mac :-(

[lomapaseo]

Sorry but most "good judgement" is in the eyes of the ignorant of life';s choices. They just seem to flit from moment to moment of what feels good.

[safetypee]

Dave, data may be a fact, but alone it does not provide understanding required for decision making.
More data is more facts which might improve understanding (how to ask better questions), but this depends how the ‘facts’ are presented and interpreted - how we judge the value of accumulated data.
Understanding is not just joining the facts, it’s knowing what counts as a fact in the first place; what is the context of the enquiry. Simple views might only consider poor outcomes - fatalities, but in a highly reliable industry we require the more complex view - knowledge of factual relevance and how contributions interact. Without this knowledge or ability to identify interactions, all that data-driven might mean is there are many facts but without any clear understanding of what they mean.

derjodel, good link and follow-on links for those people wishing to be ‘data educated’.

Mac, :ok:

loma, good judgement can be improved with expertise (for the less ignorant).
Choosing to be data-driven indicates the lack of expertise - even unwillingness or impossibility to become an expert in the modern world.
This ‘unwillingness’ is an attitude that someone else (or some machine) will apply judgement; regulator trades judgement for data, or down-loads the task to the manufacturer, who similarly down-loads the task to the operator (“give us the data”), and where the operator just generates data without knowledge of how it should / could be used.
Our regulation, airworthiness, safety, all depend on data assumption. That some one / some machine will manage these data. This is a continuing problem and often appears as the inability to manage the complexities involving human interaction, thus solutions revert to the more tangible ‘use automation’, which overlook that this ‘solution’ too involves human interaction. - grog.

If the answer to #1 is yes, then what is the next question; what might be done, what future for aviation safety.

[Takwis]

I noticed an ALPA message yesterday that promised to be 'data driven' in their return to flying the MAX.

[MurphyWasRight]

From the Senate hearings as reported by WSJ, 29 Oct, criticising the FAA’s role in the 737 Max.

After Lion, the FAA’s analytics showed a good chance that ‘the same malfunction would crop up again’, - presumably AoA failure, but neither the data source or method of analysis was stated (‘a rigid and well defined process’). Thence the reliance on checklist and crew action (Boeing) - the FAA, waits for the report, for more data.

After Ethiopian, ‘the FAA began a fresh risk analysis’ - ‘seeking to quantify the likelihood of a third such emergency’. ‘The FAA maintained … that the specifics (data) were to unclear to merit decisive action’. Thence becoming the last authority to ground the Max.

“We (FAA) have said all along that we are a data-driven organisation.”



What is unstated in above (from OP) is whether the data was used to 'prove it is safe to fly' or 'prove that it should be grounded'.
Same data can be used in totally opposite ways.

Echoes of the Challenger accident where engineers were challenged to 'prove it was not safe to fly' after raising doubts about the O rings operating outside designed temperature range.

[gums]

Salute!

As an unwilling IP for most of my time in USAF, I feel qualified to add my two cents here. 3 jets, single-seat, one engine for two of them.

I checked out several folks that I called "technicians". They had procedures nailed down cold, knew all about the plane but were basically automatons. When it came to "feeling" the plane and where it was in the envelope, they were in the lower third of all the folks I helped learn. But their judgement was great and they knew their own limits and the "published" airplane limits. They went on to fruitful careers. But the folks we encountered with pee poor judgement, next to no "feeling" for the plane and lack of system knowledge got sent on to something less demanding. Sorry, but no room for "equal opportunity" single-seat pilots.

So my point is that good pilots have a good ratio of 1) knowledge of data concerning the plane and basic aero principles, 2) know their own limits and those of the plane/crew/support folks, and then 3) their skill.

The integration of those factors, and a small amount of luck involving catastrophic equipment failure or enemy fire, result in several of us here entering our 5th decade of flying and posting here.

Gums opines....

[lomapaseo]

OK, a few of us seem to be data driven with a mixture of good judgement. So where's the beef?

[PAXboy]

safteypee
PAX, The OP question relates to safety activity.
Indeed, which is why I mentioned money. We all know that insurers have a checklist of how much each human limb and life is worth. The book The Tombstone Imperative ~ The Truth About Air Safety by Andrew Weir said it all 20 years ago.

[PEI_3721]

gums, you identify a root issue of expertise. The need of knowledge, both ‘know what’ and ‘know how’, and the skill to call upon these knowledge types, identifying the situations when they should be used, the ‘how to’ of operating; professionalism.

The availability of instant ‘data’ from search engines, accessible flight records, and computer analysis / visualisation, minimise the need to generate knowledge - the ‘what’. This also applies to know how, no need to crank the numbers, reducing opportunity to practice critical thinking and thus improve experience of interpretation and judgement.

Being data driven is another facet of technology dependency, automation dependency. We use ‘it’ because it’s there, because it’s easy, but rarely question why we use it, a lost skill.

[RevMan2]

From data comes information.

From information comes knowledge.

if you’re fixated on data alone,you’re stuffed....

[Bergerie1]

And from information and experience comes judgement

[tdracer]

It sort of begs the question, if we're not going to use facts and data to make safety decisions, what would take it's place?
Judgement - in the absence of facts and data - is simply opinion with a nice name.

[MurphyWasRight]

It sort of begs the question, if we're not going to use facts and data to make safety decisions, what would take it's place?
Judgement - in the absence of facts and data - is simply opinion with a nice name.

Of course facts and data are vital, combined with general subject matter knowledge and experience is all that we have.

That said try not to confuse plain English meaning with the latest buzzwords :)

From the original post:
‘The FAA maintained … that the specifics (data) were to unclear to merit decisive action’. Thence becoming the last authority to ground the Max.

“We (FAA) have said all along that we are a data-driven organisation.”

is bureaucrat speak for "we don't have proof that the MAX is unsafe" even though the 'raw data' of 2 accidents shortly after takeoff of a new type should have been clear enough, especially given that the first accident had revealed the existence and power of MCAS. They certainly did not have the data to prove it was safe.

[Tomaski]

An informative read on the abuses of data: The Tyranny of Metrics by Jerry Muller

[GordonR_Cape]

An informative read on the abuses of data: The Tyranny of Metrics by Jerry Muller (https://www.amazon.com/Tyranny-Metrics-Jerry-Z-Muller/dp/0691174954)

Excellent link, thanks for posting. The Amazon review page has a number of interesting comments. Perhaps the most poignant one:
It should be required reading for any manager on the verge of making the Vietnam body count mistake all over again." ---Tim Harford, Financial Times

[safetypee]

From data comes information.
From information comes knowledge.
And from information and experience comes judgement.

These depict a flow chart, but lack the action of a process.
Information, knowledge, and judgement do not just appear - data does not become information without thought, which in turn depends on the ability and willingness to actively engage with data, to form an understanding necessary for judgement - making sense of data - thinking about how we use the data and not delegating to a data handling machine.

Related ref:
Critical Thinking https://www.dropbox.com/s/c5otpsl20awxr35/Critical%20Thinking.pdf?dl=0 (use web view as required)

[Centaurus]

I was told that because "the data" indicated that this was a very rare event on the 737 and thus not a good use of sim time.
A relatively recent buzz-word to describe simulator training is "Evidence Based Training." Broadly this means where QAR incidents (evidence) indicate an undesirable trend; long landings for example, then future simulator sessions are tailored to accent touch downs at the right speed and correct zone on the runway. As simulator sessions are expensive and often tightly programmed, this means other normal simulator exercises may have to be reduced or left out.

Let's face it -all simulator training is by definition "evidenced based." All student pilots start off learning to fly with an instructor from a syllabus of training. Takeoff and landing competency is achieved by trial and error. When a student is deemed by his instructor to be safe for first solo, that judgement is based on real time evidence. In other words the student has completed the syllabus of training so far, without crashing. Surely that is evidence based training? Evidence based training is just another name or buzz-word for normal training - or am I missing something hidden in the buzz-word?

Turbo jet engines have proved so reliable that it is rare for a pilot to experience a real engine failure on take off in his career. Nevertheless if the worst happens - an engine failure at lift off at max takeoff weight - we know it can require good flying skill to handle such a situation. Yet, with the chance of a real engine failure so rare, there is a school of thought that the number of practice engine failures in the simulator should be significantly reduced and the time saved spent on more likely scenarios. In fact, experience in simulator training has shown that more pilot error stuff ups occur during practice engine failures on takeoff, than straight forward system failures where physical hand flying by crews takes place.

On that evidence alone the number of practice engine failures during takeoff in simulator training should remain unchanged. And while we are on the subject could we please dispense with the term Evidence Based Training. State the bleeding obvious - it is Normal Training.

[gums]

Salute!

Thank you, Centaurus, the use of the latest buzzwords for all kindsa programs, methods, prodedures and so forth is now a part of the "management" approach to things versus "the empirical approach that has its failures, get up and dust off, then don't make that mistake again" way of doing business.

I like the idea of illustrating problems in the sim due to recent incidents. Nevertheless, there is no substitute for knowledge and experience when something weird happens like my most serious emergency. I can guarantee that the last thing I wanted to do or could do was ask my FO to look up some procedure for a malfunction that was not in the book.

So related to my previous post, I liked the reference that Tom provided, tho' I ain't buying the book!!

... metrics can be good when used as a complement to rather than a replacement for judgement based on personal experience

Beginning to ramble now, but I wish for all to know that the "Yeager" types can save the day, but the greybeard that has had a few exciting events and knows the systems has prolly saved more lives.

Gums opines...

[PAXboy]

Circuit training transferred to SIM as soon as it was viable to do so and they saved tons of money. Now they want to cut down on SIM time to save tons of money ...

This is what commerce does. This is why strong regulations are needed. This does NOT mean a hefty ultra 'left wing' regime - just sensible, honest, open regulation aimed at making the crew and pax safe. Too many countries have not found the right balance.

[Bergerie1]

safetypee,

Agreed - 100%

[PEI_3721]

Centaurus, # 29, interesting.
Another view of EBT focuses on the training process itself, identifying evidence of the effectiveness of what has been trained and not using accident statistics
We hope that there are no more accidents / events, thus there may be little or no data to crosscheck the training outcome; - ‘absence of evidence is not evidence of absence of continuing risk’.

Being data-driven is often based on scant data seeking to prove the effectiveness of interventions.
Thus we should use a data aiding system with human involvement, vice reliance on machine alone.

[megan]

but I wish for all to know that the "Yeager" types can save the dayNot always, Chuck had to throw away a F-104 because his instrument skills were not up to the task, and came close to losing his life in the ejection, ego had a role as well, in that he kicked the project pilot out of the seat so he could notch up another (altitude) record.

Data - what do you say to a company who doesn't comply with regs etc or believe in training but has an 27 year accident free record, mainly due to the quality of the pilots and a lot of luck (pilots messed up but recovered), major mechanical events always happening in benign circumstances.

[gums]

Salute!

Not sure where you are coming from, megan.

My comment was to point out some very serious incidents that concluded with minimum loss of life due to very good crews or pilots. I call them "yeager types", but could call them Crossfield, White, Armstrong, LeVier, et al.

The safety folks do not count on very talented pilots to save the day. So the use of statistical data plays a role in final determination of risk and such.

Bottomline is you cannot implement a system that requires an experienced golden arm test pilot to save the day.

Gums sends......

[safetypee]

This thread suggests that there is a range of views, without consistent definition of ‘data- driven’, although the trend is towards using more data.

A web search for a definition and alternative views was also inconsistent; ‘compelled by data’ - or - ‘weighing’ data.

… data-driven means that progress in an activity is compelled b data, rather than by intuition or by personal experience. - Wiki

… a “data-driven” approach, makes strategic decisions based on data analysis and interpretation. - atinternet

Data driven … a process or activity that is spurred on by data, as opposed to being driven by mere intuition or personal experience. In other words, the decision is made with hard empirical evidence and not speculation or gut feel. - techopedia

Data driven decisions are made by weighing data. This suffers from the problem that in order to make an optimal decision with data you must first select the optimal decision model. This can be shown to be paradox whereby there is no way to guarantee you are making the optimal decision because you always have to start with an assumption such as the assertion that a decision model or algorithm is correct. - simplicable

Data must be accessible and queryable. - oreilly

The last two quotes are interesting. If safety action is based on a ‘decision model’ (required data) then how are decisions to be checked; only in hindsight ?
Availability and type of data is critical; with appropriate data, it’s analysis can be used to guide decisions.
Without, there may only be intuition.

For the 737 Max accidents, the approaches taken by the regulators span the differing views.
Reactive - we lack data, no decision.
Proactive - seek additional data and use it in the decision process.

The industry’s use of data appears to depend on our outlook - reactive / proactive, the availability of appropriate data, and the situation. But what is appropriate data, and it’s use, and in which situations ?