CRM Assesment

[divinehover]

This question is mailnly directed at airline crew but all input will be welcome.

My airline (South African Airways) is about to introduce CRM assesment into ratings and management acceptance route checks. The is all part of the airlines move to OBE (Outcome Based Education) training and testing. It now means that you can fly a great test but fail on your CRM skills (or lack thereof)

My question is this. How many other airlines actually asses their pilots CRM skills as opposed to just using it as a training and safety tool?

[BOAC]

Most UK airlines I think, but it is not a 'pass/fail' item, rather a 'debrief' item.

OBE!!! I prefer Income Based Education:) Sounds like some top 'management-speak' going on there.

[Eff Oh]

All UK airlines use it. You can now fail an OPC or LPC purely on poor CRM. The UK CAA seem to be taking CRM a lot more seriously!!

[low n' slow]

I don't see a problem in it. Everyone can talk about what CRM is and how to best practise it, but it's a whole different ballgame to actually be able to use these skills. A very reasonable part of any line check.

/LnS

[divinehover]

Dear LnS

I will never doubt the importance of CRM. I fact I see it as the single most important safety tool we have. My worry is that it is now becoming a asset aspect. Are we not moving away from 'just culture' mentality which has taken so long to enstill? How do you asses CRM anyway. By what defined criteria do you measure someones CRM skills?

DH

[Non-PC Plod]

CRM is generally assessed according to very carefully defined criteria. NOTECHS is one example, developed within Europe to assess non-technical skills as objectively as possible. The University of Texas developed its own set of criteria, and various Airlines use a variation of these techniques.
In a nutshell, non-technical skills are divided into categories: eg Leadership and management, situational awareness, decision-making and cooperation. These are then subdivided into elements, each of which will have examples of desirable and undesirable behaviours. These are known as "behavioual markers" For example: poor practice - "Does not intervene in case of deviations" , good practice: "intervenes if task completion deviates from standards". There is then a need to set the pass/fail standard - do you let someone pass whose behaviour has not resulted in an unsafe situation on this occasion, but potentially could? Or do you need to have evidence of a technical failure as the outcome of the CRM problem in order to fail a line check? Dfferent operators will have different views on this.
Hope this explains the methodolgy a bit - its a big subject!

[wirgin blew]

At my last CRM course with Virgin Blue they discussed the possibility of moving to an assessment based course as CASA would like to see this happen at the moment. CASA looked at bringing it in a few years ago but had a change of heart. However with the recent change of government in Australia it has been put on the agenda again.
The big question was how do you assess it? If it is done by instructors how do you ensure that they maintain a level playing field. One persons opinion can be vastly different from another's.

[Non-PC Plod]

You may be surprised. The JAR-Tel project was designed to test NOTECHS by taking 85 different pilots from different operators in different countries around Europe. They were shown a series of videos of simulator scenarios and asked to rate the crew as depicted. There was a very high level of inter-rater agreement, not just on the final outcome pass/fail, but also on the areas of concern.
I have used these same videos on CRMI(Line) workshops, and I can say the same - generally people recognize dangerous behaviour when they can see it from a detached position.
Obviously these videos are scripted, so slightly artificial, but equally, real simulator recordings could be used if you had a database of sufficiently interesting ones to choose from.

[Angels 60]

No doubt , interactions between Capt and FO have created such distractions in the cockpit that have led to accidents. In many cases had there only been one hypothetical pilot, these issues removed, the accident would have been averted.

While not a gauge or to debate single vs crew safety issues, I am curious if CRM has created more training, more vague goal posts for pilots to attain, when the emphasis should be on solid pilot ability to accurately point out a problem and take corrective action. CRM seems to strive for a rather well constructed, psychologicaly manipulated, well thought out, emotionaly sensitive statement to the effect 'Hey capt, we are going to crash, but I want you to know, I don't want to hurt your feelings, or make you feel any less confident in your pilot skills, nor do I want you to feel that I am questioning you experience, or authority...but in ten seconds we will all be dead...so just to be clear, please don't take this wrong....."

If the goal is to hire pilots that can fly a plane and recognize problems in the furtherance of avoiding accidents...does hiring based on interpersonal skills and proper trained attitude substitute the training and experience to simply know what to do, rather then train to 'negotiate what to do'

I would think the trend to hire 600 hr pilots at some airlines that attitude (Attitude is code name for = do we like your or not) is taking prececidence over ability in hiring decisions. I am not convinced that attitude will save a plane load of passengers as a substitute for years training, ability, judgement and experience.

[Non-PC Plod]

I think you can over-emphasive the touchy-feely, "dont hurt my feelings" aspect of this. I think realistically 99% of us approach it in a fairly practical way. Hopefully the operators we work for will have produced an SOP to help this. For example, a guide on how to intervene if there is a deviation from the correct flight path, graduating from the "Check Height" call, through to the "I have control" call.
There are huge variations in training standards, company and national cultures worldwide. As a simulator instructor, I have seen plenty of people who will either sit there fat, dumb and happy, watching the other pilot (or even watching themselves) crash the aircraft without being able to make any sort of decision, or they reach across the cockpit the second a malfunction occurs and switch off ,for example, the one good engine without any consultation. This is what we are trying to avoid with CRM. As long as we have crew who are reasonably assertive, and will speak to each other and work out some sort of solution to a problem, believe me we are 95% of the way there.

[Angels 60]

As a sim instructor do you think it's wise to hire low time guys that 'get along and go along' ...fit culture and all that..or just get experienced guys that can fly?

[Non-PC Plod]

I guess all high-time guys started somewhere as low-time guys. You just have to make sure you can mould them into pilots who will fit into the company culture, and that you pay close attention as to who is crewed up with the new guys to ensure that the right lessons are passed on.

[OnePercenter]

'mould them into pilots that will fit into the company culture...'

[Non-PC Plod]

Yeah, hopefully a "safety culture". If people are coming from a miltary background, or flying as a hobby, or from teaching at a club, the way you operate in the commercial world may need a very different mindset. The way you get that is from the example set by your colleagues.

[divinehover]

Thanks for all the input. Much appreciated.

[Centaurus]

While not a gauge or to debate single vs crew safety issues, I am curious if CRM has created more training, more vague goal posts for pilots to attain, when the emphasis should be on solid pilot ability to accurately point out a problem and take corrective action. CRM seems to strive for a rather well constructed, psychologicaly manipulated, well thought out, emotionaly sensitive statement to the effect 'Hey capt, we are going to crash, but I want you to know, I don't want to hurt your feelings, or make you feel any less confident in your pilot skills, nor do I want you to feel that I am questioning you experience, or authority...but in ten seconds we will all be dead...so just to be clear, please don't take this wrong....."



Well put..CRM as a terminology is one of the greatest cons foisted upon the industry and an awful lot of money is being made by individual "providers" in promotion of pyscho-babble..

[Non-PC Plod]

I am continually amazed at how often we keep hearing this. If you think CRM is rubbish, the chances are that you dont get it, because it is a no-brainer. The fact that you speak about "psycho-babble" probably explains why you dont get it - there are a lot of bad CRM trainers and facilitators out there, who give you the psychobabble, instead of the practical stuff which will save your skin. The only way round that is for operators to do some proper quality-control on CRM training, instead of ticking a box.

[tbc]

Hear, hear.:ok:

[A37575]

Colleague attended a CRM course at start up airline in Hong Kong a year ago. Besides seemingly endless power point pictures, one of the exercises designed to instill principles of CRM included the group being blind-folded then all shuffled around in a circle with hands on the shoulders of the bloke or blokee in front of you, then reverse the conga line then try to identify the person who was behind you after blind folds removed. The participants of this crazy farce had no idea what it was all about but the facilitators certainly knew... And what the hell that had to do with CRM is your guess. But the boxes were ticked and that's the main thing.

[divinehover]

Dear Picklepilot

Yes I do think two 10000hr piots need to attend. CRM has progresed well passed teaching people to be nice to each other. My last CRM course dealt with Risk management and also fatigue management for long haul pilots. Two pilots of the same experience could be on ef the most dangerous situation levels due to potentialy flat authority gradient. CRM is for every level of pilot. I don't think it should be assesed to back to my original post.

DH

[Non-PC Plod]

Picklepilot,

If they are capable and experienced pilots.... Yes, I think CRM is EXACTLY what they will be using to deal with a problem in the cockpit! I would like to imagine that they would have extremely good CRM. I.E: leadership, management, decision-making, situational awareness, communication skills, cooperation etc. Why would they not use these CRM skills because they have 10000 hours ?????? However many hours we have, only the most arrogant will claim that they have nothing left to learn or refine. Believe it or not, 10,000 hour pilots are subject to human error too.

[SNS3Guppy]

Picklepilot, known formerly as ssg the banned poster, makes frequent reference to his vast simulator experience. Microsoft simulator, most likely. A troll who offers nothing, he will soon be banned like all his other names.

You know if the airlines hired more experienced pilots vs marshmellows, this CRM stuff wouldn't be so neccessary.. CRM is a result of strong capts screwing up and the FO watches the capt fly it to the scene of the accident. Hiring better pilots precludes this whole issue of needing CRM Do you really think two 10000 hour pilots are going to need CRM when trying to figure out what's wrong in the cockpit?

CRM isn't an airline concept, bright one. It's a crew concept, it's a single pilot concept, and it's applicable in every type of cockpit environment there is. CRM includes those outside the cockpit as well as in, it brings in the controller, it brings in everybody. It's heavily emphasized in the corporate cockpit, the fracitonal cockpit, the military cockpit; it's the basis of doing business.

Again, clearly you are without experience or knowledge in this area, and again, clearly you embarass yourself. You have an agenda to attack airlines, but know nothing about them, or about any other segment of the industry. You've tried to pass yourself off as a charter pilot, an airline pilot, a helicopter pilot, a corporate pilot...and you keep posting this ridiculous stupid statements. Can you not help yourself?

[Non-PC Plod]

Whoops, didnt realize that the gherkin had been previously identified as a git - was giving him the benefit of the doubt. Will try not to take the bait next time!

[justcurious]

While Canadian Rides do not have a CRM pass/fail assessment as yet, a recent change focuses on CRM attributes duinr the ride debrief.

Transport Canada has a DVD they released on assessing CRM, and the format for ride debriefing is the Nasa/UTexas format.

http://homepage.psy.utexas.edu/homepage/group/HelmreichLAB/

Nasa's paper on ride debriefing is here:

http://human-factors.arc.nasa.gov/flightcognition/Publications/IJAP.pdf

Nasa's rather impressive site for publications is here:

http://human-factors.arc.nasa.gov/flightcognition/publications.html

[Rightbase]

Would it be simpler to pick crew who are objective and not easily offended, rather than picking crew members who are over-elaborately trained to avoid giving offence?

If I need to learn something I want to be told about it, not have it hinted at and hidden in a long list of things I didn't do wrong!

OK - now I'll go back to my SEP.

[Roger Sofarover]

Rightbase

First post!

Would it be simpler to pick crew who are objective and not easily offended,

How do you do that?

[Non-PC Plod]

I reckon a CRM assessment would be a good start......DOH!:eek:

[Roger Sofarover]

Not quite a DOH:eek: situation Non PC Plod. So how do you assess people for selection to make sure they are not easily offended?? Swear at them in the simulator constantly? Word would soon get out that your company has a test for 'not being offended' at which point you could tell prospective employees in the sim (naughty comments deleted with a wry smile - JT), and they would turn and smile and say 'Roger, are you ready for the approach brief Captain?'

It is commonly accepted with CRM assessment that on any given sortie, one cannot test another on all of the behavioural markers. You can only assess those markers that arise during the conduct of the trip.

So back to you. DOH:sad:

[Non-PC Plod]

Roger,

I wasnt suggesting any Homer Simpsonisms for yourself, it was a poor attempt at incisive wit on a bored afternoon.
The serious point is, if non-technical skills are as critical to the safe outcome of a sortie as are technical skills (a fact borne out by LOSA data), then we should be looking at them during selection, line training, recurrent training, and indeed, all stages of the training process. Of course, you cant prove a negative, so if someone does not throw a wobbly on one occasion in the simulator, you cannot de facto say that they are necessarily an all-round good egg. But, if we maintain good training records, hopefully a pattern will emerge if someone has a tendency, say, to rush their checks and miss items, just as if they have a tendency always to be too high on the approach. In this way we can weed out the wrong candidates during the training process IF they display poor CRM skills, just as if they displayed poor handling skills. Of course you cant artificially test all aspects of someone's character, but if you put a crew under a bit of pressure, you can see if they are capable of working together effectively as a team, whether or not they like each other, are offended, etc etc. You can see their behaviour, and make an objective assessment about it.

[alf5071h]

Non PC Plod Re: … if non-technical skills are as critical to the safe outcome of a sortie as are technical skills (a fact borne out by LOSA data)…
One of the problems in aviation is that there are few ‘real’ facts, thus it would be interesting to see the LOSA data which you suggest supports this.
I would agree that NTS are very important and should be assessed (but never scored).
Whilst the ‘social’ NTS subjects might be judged on resultant behaviour – (normal, everyday aspects of human interaction), the cognitive subjects (situation awareness, decision making) are much more difficult to assess. As with LOSA, a weakness in the assessment is that the assessor cannot know what the subject is actual thinking – the aspects of knowledge, understanding, bias, or belief that s/he applies to the situation at the time of gaining situation awareness or making decisions.

With NTS ground based assessment, there is opportunity when debriefing to explore differing viewpoints, but this is always after the fact – in hindsight. Often the subject will recall a thought process matching the outcome, as opposed to what actually was considered at the time; the subject may even be unaware of the aspects which affected the thinking (the cognitive NTS).

On the other hand, LOSA does not involve any debrief and thus any conclusion is based only that of the observer and not necessarily what the subject perceived or deduced in the thinking process. Therefore LOSA data more often fits the facts after the event (outcome as opposed to process), which in identifying errors, is always the case, i.e pilots do not intend to commit an error (excluding violations), but the outcome (in LOSA) is judged as an error, without correlating the thinking that led to it. This masks any thinking skills, which could have been very good - the subject just used the wrong data.
This might be what you describe as ‘proving a negative’ or the subject ‘throwing a wobbly’; with this, I agree. However, I suggest that LOSA cannot prove that NTS are critical to a safe out come – they probably are, but still have to be proven by some other means.

With the use of trained assessors, and with knowledge of the problems above, then simulator training (LOFT) using controlled scenarios, stop and assess capability, and opportunity for instant debrief or repetition, enables a reasonable fair ‘outcome’ based assessment.
However, outcome assessment in line flying (route checks) is more difficult due to the lack of these controls. Also, it does not guard against both crew members simultaneously misperceiving a situation or suffering groupthink to arrive at an erroneous conclusion, e.g. weather is not as bad as indicated, or a minor organizational problem is discarded by a local work-around.
As Roger states “You can only assess those markers that arise during the conduct of the trip.”

[mesh]

CRM is not in isolation a pass/fail subject as far as the caa are concerned. If the flight crew do fail in one area of there LPC ie single engine go around and it is believed that poor crm brought this about then they can fail the LPC and the fail can be attributed to CRM. Ie if you had two pilots who got in the sim, didn't say a word to each other apart from sop's and were seen to pass all the usual technical aspect of an LPC then they could not be failed on CRM (that's the easiest way of thinking about it).

[Non-PC Plod]

Alf,

I agree- its very difficult to "prove" that NTS are critical to a safe outcome, but there is plenty of evidence to support the proposition. Every accident or incident that is put down to pilot error adds weight to this idea.

LOSA data is of course subjective, but looking at the papers produced by Helmreich et al (University of Texas), I think anyone can see that there is a lot of useful information available provided the observers are sufficiently well-trained and capable. If nothing else, it can help to identify trends if there is a history of, for example, unstabilised approaches using a particular procedure at a particular airport. There may be a plethora of different reasons why pilots have got the aircraft into this situation, but unless someone has flagged up the fact that there is a recurrent issue not related to aircraft serviceability, then it will not get addressed until it is too late.

As Mesh says, the CAA require technical consequences before NTS causes a failure on a line check. This does not, of course, prevent a comment or exploration of CRM issues in a debrief even if the line check was passed.
There is always a problem with debriefing an event, and having your episodic memory massage the facts. But, at least a well-facilitated debrief (particularly if video debriefing stations are available), will allow you to re-visit the events and allow you to formulate better ways of coping with particular situations.

There is always something to learn - there is always someone who has had an experience that you have no come across yet.

[alf5071h]

Non-PC Plod whether provable or not, we agree that NTS will aid a safe outcome. However our opinions diverge on the evidence of pilot error from incidents, accidents, or LOSA.
Many accident reports (perhaps the more older ones) conclude pilot error, or CRM failure as causal factors, but these terms only state the outcome and not the reason for it (the old way of thinking about human error – Dekker).

Many of the errors reported by LOSA are outcomes; they do not uncover the underlying reasons for the behaviour.
Your example of unstabilised approaches from LOSA is a record of what happened; I have yet to see reasons why they happened. In one instance crews might be overly influenced by organisation pressure for on time arrivals or ATC requests (misappropriated goal), which could be considered NTS items. Alternatively, the approach procedure, aircraft configuration, weight, wind, rules / procedures may inevitably result in a fast approach, which being outside of the pilots control should not be a NTS issue – it’s a system / organisational weakness that require fixing. [You could assess the ability to identify the reasons for the fast approach and the attitude towards submitting a safety report – but that might be unfair].
I suggest that the majority of LOSA reports involve the latter situation, a system or organisational culture issue which result in violation. This is not to say that there are many occurrences of ‘press-on-it is’ (failure of cognition), but without discussion how can these be identified? Trained LOSA observers can help, but they are equally subject to bias and misperception. Is the inability to differentiate the unprovable aspect of cognitive NTS and a major weakness of LOSA?

As stated previously, NTS assessment has the advantage of discussion and to some degree control of perceptual or systems issues in ground training, but not so in airborne assessment.
Thus in-flight, if a fast approach is deemed a technical consequence, and that the underlying reason is not identified (differing opinions), then is this a NTS failure?
Not in my book; it’s a ‘no test’, there is insufficient information on which to make an assessment, but plenty for both parties to learn from.
As de Bono states (many books) there are positive, negative and ‘interesting’ aspects in most situations; I think that the majority of interesting aspects involve cognitive NTS.

[Non-PC Plod]

Alf,

Given that we agree that good NTS are a positive thing, and I think we agree that the best environment to examine both our own and other people's NTS are in the controlled environment of the simulator.
But - that is surely not to say that accident investigations and LOSA are of no value because they record outcomes rather than underlying reasons.
Clearly not in all, but in many cases there is sufficient evidence to infer underlying reasons (whether these are down to NTS or wider-CRM issues, such as management, supervision, regulation etc) If this can spur us to re-examine the way we do things, it can surely only help to improve the safety of our operations in the future?

[alf5071h]

“ … not to say that accident investigations and LOSA are of no value because they record outcomes rather than underlying reasons”.

I agree, but accident reports must be used with care as these are critically dependent on the interpretation of the instructor/reader – the inference.
Many accident reports adhere strictly to the ICAO guidance in only presenting ‘factual’ evidence, thus many of the NTS aspects, being supposition, are avoided, or glibly labeled ‘pilot error’ or CRM. This is an opportunity for poor association and reduced training value. As stated previously there are few ‘facts’ in aviation and where given, often involve hindsight or other biases – in this instance, originating at the point of use.

However, recently some accident investigations have attempted to address the human issues. In some reports, ASTB in Australia have presented human factors in a speculative manner, with arguments for and against, and concluding on the balance of probabilities. This enables a reader to consider all of the possible contributions in the context of the accident, and although not ‘provable’ as fact, are of more value than just stating the outcome.
Other investigators have also attempted to grasp the human factors issues, most very successfully, and usually with informed or speculative discussion of the cognitive issues. These approaches will benefit NTS training and provide assessors with a wider baseline of context based behaviors – the real world.

There are fewer similarities with LOSA; the value of the report depends on the observer (incident report), but to date, the reporting method does not address the ‘factors’ or even speculate on NTS. Thus, I remain skeptical about use of LOSA in this area.

However, consider if we encouraged pilots to conduct their own (personal) LOSA during normal operations (threat and error management), this might identify some unseen threats, and, with honest self reflective assessment, pilots could improve their NTS – greater opportunity for practice.

For interest:-
What Can You Learn from Accident Reports? Page 7. (www.tc.gc.ca/publications/EN/TP185/PDF/HR/TP185E_3_2007.pdf)
Also: TRIS Online: The Effect of Accident Report Formats on Invulnerability and Hindsight Bias (http://ntlsearch.bts.gov/tris/record/tris/01044731.html)

Human Factors in the Investigation of Accidents and Incidents. (http://static.scribd.com/docs/ukwkn4ez9kwh.pdf)

Why System Safety Professionals Should Read Accident Reports. (http://shemesh.larc.nasa.gov/ssse/iet2006-reading.pdf)

[Non-PC Plod]

We are in agreement on that one - I wonder how often crews take the time for a debrief if they are not in a formal training environment. Is it the norm for everyone to take a couple of minutes to go through the flight and discuss whether there was anything they could have done better......or do they crack the paperwork and get off home?