The serious point …
Non PC Plod Re: … if non-technical skills are as critical to the safe outcome of a sortie as are technical skills (a fact borne out by LOSA data)…
One of the problems in aviation is that there are few ‘real’ facts, thus it would be interesting to see the LOSA data which you suggest supports this.
I would agree that NTS are very important and should be assessed (but never scored).
Whilst the ‘social’ NTS subjects might be judged on resultant behaviour – (normal, everyday aspects of human interaction), the cognitive subjects (situation awareness, decision making) are much more difficult to assess. As with LOSA, a weakness in the assessment is that the assessor cannot know what the subject is actual thinking – the aspects of knowledge, understanding, bias, or belief that s/he applies to the situation at the time of gaining situation awareness or making decisions.
With NTS ground based assessment, there is opportunity when debriefing to explore differing viewpoints, but this is always after the fact – in hindsight. Often the subject will recall a thought process matching the outcome, as opposed to what actually was considered at the time; the subject may even be unaware of the aspects which affected the thinking (the cognitive NTS).
On the other hand, LOSA does not involve any debrief and thus any conclusion is based only that of the observer and not necessarily what the subject perceived or deduced in the thinking process. Therefore LOSA data more often fits the facts after the event (outcome as opposed to process), which in identifying errors, is always the case, i.e pilots do not intend to commit an error (excluding violations), but the outcome (in LOSA) is judged as an error, without correlating the thinking that led to it. This masks any thinking skills, which could have been very good - the subject just used the wrong data.
This might be what you describe as ‘proving a negative’ or the subject ‘throwing a wobbly’; with this, I agree. However, I suggest that LOSA cannot prove that NTS are critical to a safe out come – they probably are, but still have to be proven by some other means.
With the use of trained assessors, and with knowledge of the problems above, then simulator training (LOFT) using controlled scenarios, stop and assess capability, and opportunity for instant debrief or repetition, enables a reasonable fair ‘outcome’ based assessment.
However, outcome assessment in line flying (route checks) is more difficult due to the lack of these controls. Also, it does not guard against both crew members simultaneously misperceiving a situation or suffering groupthink to arrive at an erroneous conclusion, e.g. weather is not as bad as indicated, or a minor organizational problem is discarded by a local work-around.
As Roger states “You can only assess those markers that arise during the conduct of the trip.”