I previously posted this on the related software thread - it might provide some additional context to the discussions that are going on here:
I stumbled upon an interesting case study on the use of memory items and (subsequent) checklists using 15 volunteer current 737 line pilots that were present at a crew base of a major US airline. Half captains, half FO’s. Half of all pilots had (varied) military experience. Pilots reported being trained in both 737 Classic and 737 NG. The 737 Classic flight deck was used.
I summarize ...
Each scenario began by describing a normal flight situation, and then interjecting cues that suggest a particular failure. Subjects were asked to react to the cues as they would inflight, performing any procedures they felt were necessary. The participants were provided with their airline QRH, and were allowed to select the checklist they felt was most appropriate for the situation.
Five non-alerted abnormal procedures that contain memory items were used.
1. aborted engine start ----- correct checklist by 10 of 15, ..
2. engine limit/surge/stall - correct checklist by 2 of 15, ..
3. rapid depressurization -- correct checklist by 14 of 15, two pilots added memory item steps,
4. runaway stabilizer trim - correct checklist by 14 of 15, four pilots added memory item steps,
5. dual engine failure ------ correct checklist by [12] of 15, this scenario had most errors in memory item steps of all 5 scenarios, multiple item steps were added by pilots.
The pilots in this study demonstrated a tendency to fixate on the most prominent cue, and perform the checklist appropriate to that cue. However, a thorough analysis of the situation can reveal that the single most prominent cue does not always lead the pilot to the correct checklist. There were 23 checklist selection errors. With three exceptions, the errors appear to be caused by the pilots’ fixation on a single cue.
There appear to be consistent patterns in the observed checklist step errors. Many of the ‘adding item steps’ errors appear to result from the pilots’ creativity (read: (experience based) troubleshooting) in dealing with an abnormal situation. It was observed that many pilots perform steps in addition to what was required based on their understanding of how the airplane systems functioned, even though their understanding of the systems may be incorrect.
... End of summary.
You might think on reading the study that: A single study aimed at being as realistic and objective as possible does not present a ‘truth’. But, that not everybody is perfect, might be a reasonable conclusion.