Pilot skill assessment in simulators - Hit and miss
 

It is interesting to observe the different simulator tests applied by airlines when recruiting pilots. This article assumes that applicants have already undergone a battery of various aptitude tests to make sure they are the 'Right Stuff" before being tested for their flying ability in a flight simulator.

More often than not, the candidate will not have flown the type of aircraft represented by a particular simulator. The test could be in a 747, 777, 737, Dash Eight, or A320 or A330 full flight simulator; or even a Microsoft type flight training device similar to those found in shopping centres.

Candidates are usually given a briefing sheet covering sequences to be flown, along with suggested airspeeds, flap settings, thrust parameters and nose attitudes. This information may be sent to the applicant a few days before the assessment test, or given during the pre-flight briefing before entering the simulator.

If the latter, the candidate may typically have less than one hour to absorb all the relevant information. This may not only pertain to an aircraft he has probably never flown, but at the same time, he must listen attentively to the testing officer’s briefing. As one testing officer explained, this is to test the candidate’s ability to retain technical knowledge. If that reason is to be valid, one would think it would be a job for a qualified psychologist; not a simulator instructor.

Profiles flown vary considerably between operators. There is at least one instrument approach; usually with a go-around at the DA/MDA. Some profiles include climbing and descending at specified airspeed and rates of climb/descent. An engine failure may be simulated at some point during the test. This could happen either in level flight; or in the worst case, during the go-around from an instrument approach. The latter is a difficult task for a candidate not type rated on the type, especially as no time is given for the candidate to study the relevant flight crew training manual before the test.

Candidates may be required to intercept VOR radials using only RMI needle indications rather than course bars. Demonstrations of steep turns at 45 degrees angle of bank are often required. Candidates may be asked to fix present position on a chart by means of VOR/DME or NDB bearings with particular emphasis on the relationship between the aircraft position and altitude inside the 25 mile MSA surrounding the airport.

The complexity of the simulator test varies between operators. Generally, the aim is to see if the candidate can fly manually within instrument rating tolerances while displaying sound airmanship. Some operators require the test to be flown without the use of automation. That means no flight director, autothrottle or autopilot. Candidates coming off aircraft with sophisticated automatics where full use of automation is company mandated, often experience difficulty with hand flying on instruments with no help from these automatic features.

On the other hand, some operators may require candidates to use the flight director for the majority of the test including the takeoff and instrument approach. This, despite the fact a candidate may not have used flight directors in previous aircraft they have flown. Not only does the candidate have to quickly learn a new skill with regard to flight director use during the test, but then is supposed to learn and apply that skill while on the job, so to speak.

Some operators have an unrealistic expectation required of candidates being assessed in the simulator for an airline job. Not only having to be tested on an aircraft type they are unfamiliar with, but they may be faced with unfamiliar instrumentation, having to remember checklists or call-outs they have only seen an hour earlier in some cases, and even operating from an unfamiliar airport with little time to absorb all the information presented to them on an instrument approach chart. Throw in an engine failure on a strange aircraft and even a single engine go-around followed by a single engine circuit and landing and it is no wonder that many candidates fail to cope. At the same time the airlines complain of poor quality candidates and scream blue murder that they cannot find good pilots.

There are pilots who are 'naturals' able to cope with whatever is thrown at them during simulator assessments. They may never have flown a jet transport yet are fortunate to possess natural flying ability. These pilots are often characterised by their smooth handling of flying controls and power levers. Others can fly but lose concentration when trying to fly and navigate using radio aids while at the same time remembering all the call-outs and checklist requirements.

One operator is known for his propensity to deliberately ask mental arithmetic questions during flight by requiring the candidate to simultaneously calculate remaining endurance based upon current fuel flow while making a sector entry into a holding pattern on raw data. The purported aim being to load up the candidate while assessing his ability to multi-task while hand flying on instruments. A psychologist would doubtless shake his head in wonder at this gross example of amateurism at work.

Experienced testing officers can generally tell in the first few minutes after takeoff how the candidate will fare for the remainder of the test. As said earlier, some pilots are naturals while the majority will cope given proper training. Jerky flying caused by over-controlling can sometimes be put down to nervousness or test-fright. Or maybe the candidate always flies like that? If one aim of the assessment is to test the ability of the candidate to safely and smoothly fly on instruments, it can be counter- productive to load up the candidate with extra tasks including the remembering of specific call-outs and in-flight questions while the candidate has his hands full with instrument scans of a strange cockpit layout.

One airline assessment requires the candidate to fly a low-drag ILS where flap and speed changes are made all the way down from 3000 ft.to final flap at 1500 ft. Power is quite low as speed has to be bled off to a new figure every few seconds or so. Trim changes are frequent as are airspeed changes. When used, the automatics can help, but tests are normally conducted using manual flight on raw data with manual throttles. A low drag fuel savings approach profile requires flying skill, energy management and good judgement from even the most experienced pilot on type. For a candidate flying this type of approach without previous dual instruction on type, it is asking too much and is an unfair task.

Understandably, every operator thinks his assessment method is the best. That said, the wide variety of simulator assessment methods reveal some are over prescriptive and thus counter- productive. There is no such thing as the average candidate. Depending on an operator’s recruiting policy, a candidate could be an experienced former airline or military pilot down to an airline sponsored cadet. The assessment test profile should be uncomplicated, logical, and not encumbered with superfluous company checklists and call-out requirements. Configuration changes should be minimal. Some testing officers say they look for steady improvement throughout the session. Yet how can a candidate demonstrate improvement when second or third attempts at a sequence are normally not permitted and each sequence may be entirely different in execution to the previous?

Operators need to apply more thought and logic when designing simulator assessment profiles. At present we see candidates that have good instrument flying experience and all the attributes required of an airline pilot, yet fail a simulator assessment that is often thrown together and illogical in design.

Well, I think in most cases this is not about expecting too much. This is rather meant to see how the candidate behave in stressful situation. In other words - there is no expectation that candidate will fly perfectly but rather to see how he manages such sensory and mental overload.
There is a post about Ryanair thread for example where some candidate laments that he did everything perfectly yet RYR told him he failed.
So maybe his approach to problem solving was different than what RYR expected?
On the other hand I have see a post on, I believe, Emirates sim test. Somebody told that he was convinced he failed because he made several mistakes but they told him heh did fine.

&

My personal favourite is the manually flown sector entry and holding pattern without FMC. Can't remember the last time I did that!!!

Gee Centy I was exhausted at just reading what you typed never lone fly it!:-) It's obvious that these types of assessments gives the operators plenty of opportunities to wash out someone, it's to their advantage it seems. Flying a jet is actually fairly basic it's all the BS that surrounds the task on a daily basis from unrealistic requirements from ATC to the usual commercial pressures that are placed subtilty on the pilots along with fatigue, that wonderful word that the Airlines dodge by way of making our regulator think they have it sorted with FRM procedures in place.
I recall in the 80's I was told I wasn't suitable to fly a popular twin turboprop by a well known Melb based operator, I didn't have the skills. Advance a few years and I am about to retire with 1000's of hrs multi turboprop & turbojet behind me, yep the word 'suitable' is very subjective!
It's an industry now that is nothing like it was all those years ago! I wish future pilots all the best, they'll need it for sure!:-)

So what do you do when the automatics fail? I don't think it is unreasonable to expect a pilot to be able to fly with no automatics and have sufficient spare brain capacity to do something else, whether that is calculating a hold, answering irrelevant questions, or whatever. It's the kind of bread and butter stuff we used to do flying single pilot IFR right?

Centaurus obviously sees a lot of different test profiles and I have no doubt some are better than others. I've only done one recently and found it to be really quite good. It was in a type I'd never flown (B777) with as basic an instrument panel as possible. There were no automatics or flight director and a simple profile was briefed by the instructor. A second instructor was employed as my PNF / PM and would do what I asked, set headings, respond to SOP calls etc. I was encouraged to use whatever SOP calls I was comfortable with and he would respond as well as he could. The configuration was kept at something suitable for a speed of around 200 knots. The profile was a take off, climb to 5000, fly straight and level, once I was happy I did a steep turn and then after being asked for a self-critique and given a few tips I was invited to fly a second steep turn which was flown more accurately (showing improvement!). From there my offsider took over and I was shown a chart and asked to fix a position using the RMI needles. After my initial answer I was asked to be more accurate so I refined the position. After that I was asked to calculate and fly a hold followed by an ILS. All the while they chatted to me, except for when I asked them to "stand by" while I concentrated on something (LOC intercept for example).

I didn't practice for it. I looked forward to it and enjoyed it. I freely admit that I consider flying aeroplanes to be one of my "strengths" so when asked to show how I fly an aeroplane during an interview I was quite happy to oblige. I was much more concerned about the panel interview, but that wasn't nearly as bad as I imagined.

Centaurus criticises the idea of an interviewer assessing whether someone can retain knowledge or multi-task, suggesting a psychologist would be better equipped to do this, but misses the point that the interview is not intended to be a clinical assessment but rather aimed towards answering some questions about the candidate. Are they a good fit for the company? Could I sit in a flight deck with this person for four hours? Are they a training risk? Will the company waste money by hiring them only to have them not make the grade or require additional training? You don't need to be a doctor to get a feel for these things.

Centaurus also criticises the idea of getting pilots who are used to using automatics to fly without them and vice versa. I disagree. Someone who is used to flying with automatics and has had reasonable hands on skills in the past should be able to get back into the swing of things quite quickly. If they didn't have reasonable hands-on skills in the past and the company is putting some emphasis on basic flying skills then they should rightly be failed if they struggle. Likewise, a pilot who has never seen a flight director needs to be able to get used to it quite quickly. When the type rating starts you're not going to be given extra hours just because you don't know how to use a flight director.

Centaurus questions how improvement during the detail can be assessed when candidates might not be given second chances at a sequence and yet he provides the answer himself. A pilot who has not flown the simulated type before and has not seen the procedures and is maybe unfamiliar with the instruments etc, is able to show general improvement in how they fly the aircraft.

Creaaaaaak! Sound of Pandora's box opening. I think a useful test is getting the candidate to decipher the YMMM/YBBB Head Office NOTAMS in under 3 mins. How about a test which makes the candidates monitor the assessment pilot's flying to determine what he/she is doing wrong?

To be fair to the HR girl, she's just passing on information, at least that's how it worked with my feedback.

Candidates applying to join the RAAF as pilots. Are they required to be assessed in a simulator as part of the interview process? If so, what simulator type.

Most RAAF candidates would have little if any flying experience as it's not an organisation on the career radar once someone has their CPL in hand. Tests normally include eye/hand coordination, reaction times, ability to rapidly assess information, response to stress, intelligence, teamwork etc.

Already having a licence won't help if you don't meet their criteria in the other areas. MIlitary pilot selection is tough with a high chop rate for those not meeting standards during training.

If your pappy flies for a major you would have used the sim since you were 10 years old. Beats going in cold turkey having only ever seen a Baron cockpit.

CZ,

LMFAO. But really, do you ACTUALLY read them, or just say so?

The ground training syllabus pertaining to operation of the flight director system takes at least 2-3 hours and that is before getting into a simulator where all the modes are demonstrated and practiced.

Precisely. So is the point of the assessment test to assess a pilot's general instrument flying ability? Or to assess how quickly he picks up the all the modes of flight director operation - without having undergone a course of training? Not exactly a level playing field, don't you think?

I don't see the big deal really, if you want to fly for said airline then you jump through the hoops that they put in front of you. If they are not getting enough candidates who can actually pass the sequence then I guess they will change it. So if you don't come up to their standard on the day then I guess that is 'just life', we have all been there at one point or another.

Just the point of the OP's post which stated: "Operators need to apply more thought and logic when designing simulator assessment profiles. At present we see candidates that have good instrument flying experience and all the attributes required of an airline pilot, yet fail a simulator assessment that is often thrown together and illogical in design."

A novel assessment policy indeed - but horses for courses. However, it is often the case where there are just the two people in the simulator - the testing officer and the candidate. Under those conditions the testing officer will operate the gear, flaps and MCP panel on direction from the PF. There are no support calls since the LH seat candidate is being assessed on general flying ability possibly in a simulator type he has not flown before.. Support calls, particularly if the candidate is flying close to briefed tolerances, could be construed as a form of coaching. Coaching by the PNF is not permitted. This equally applies if the PNF is another pilot and not the testing officer.

Interestingly we had a case where the PNF would not shut up and was constantly "advising" the PF to the annoyance of the PF. It is a common habit with some pilots who mistakenly consider superfluous talking as evidence of good CRM. A more apt term is Fly by Mouth. The PNF held captain's rank which probably explained his propensity to micro-manage from the RH seat..