Pilot skill assessment in simulators - Hit and miss
It is interesting to observe the different simulator tests applied by airlines when recruiting pilots. This article assumes that applicants have already undergone a battery of various aptitude tests to make sure they are the 'Right Stuff" before being tested for their flying ability in a flight simulator.
More often than not, the candidate will not have flown the type of aircraft represented by a particular simulator. The test could be in a 747, 777, 737, Dash Eight, or A320 or A330 full flight simulator; or even a Microsoft type flight training device similar to those found in shopping centres.
Candidates are usually given a briefing sheet covering sequences to be flown, along with suggested airspeeds, flap settings, thrust parameters and nose attitudes. This information may be sent to the applicant a few days before the assessment test, or given during the pre-flight briefing before entering the simulator.
If the latter, the candidate may typically have less than one hour to absorb all the relevant information. This may not only pertain to an aircraft he has probably never flown, but at the same time, he must listen attentively to the testing officer’s briefing. As one testing officer explained, this is to test the candidate’s ability to retain technical knowledge. If that reason is to be valid, one would think it would be a job for a qualified psychologist; not a simulator instructor.
Profiles flown vary considerably between operators. There is at least one instrument approach; usually with a go-around at the DA/MDA. Some profiles include climbing and descending at specified airspeed and rates of climb/descent. An engine failure may be simulated at some point during the test. This could happen either in level flight; or in the worst case, during the go-around from an instrument approach. The latter is a difficult task for a candidate not type rated on the type, especially as no time is given for the candidate to study the relevant flight crew training manual before the test.
Candidates may be required to intercept VOR radials using only RMI needle indications rather than course bars. Demonstrations of steep turns at 45 degrees angle of bank are often required. Candidates may be asked to fix present position on a chart by means of VOR/DME or NDB bearings with particular emphasis on the relationship between the aircraft position and altitude inside the 25 mile MSA surrounding the airport.
The complexity of the simulator test varies between operators. Generally, the aim is to see if the candidate can fly manually within instrument rating tolerances while displaying sound airmanship. Some operators require the test to be flown without the use of automation. That means no flight director, autothrottle or autopilot. Candidates coming off aircraft with sophisticated automatics where full use of automation is company mandated, often experience difficulty with hand flying on instruments with no help from these automatic features.
On the other hand, some operators may require candidates to use the flight director for the majority of the test including the takeoff and instrument approach. This, despite the fact a candidate may not have used flight directors in previous aircraft they have flown. Not only does the candidate have to quickly learn a new skill with regard to flight director use during the test, but then is supposed to learn and apply that skill while on the job, so to speak.
Some operators have an unrealistic expectation required of candidates being assessed in the simulator for an airline job. Not only having to be tested on an aircraft type they are unfamiliar with, but they may be faced with unfamiliar instrumentation, having to remember checklists or call-outs they have only seen an hour earlier in some cases, and even operating from an unfamiliar airport with little time to absorb all the information presented to them on an instrument approach chart. Throw in an engine failure on a strange aircraft and even a single engine go-around followed by a single engine circuit and landing and it is no wonder that many candidates fail to cope. At the same time the airlines complain of poor quality candidates and scream blue murder that they cannot find good pilots.
There are pilots who are 'naturals' able to cope with whatever is thrown at them during simulator assessments. They may never have flown a jet transport yet are fortunate to possess natural flying ability. These pilots are often characterised by their smooth handling of flying controls and power levers. Others can fly but lose concentration when trying to fly and navigate using radio aids while at the same time remembering all the call-outs and checklist requirements.
One operator is known for his propensity to deliberately ask mental arithmetic questions during flight by requiring the candidate to simultaneously calculate remaining endurance based upon current fuel flow while making a sector entry into a holding pattern on raw data. The purported aim being to load up the candidate while assessing his ability to multi-task while hand flying on instruments. A psychologist would doubtless shake his head in wonder at this gross example of amateurism at work.
Experienced testing officers can generally tell in the first few minutes after takeoff how the candidate will fare for the remainder of the test. As said earlier, some pilots are naturals while the majority will cope given proper training. Jerky flying caused by over-controlling can sometimes be put down to nervousness or test-fright. Or maybe the candidate always flies like that? If one aim of the assessment is to test the ability of the candidate to safely and smoothly fly on instruments, it can be counter- productive to load up the candidate with extra tasks including the remembering of specific call-outs and in-flight questions while the candidate has his hands full with instrument scans of a strange cockpit layout.
One airline assessment requires the candidate to fly a low-drag ILS where flap and speed changes are made all the way down from 3000 ft.to final flap at 1500 ft. Power is quite low as speed has to be bled off to a new figure every few seconds or so. Trim changes are frequent as are airspeed changes. When used, the automatics can help, but tests are normally conducted using manual flight on raw data with manual throttles. A low drag fuel savings approach profile requires flying skill, energy management and good judgement from even the most experienced pilot on type. For a candidate flying this type of approach without previous dual instruction on type, it is asking too much and is an unfair task.
Understandably, every operator thinks his assessment method is the best. That said, the wide variety of simulator assessment methods reveal some are over prescriptive and thus counter- productive. There is no such thing as the average candidate. Depending on an operator’s recruiting policy, a candidate could be an experienced former airline or military pilot down to an airline sponsored cadet. The assessment test profile should be uncomplicated, logical, and not encumbered with superfluous company checklists and call-out requirements. Configuration changes should be minimal. Some testing officers say they look for steady improvement throughout the session. Yet how can a candidate demonstrate improvement when second or third attempts at a sequence are normally not permitted and each sequence may be entirely different in execution to the previous?
Operators need to apply more thought and logic when designing simulator assessment profiles. At present we see candidates that have good instrument flying experience and all the attributes required of an airline pilot, yet fail a simulator assessment that is often thrown together and illogical in design.