PDA

View Full Version : The simulator assessment test as part of airline interview. How fair?


A37575
30th Nov 2013, 12:49
Airline pilot interviews generally follow a common thread. Aptitude tests, interview and perhaps a simulator test. Depending on the airline, the interview may also include searching questions on technical knowledge. There are even books available that have a comprehensive list of technical questions a candidate is wise to study.

Cathay Pacific interviews for example include knowledge of high speed flight, meteorology, icing and engine theory as well as performance questions. On the other hand, Jetstar interviews do not contain technical questions but rather concentrate on human factors or how well do you get on with your fellow crew member. It could be that Jetstar assume that the qualification of CPL is enough proof the candidate has technical competency and that in the recruitment process, Human Factors knowledge is more important than overall technical knowledge.
.
Aptitude testing varies between airlines. One thing is for certain and that is fail the aptitude test and you don’t have a job. For example, Jetstar include a logical thinking exercise called flight planning where the candidate is given a typical scenario of weather, performance and several airports and chooses which airport is suitable.
One particular candidate, an experienced general aviation pilot, gained excellent marks in the Boeing 737 simulator assessment, only to be told he wasn’t the right stuff for Jetstar because he failed the flight planning scenario. It’s a good bet many more candidates were unsuccessful for similar reasons. They could fly the simulator well but picking the wrong airport in the flight planning game was considered a worse sin that picking one’s nose at the interview. Opinions vary on the effectiveness of aptitude tests. As a means of culling hundreds of airline applicants they have their place.

A 21 year old, 1500 hour general aviation pilot was called to Sydney for a Qantas interview. He was a grade one instructor as well as a current charter pilot on twins. He never got beyond the first day aptitude test. Fortunately he was later accepted by Ansett and was on the 767 when Ansett folded. Within weeks, he was accepted by a highly regarded major overseas carrier. Within three years he had an A320 command and a few years later an A330 command. Not bad for someone that never got past first base at the Qantas interview.

In the 1950’s, an 18 year old applied to join the RAAF. Part of the aptitude tests in those days consisted of sitting in a booth and moving a rudimentary Tiger Moth type joy-stick to “fly” a ping-pong type ball around a screen. Try as he may, he could not place the ball where it should be and was told he failed. Not only that, but the flight sergeant that supervised the test said the test was practically infallible and if you couldn’t fly the ball it was God’s own proof you lacked the brains and coordination skills to be a pilot. But good fortune was at hand.

The flight sergeant was friends with the 18 year old candidate’s father. They had been through the war together. As a favour to the father he fiddled the numbers and gave the lad a pass – but not without warning him he probably would be scrubbed on RAAF Pilots course because the aptitude test never lied.

Well, not only did the now 19 year old get his RAAF wings but he went on to fly Mustangs, Sabres,and Meteors and later as a Wing Commander commanded a C130 Hercules squadron at Richmond. Now here is the interesting part.

One of his postings was Commanding Officer of the RAAF Recruiting Centre at Rushcutters Bay, Sydney where he had initially joined up. He was now a Squadron Leader. One evening when the staff had gone home, he went into the aptitude test room and tested himself on the joy-stick ping pong ball contraption. He failed. He tried again and failed again. Read into that what you wish. But he sure could fly a Mustang...

Let’s look at the simulator assessment used by some operators. Jetstar candidates are tested in a Level 5 Boeing 737 Classic simulator. A real simulator - not a flight training device or FTD. Before that, the candidate is sent a list of airspeeds, power settings and nose attitudes to study. The test consists of general low speed handling, plus ILS and NDB approach which are flown raw data and no automatics. The instructor is responsible for selecting landing gear and flaps. The candidate is not required to brief the instrument approaches. The test is virtually flown as a single pilot operation. There are no standard airline call-outs. The instructor merely observes the general smoothness of handling as well as the instrument flying standard of the candidate. Most candidates are experienced general aviation pilots on light twins and some are from the regionals. Their flying skills are generally very sound.

Because there is no requirement to remember Boeing 737 checklists, company standard call-outs or conduct instrument approach briefings, the advantage of this policy is the candidate can give his full concentration to instrument flying without the additional hassle of remembering mandatory call-outs, and the parent company SOP.

Each company has different simulator assessment policies. Cathay Pacific candidates are tested on a Boeing 747 full flight simulator. A briefing sheet, with standard calls are provided. Keep in mind that, apart from cadets, the majority of applicants for Cathay Pacific, have extensive previous jet experience.

In Australia, there are several operators that use a Microsoft based generic Boeing 737NG flight training device as an assessment tool. The candidate is sent a blurb on the procedures to be expected and the test is based upon a two crew operation. The company supply a support pilot who is PNF. However, the company test includes much more than a demonstration of the candidate’s instrument flying ability. The candidate is told the primary objective of the simulator assessment is to evaluate Adaption and Learning Abilities, CRM and Communication skills, Decision Making Skills, Instrument Flying skills, Spatial Orientation and manipulative abilities. All this after a short 30 minutes briefing and 45 minutes in a replica of a generic Boeing 737cockpit..

The candidate is given a list of out of tolerance calls common to most airlines. He must first learn these. In addition to the challenge and response checklists which form the before take-off, after take-off and landing checklist which the candidate must commit to memory, the candidate is also tasked with remembering every one of the company published standard calls - of which eleven are listed. During the conduct of the simulator flying, the support pilot will call any time the candidate goes outside the published company standard tolerance. For candidates not rated on a Boeing 737, naturally there will be frequent minor excursions beyond company tolerances and this inevitably leads to a constant barrage of “support” calls.

It would be logical to expect type rated pilots to have all these standard operating procedures off pat and in fact similar procedures are common to most airlines with 737’s. But for someone who has not flown a real 737 to be expected to not only hop in a FTD 737 replica that does not have the fidelity of a level 5 simulator, and fly within command instrument rating limits straight off, and be assessed on their memory of company standard calls, and checklists, is surely too much to ask of a candidate – especially if the candidate has been used to flying single pilot IFR.

As all this is happening in 45-60 minutes, the assessing person observing the test is busy making notes on the candidate’s (quote) Adaption learning abilities, CRM and communication skills, decision making skills, instrument flying skills, spatial orientation and manipulative skills. That is some test…

How can an assessor, within one hour or less, conduct an honest and accurate evaluation of a candidate’s true ability to be a competent first officer, when the assessment requirements as laid down in the company SOP are so numerous?

Recently, a number of experienced 737 pilots flying for a South Pacific operator underwent an aptitude test conducted on the internet on behalf of an Australian jet equipped charter company. The cost of each aptitude test at around $170 was borne by the applicants. Every applicant failed the aptitude tests. They didn’t even get as far as a simulator test. Indeed, an astonishing situation.

Within the handling limit of a one hour simulator session, what should be the aim of the exercise? For instance, should it be simply to have a good look at the candidate’s general handling and instrument flying ability? In other words, Jetstar style test.

Or within the one hour limit, besides testing of manipulative ability in a type of simulator foreign to the candidate, should the candidate be simultaneously assessed on his knowledge of company SOP, checklists, flight tolerances, adaption/learning abilities, CRM and communication skills, two pilot crew procedures, decision making skills, instrument flying skills, spatial orientation and manipulative abilities? Is that not a bridge too far?

More to the point, is this not an unfair imposition on the assessing pilot, who must surely have great difficulty in making a fair and studied assessment of each listed item, without resorting to a box ticking exercise?
Comments invited.

Wally Mk2
30th Nov 2013, 14:11
.............sheeeez 'A3' I've got a headache after reading all that:)

It's obvious these days that Airlines don't want pilots per say they want puppets!!
Flying the plane is actually not considered a benefit what is though the ability to appease the HR guru's & the Uni freaks whom have more degrees than a compass!!! All the garbage that now goes with becoming an Airline pilot is out of control!!.
Beats me whom would want to these days as there's no security in it & there's not even any glamor in it anymore, it's really not a profession more like a trade with a white collar edge!

Wmk2

Jack Ranga
30th Nov 2013, 14:12
I thought if you bought the endorsement you're in?

The Green Goblin
30th Nov 2013, 15:37
Life wasn't meant to be fair.

If no one passed, or they couldn't get enough candidates, the'd rejigg the assessment protocol.

There will always be winners and losers in life, not all candidates are supposed to be pilots, and some very good ones are not what a particular company is looking for.

Try another company.

Oakape
30th Nov 2013, 19:15
Beats me whom would want to these days as there's no security in it & there's not even any glamor in it anymore, it's really not a profession more like a trade with a white collar edge!


Couldn't have said it better myself!

haughtney1
30th Nov 2013, 19:39
Assessments are a funny old thing, but I think the ever spreading tentacles of the 32 year old career HR person has a lot to do with it.
My first proper full time flying job consisted of a tech quiz, a group exercise, and a sim ride. No BS, no huggy fluffiness just good old fashioned common sense.
My second job interview consisted of a chat with the fleet manager, a cup of tea and the pointed question "are you the sort of chap that doesn't piss off people in confined spaces?"
The second job was for an airline that has been in existence for over 40 years, has a fleet of about 50 narrow body/wide body jets that fly all over the planet.
I compare that to where I am today, which by the way involved 4 days of testing, assessments, sim checks, medical checks etc etc etc etc and yet it seems there are an awful lot of feckwits that can get through these filters that I have to work with, when a 5 minute chat about nothing in particular should have told the interviewer all they needed to know..
After 15 years in the airline industry, I'm on the side of the argument that says that the assessments carried out are at best merely smoke and mirrors, and at worst deeply flawed.

Killaroo
30th Nov 2013, 22:08
Yes, the whole screening 'industry' is an insult to the proud profession this once was. Though I doubt it can be classed as one any more.

Would a Doctor be asked to demonstrate his surgical skills before being hired? I think not. Even though they too have their share of fraudsters and charlatans.

We used to be on a par with medics in terms of prestige and respect. Now we're all toe rags.

So 'group exercises' are considered kosher haughtney?
Gimme a fricken break. They're nonsense.

The problem is that too many of the wrong type of people have bought their way into the industry, and employers are suspicious of everyone now. Given the quality of some of the characters I've met of late I don't blame them. There are a lot of rotten apples out there.

As to the Sim assessment being viable in such a short time - I've done my share of Sim instruction with fresh recruits. The ones with quality stood out in minutes, and the hopeless cases likewise. I could tell in 5 minutes what I was dealing with. I'm sure any experienced pilot could. Not as hard as you'd think.

Ollie Onion
30th Nov 2013, 22:35
Having been involved in a few simulator assessments over the years for a couple of major airlines I think that these assessments are as good as anything for selecting candidates.

Due to the shear volume of candidates that apply to each airline the airline needs a process to whittle down the pile to a number that is easier to handle. When I started flying the selection process consisted of an HR person sitting down and reading through CV's, that ONE person would then select a bunch of interview candidates who would then meet the chief pilot for a one on one interview. He/She would select whoever they wanted based on that 20 minute interview. Some would say that this process was grossly unfair as there was no transparency to who got the nod. There was always lots of 'its just an old boys club'.

There is that famous urban myth of the HR person who had to shift through a pile of CV's one day to get the 10 required for interview, it is said that he took the top half of CV's and threw them in the bin, when asked why he said 'we don't want to employ unlucky pilots'!!

Whenever I ran an assessment in the simulator for a candidate I NEVER expected Command Instrument Rating standard straight off the bat, nor did I expect word perfect SOP's, my airline used a clapped out old Trident simulator for the assessments, this was chosen for the fact that no 'current' candidate would have time on this machine which meant you could truly judge their ability to perform. What we looked for was that over the 3 or 4 sequences that we put them through was 'improvement', I know of people that passed having crashed the thing on the first circuit but having flown a pretty dam good circuit after a couple of tries. This shows a capacity to learn and adapt which is the best thing a candidate can possess to obtain success in the upcoming type rating / training. As for the SOP's we didn't expect anything really, what we wanted to see was a candidate who would make full use of his resources including asking the PNF to carry out tasks thoughout that would take some workload off him/her, how they asked the PNF to do that was up to them.

As with all of these assessment tools their is an element of unfairness to them, I have failed soom candidates who no doubt could have passed with flying colours if they had 'slept better, not been so nervous, done the test on a tuesday instead of monday' etc, but hey that is life. Your example above of the guy who missed out on Qantas to go on to a full career to me just says one thing, you have to be persistent in this game, most of us have had knock backs in this game, just put it behind you and move on.

At the end of the day it is their train set and if they decide that the test to select employees is an 'egg and spoon' race then that is the way it is. Moaning about it will not help you win the race, just give it a good go. Ultimately the airlines do a pretty good job overall in selecting pilots, with all my years of experience I can honestly say there is only very few pilots who I would not like to fly with out of hundreds or thousands that I have flown with. The tests may be though on some, but they are passed by enough candidates to keep the flow of newbies going.

airdualbleedfault
1st Dec 2013, 00:06
For guys/gals with previous jet experience :
1) pi55 HR off, a bigger WOFTAM you'd struggle to find
2) actuallydo some due diligence, ask around, you'd be surprised what you can find out
3) ask candidate for a copy of previous sim checks
4) take candidate to pub for counter meal

Company saves 10s of 1000s of dollars per year and I guarantee you'd end up with less c*ck heads than most major airlines

DeltaT
1st Dec 2013, 07:36
Buzz Aldrin was the second man to walk on the moon, on 21 July 1969.
He originally didn't meet the NASA test pilot requirement.
That following year, 1963, NASA changed their screening procedure.

Ref: The Frost Interview

Exascot
1st Dec 2013, 08:37
I made a right old Horlicks of my CX sim check ride. Five days previous I had been flying the PM. He didn't criticise me as much as the interview board. If I had been current on type (L-1011) it would have been a doddle. I can't see the point when you have 1, 000s of heavy 4 jet time and an ATPL.

AviatoR21
1st Dec 2013, 10:07
Obviously if you can find that much info on the Jetstar process alone how can you not do well?....there is so much info out there for applying candidates that if you don't get through there is an obvious reason.

Shagpile
1st Dec 2013, 18:20
It's pretty simple really. Airliners and RAAF don't have enough resources to train/employ the entire population.

There is limited simulator/flight screening positions available. They need to cut somehow and each company has decided their own way.

I quite like the RAAF method that first filters for generic officer qualities, then pilot specific raw maths & mental DR skills, followed by flight screening. Being a training organisation it has the luxury of being able to disregard experience and recruit directly based on raw talent/potential. I'm doing flight screening currently and two weeks of flying is an excellent way of picking talent (or not), no matter the starting level.

mattyj
2nd Dec 2013, 02:51
When I got the call from the (very nice) hr guy at VA to say that my sim ride wasn't up to scratch but thanks for coming and try again next year, he asked "why didn't you buy a couple of hours in the sim like the other pilots?"

Wish I did now but at the time, my visa was at Max all up Weight..

Howard Hughes
2nd Dec 2013, 06:39
he asked "why didn't you buy a couple of hours in the sim like the other pilots?"
That in itself attests to the value of a sim ride... Finding out if someone is a dick head is far more important in my opinion! ;)

BPA
2nd Dec 2013, 08:23
Virgin Australia (when it was Virgin Blue) employed around 400 pilots without the use of a SIM check or HR involvement. The interview was with the Chief Pilot, his Deputy or the head of Check and Training and a former pilot.

Some got a few basic tech questions, but for most it was just 20-30 minute chat to see if you would fit in ie could they stand to be in the flight deck with you.
Once the interview was over you were told a yes or no with a day. Most of these 400 pilots now have a command, including B777 and A330 commands.

This process worked well, shame it was changed:ugh:

Biggles78
2nd Dec 2013, 12:52
In the early 80s when the major domestic airline in NZ still had the call sign National, there was a story floating round about 2 pilots applying to fly for them (likely the F27 to start with). Pilot 1 had 300 hours in a C172 and an IR, that's it. Pilot 2 had some 3,500 hours Multi Single Pilot IFR. Pilot 1 was hired. Pilot 2 was told he needed to go and build some more hours as he was a bit light on what was required.

When NAC was based in NZCH, the Canterbury Aero Club used to have a higher proportion of appropriately qualified pilots hired from their ranks. When head office was moved to Auckland the trend, after time, changed to where more Auckland trained (Ardmore) pilots would get the nod. Guess the ex aero club members on the Panel changed, eh. :oh:

Like to post above about the 18 year old who had his results fiddled by his war time colleague, it is quite often who you know and not what you know. Such it is in all walk of life.

I was doing my IR subjects with an Ansett Sim instructor who had the misfortune to lose his medical after reaching the multi IFR charter arena. He told me that in general most pilots treated him with the standard respect and he in turn treated them with fair check rides. However there were the tin skygods who looked down their nose at this "non pilot" and let their feelings known. What they didn't realise was that they always had tougher sim check ride than their polite colleagues. The man was a gentleman and reminded me of Peter Bini (who did my initial multi endorsement) and Lionel Taylor with whom I did the ground subject with (due to the Dept of Constant Name Change) not recognising my NZ CPL. Sorry, drifted off track. I guess what I am trying to say, haughtney1's post about seems to sum it up. I the "good old days" you got the job if the interview panel liked you. Now it seems to be you get the gig if the beancounters like you.

There was a thread last year (?) about an experienced GA guy who did the J* TR, passed the course but failed the line check because although he could hand fly an approach from 10,000 feet, due to his training FUBAR he got a bit behind on the automation on approach. Surely the automatics remedials would be easier to rectify than trying to fix a manual control manipulation. I sure know who I would want at the joystick if things went south.

[DISCLAIMER] I am a non airline pilot (please note NOT driver) rather a humble former lighty pilot (again, NOT a driver)

waren9
2nd Dec 2013, 16:08
In Australia, there are several operators that use a Microsoft based generic Boeing 737NG flight training device as an assessment tool. The candidate is sent a blurb on the procedures to be expected and the test is based upon a two crew operation. The company supply a support pilot who is PNF. However, the company test includes much more than a demonstration of the candidate’s instrument flying ability. The candidate is told the primary objective of the simulator assessment is to evaluate Adaption and Learning Abilities, CRM and Communication skills, Decision Making Skills, Instrument Flying skills, Spatial Orientation and manipulative abilities. All this after a short 30 minutes briefing and 45 minutes in a replica of a generic Boeing 737cockpit..

The candidate is given a list of out of tolerance calls common to most airlines. He must first learn these. In addition to the challenge and response checklists which form the before take-off, after take-off and landing checklist which the candidate must commit to memory, the candidate is also tasked with remembering every one of the company published standard calls - of which eleven are listed. During the conduct of the simulator flying, the support pilot will call any time the candidate goes outside the published company standard tolerance. For candidates not rated on a Boeing 737, naturally there will be frequent minor excursions beyond company tolerances and this inevitably leads to a constant barrage of “support” calls.

i'd be glad to fail that. imagine what working for these tools must be like.

a 20min sim, hand flying some basic ifr manoeuvres on instruments, raw data is all you need to tell you if the bloke knows which way is up.