EASA to create a massive question bank?

[IO540]

This (http://easa.europa.eu/rulemaking/docs/research/Final%20Report%20on%20publication%20assessment.pdf) ludicrous study, which looks like a stunning waste of EU taxpayers money, suggests they are trying to get away from publishing a workable QB - either by not publishing a QB, or by creating a QB which is too huge to be of any use for revision.

This is another reason to get the present "JAA" exams in the bag sooner rather than later, because despite being a good 90% useless dross they are at least doable using concentrated QB revision, over a week or two.

Without a usable QB, the IR or ATPL exam study workload goes up a lot - perhaps 5x to 10x.

All this applies not just to the ATPL but also to the proposed new "CBM" IR.

[proudprivate]

What makes you deduce that they are trying to get away from publishing a workable QB - either by not publishing a QB, or by creating a QB which is too huge to be of any use for revision. ?

Here are my comments about this "study" :

1. EASA is wasting our money handing it out to some Swiss consultants to collect some undergraduate level research and to repeat a few experiments of which the results would be obvious from the start.

2. Several important points are missed or wrongly interpreted by their "experts" :

- the use of a question database is an important guidance to let the applicant know what knowledge (depth) is required to comply with an "ability to demonstrate theoretical knowledge". The fact that the applicant familiarizes herself with this level of depth through studying of a question database should be welcomed.


- the classical example of the FAA "calculation questions" being answered in just a few seconds needs to be seen in the context of the learning process. While reviewing the following test problem on the FAA Commercial Exam :

(Refer to Figure 12 in FAA-CT-8080-1C)
GIVEN:
Pressure altitude 18,000 ft
Temperature -1°C
Power 2,200 RPM - 20'' MP
Best fuel economy usable fuel 344 lb
What is the approximate flight time available under the given conditions? (Allow for VFR day fuel reserve.)

A) 4 hours 50 minutes
B) 5 hours 20 minutes
C) 5 hours 59 minutes

an unobservant student would look at the power settings table and read for 2200 RPM @ 20" Manifold Pressure in the column for 20° above standard temperature : 59 PPH. Dividing 344 lbs by 59, substracting 0,5 hours of VFR reserve, she would obtain 5 hours and 20 minutes, the incorrect answer. The correct answer is C (because a note says that for best fuel economy at 70% power or less - which is the case here - one should operate at 6 PPH leaner than shown in this chart).

After having reviewed this question beforehand, the student would probably answer the question in the test in less than 30 seconds, because she had already reviewed it. However, thanks to her preparation, the learning objectives (computing endurance whilst considering all the fine print of the performance tables) is clearly achieved. No sensible student in aviation would learn the answer 5 hours 59 minutes by heart without at least attempting to understand where the answer is coming from.

The fact that memory helps a student sitting exams does not imply that the student does "not rise above the rote level". Even if one were to fear this, one could possibly extend the number of questions and label them clearly as "equivalent", so that the student could work through the type of question by just solving 3-5 typical problems (out of a total maybe 20 or so in the database).

- The researchers rehash some well known results on timely exponential decline in theoretical knowledge, which they apply to rote memory of multiple choice questions but which are just as well applicable to retention of any theory memorization process.

We all recall the horror of "Human performance and limitations" questions, but because of the irrelevance in practice, we loose that knowledge exponentially fast. The authors do remark that "They found that there is a trend for pilots to remember information that is relevant to their day-to-day operation of aircraft, and that irrelevant knowledge is preferentially forgotten".

But they then go on to argue that because rote memorization is a bad way of storing knowledge, everything should be put to work to avoid students using this methodology. The latter reasoning is unfounded. After all, there is no such thing as an "in depth study" of Air Law for instance. Studying a part-FCL text can be just a numbing as studying multiple choice questions about that same part FCL.

- an inordinate amount of pages is wasted on the statistical analysis of their new experiments with university undergraduates. Some of the stats is downright laughable, such as coming up with an empirical formula for test accuracy as a function of the question database size.

- after this and without much scientific evidence in support, the researchers the argue in favour of increasing the number of questions in the test database. Although they purport to be "aviation consultants", no analysis is made of the learning processes associated with pilot training.

3. Probably preaching in the desert again, but some recommendations could be :
- publish (and make freely available) all question databases
- avoid trick questions, unless they convey a specific and clear learning objective.
- avoid all subjects that are not directly relevant to flight safety
- publish (and make freely available) reference manuals that can serve as definitive reference for study
- make sure that the context of each question is clearly understood / outlined, even if this comes at the cost of making the question longer.
- formulate each question so that eliminating the incorrect answers and giving the correct achieves by itself a learning objective.
- do not use the theoretical knowledge exam as a eliminating factor. Rather, encourage students to use it to make the flight training experience more rewarding. Instead, use the flight examiner to verify up to speed practical knowledge.

[IO540]

The JAA ATPL QB was obtained under the FOIA in Europe, several years ago.

Prior to that, each FTO obtained their own version by assigning students to memorise a few questions each, and they had a man standing outside the exam room noting them down as soon as they left the exam room.

I read elsewhere that EASA had proposed that no QB should be published in this case.

Also, they paid 50 euros to this firm (http://www.lplus.de/) per question to generate the QB, and a big QB would cost a lot to generate :)

[RTN11]

I think the whole syllabus should be reviewed, leaving the more relevant sections, which should then be tested more thoroughly.

The fact is that in the current system people use the question banks to memorise the answers to recognisable questions to pass the exams. When they have finished the groundschool course, they then dump most of this knowledge thinking that most of it is useless - mainly because there is so much crap you have to learn that has no real relevance to anything.

If the syllabus was more relevant, and the exams required you to know that syllabus properly, then the quality of pilot would be better.

Simply adding more questions won't help, as the "unofficial" question revision bank will still be generated, it will just be harder to memorise them all.

P.S. shouldn't this be under profesional training rather than private flying?

[proudprivate]

Also, they paid 50 euros to this firm per question to generate the QB


Is that accurate ? That is seems like a very expensive price to pay per question. I recall that the French - although admittedly the questions are 90% bollox too - paid only € 11 per question.

The mistake is obviously to work on a "per question" basis. The result has been creative tinkering of trick questions of the same concept all over again, missing the learning objectives by a mile.

Here's a secret : there is nothing wrong if students learn a concept by memorizing a set of questions about it, provided the questions adequately cover the learning materials you like students to take in.

But companies like LPLUS GmbH obviously don't care about the cost to the aviation community, as long as they get better from it. ICAO Language Proficiency anyone ?


P.S. shouldn't this be under profesional training rather than private flying?

It affects us private pilots too, especially in the context of the ability to travel as opposed to burning holes into the sky on a sunny day in uncontrolled airspace (the latter being EASA's vision on General Aviation)


I think the whole syllabus should be reviewed, leaving the more relevant sections, which should then be tested more thoroughly.


What is this obsession of some many about "having to test more thoroughly" ? How thorough do you want to question a candidate PPL on dead reckoning and wind triangles ? On VFR visibility and cloud separation requirements ? On accident reporting procedures ? On VOR navigation ?

Can you name me 5 topics where evidence shows that students do not grasp a particular subject where changing the question database would make a positive contribution ?

I agree with your point that the syllabus should be reviewed and some topics ditched entirely. Also, depending on the study level, some topics, although relevant to the airline transport pilot are nonsensical to the private pilot or the commercial pilot.


Simply adding more questions won't help, as the "unofficial" question revision bank will still be generated, it will just be harder to memorise them all.

Quite so. As a matter of fact, about half of the questions should be binned at once and another quarter reformulated. And it shouldn't be left to University spin off generalists like LPLUS, who wouldn't understand the first thing about aviation learning objectives.

A good start for generating relevant questions would be to look into recent accidents' reports and convert the topics pertaining to them (weather, aircraft loading, VFR rules, traffic patterns) into useful questions. This in contrast to looking at some silly paragraph in the rulebook, smoking an illegal substance and then writing down one's thoughts.

[Katamarino]

P.S. shouldn't this be under profesional training rather than private flying?

Do you work for EASA by any chance? Private Pilots find IRs useful too, you know; despite the view of the European aviation authorities that an IR should only be for airline pilots with large bundles of cash :rolleyes:

[S-Works]

So what if the actual questions are not available. Most training providers already have there own question banks for training that are in the same format as the JAA questions. If you have learnt the material, practiced the questions that accompany it then you don't need to know the actual answers to the actual questions.

Unless of course youa re always looking for a curve on a straight road?

Despite the protestations of many, there is a lot of relevant stuff in the CPL/IR/ATPL theory that may not seem to be pertinant at the time but often comes in handy later as you progress through your career.

If as much energy was spent on the study as is spent on trying to find shortcuts to avoid it then the whole discussion would be a non event!!

[Whopity]

The JAA attempted to set up a comprehensive question bank, but in their 10 years of existence failed to achieve their goal, and during that time the standard of questions deteriorated markedly. Like so many education systems, the exams have simply become a target to be achieved, their relevance with the end product has all but totally disappeared. In the last 12 years there has been no correlation between the theoretical exams and the flying training syllabus, which itself is totally vague and left to the schools. The Theoretical Training is based around Learning Objectives that were determined from the questions, all totally arse about face!

At the end of the day the purpose of the exams is to determine if a pilot knows the relevant information to perform his role as a pilot. How he learns that information is irrelevant. To mount a survey that examines the pitfalls of the multiple choice examination system, which we all know only too well, seems totally out of kelter when the real issue is: "what does a pilot need to know" and how are we going to integrate that knowledge into a course of flight training? If we do that, testing the knowledge is quite straight forward. The current administrators are so far removed from the process, and knowledge base, that they haven't got a clue what questions to ask of those they pay to produce reports that are at best irrelevant, and in most cases total garbage.

The only justification for a CQB is to standardise questions between States and to eliminate the need for professional examiners; the demise of such people has led to the inevitable reduction in quality and relevance of examination questions. Schools now simply teach students to pass the exams; that's what gets them a licence, not having relevant knowledge.

[IO540]

P.S. shouldn't this be under profesional training rather than private flying?No, because the Prof Training forum is moderated by a bunch of FTO owners who severely jump on any discussion of the bollox theory :)

Also almost no private pilot is doing the ATPL theory :)

there is a lot of relevant stuff in the CPL/IR/ATPL theory that may not seem to be pertinant at the time but often comes in handy later as you progress through your career.Only if you fly DC3s in the Republic of Upper Volta (http://en.wikipedia.org/wiki/Upper_Volta).

The stuff gets forgotten after a few weeks, if not days, anyway.

Most training providers already have there own question banks for training that are in the same format as the JAA questionsOnly because they got them using the two methods I described. The process takes time. It took several years for the FTOs to "develop" half decent QBs, after JAA came in.

A part point of my original post here was to suggest that people working towards the "Euro IR" might consider grabbing what is on the table right now i.e. the "new" "slightly reduced" JAA IR theory which came in c. August 2011. FTOs such as CATS should have a QB which reasonably corresponds to that TK syllabus.

I would not wait for the TK being developed for the "FCL008/CBM" IR which is due sometime in the next 1-2 years (and which might get squashed as a result of FTO objections anyway).

Those wishing to get themselves a Euro IR, as an insurance policy against whatever EASA might do to shaft US licensed pilots, are IMHO best advised to do the currently available TK because it is supported by a workable QB, and then pop down to e.g. Spain (http://www.fly-in-spain.com/en/) and do the training there.

[pegasus-9]

Whilst some may bemoan more questions, you have to look at it from the point of view, if I know the subject then I will pass whatever is put in front of me.

In the days before the internet, one had to learn the subject matter and even today should be able to remember the important points that the instructor tried to get in your head, which at some time may assist you in getting out of a problem.

Unfortunately these days one sees to many student pilots, with a laptop in front of them relying on what is on the web to get them their pass, rather than revising.

May be there is some good coming out of EASA.

[IO540]

The problem, pegasus-9, is that most of the TK is not related to contemporary aviation.

Look at the crap in HP&L - all those phychobabble questions based on long discredited behavioural psychology models. Look at the crap in Aircraft Tech, which is stuffed with 1950s RAF radar set stuff.

Actually HP&L is a good example of what EASA want to do. The QB is over 800 questions (one of the biggest QBs) but the exam is just 24. They had to do that, because they knew it was mostly garbage and they didn't want people to learn it the only practical way which is by memorising the questions.

If the stuff was relevant, few would complain.

And pilots who have previous IFR experience would pass it straight off, without much if any revision. As it stands, a pilot with any amount of IFR experience will get roughly 50%.

[bookworm]

At the end of the day the purpose of the exams is to determine if a pilot knows the relevant information to perform his role as a pilot. How he learns that information is irrelevant. To mount a survey that examines the pitfalls of the multiple choice examination system, which we all know only too well, seems totally out of kelter when the real issue is: "what does a pilot need to know" and how are we going to integrate that knowledge into a course of flight training? If we do that, testing the knowledge is quite straight forward.

What criteria would you set to answer the question "what does a pilot need to know"? In other words, how would you decide what should go in the learning objectives?

[proudprivate]

Whilst some may bemoan more questions, you have to look at it from the point of view, if I know the subject then I will pass whatever is put in front of me.

You're deliberately provocative, aren't you ? Otherwise I would think you need your head examined. Or maybe you work for the CAA ?

We are not bemoaning more questions, we are bemoaning
1) the de-correlation between questions and learning objectives; or put it differently, the absence of relevance in the TK questions themselves.
2) the absence of consistent study materials that help us pass the exam if we study them.
3) that EASA wasting taxpayers' money on substandard consultancy
4) that National CAA's (and now possibly EASA) are wasting money to set up question databases that still bear no relevance to the subject matter / learning objectives


In the days before the internet, one had to learn the subject matter [...]


No comment on your age here :eek:. But today one should learn the subject matter too. The trouble is that learning the subject matter does not allow you to pass an exam composed of ill formulated questions, most of which focus on irrelevant trivia.



So what if the actual questions are not available.


Steve, you're not reading my previous post. Actual representative sample questions (preferably the whole QDB) needs to be publicly available because it
is important guidance to let the applicant know what knowledge (depth) is required to comply with an "ability to demonstrate theoretical knowledge". The fact that the applicant familiarizes herself with this level of depth through studying of a question database should be welcomed.


Most training providers already have there own question banks for training that are in the same format as the JAA questions. If you have learnt the material, practiced the questions that accompany it then you don't need to know the actual answers to the actual questions.


You do if the question database diverges in style / subject matter / etc... from the actual questions. Or are you advocating a deliberately inefficient learning process ? Want to make more money teaching "ground school" ? Sounds like yet another FTO protection scheme to me.

When the learning process is deliberately made inefficient (like with the medieval "guilds"), the cost to society (who eventually ends up paying through reduced productivity) quickly mounts.


Actually HP&L is a good example of what EASA want to do. The QB is over 800 questions (one of the biggest QBs) but the exam is just 24. They had to do that, because they knew it was mostly garbage and they didn't want people to learn it the only practical way which is by memorising the questions.

That is the point. A real life Aviation Administration/Safety Agency would stop and think about whether a specific topic is relevant to contemporary flight safety and only then include it. But again they seem to confuse safety with politics and FTO lobbying.

[proudprivate]

What criteria would you set to answer the question "what does a pilot need to know"? In other words, how would you decide what should go in the learning objectives?


That is straightforward : start from all known accident and incident causes and other mishaps and build questions from that. If it is unrelated to any of those, it shouldn't be in the syllabus or question database.

[BillieBob]

What criteria would you set to answer the question "what does a pilot need to know"? You carry out a formal training needs analysis - something that has never been done for civil pilot training. It would also cost significantly less than the utterly pointless Moebus study (which achieved no more than the production of a 62 page report that concludes that rain is wet) and the inept rubbish produced by LPLUS for the CQB.

[IO540]

That is straightforward : start from all known accident and incident causes and other mishaps and build questions from that. If it is unrelated to any of those, it shouldn't be in the syllabus or question database.

I wouldn't say it should be just that.

The IR TK ought to contain stuff which is relevant to the IR procedures.

Look at the FAA IR TK. It is mostly (maybe about 80%) relevant. So why does Europe make such a hash of it, recycling garbage which was partly "lifted" from stuff discarded by the RAF and bashed into various "approved" FTO study syllabi.

I think the answer is partly that most of the people behind the TK are not pilots and never were pilots. Some were quite obviously ATCOs. But most were just desk drivers.

Another reason is that nobody gives a damn. Most people sitting it has little or no aviation knowledge (at the time they do it) and the FTOs just push people through it ASAP so they can get £££ off them for flight training which is their real Raison d'être. Some people might not like it but they are not going to rock the boat. The punters then go on to get type ratings where they learn the stuff actually needed, and they spend their initial years with a senior ATP in the LHS who teaches them the rest (hopefully).

A related issue with the JAA IR TK is that has accreted stuff which is nothing to do with the IR.

One should be teaching stuff like safe ways to fly a plate. You have to respect the MSA circle, etc. Specific meanings of ATC clearances e.g. the vertical and lateral clearances are totally separate. But some are implied e.g. once "cleared for the approach" you are authorised to descend to the platform altitude (but still have to watch the MSA :) ). Etc.

[deefer dog]

60% of the actual questions in a recent exam were made available to a candidate known to me, in advance of the exam he took. There was a price to pay of course, and the exam took place outside of the UK, but the system has leaks and one does not need to look too far east, or south, to find them. Who said that the financial depression was all bad?

Who in their right mind would contemplate taking the exams in the UK, France or Germany when soooooo many other avenues are open to those who have the gumption to seek out the shortcuts that EASA will never manage to plug?

For goodness sake, attempting to standardise the scrutiny of any examination process is akin to aligning France, Italy and Spain with common rules of food safety hygiene!

[24Carrot]

Who in their right mind would contemplate taking the exams in the UK, France or Germany when soooooo many other avenues are open to those who have the gumption to seek out the shortcuts that EASA will never manage to plug?

Hmmm, tricky one.

Maybe people who can pass the exams without resorting to bribery?:uhoh:

[proudprivate]

The IR TK ought to contain stuff which is relevant to the IR procedures.

Almost all IR procedures are related to one or the other incident that has actually happened (radio failure; (not) flying a compulsory hold when not cleared for an approach; excessive speed in a hold; flying the wrong entry pattern; taking off without a textual description of a DP; etc, etc...

But I concede that my answer is oversimplifying, more of a reaction to bookworm inferring that it would be difficult to discern what the learning objectives should be.

There is some mileage in performing a formal training needs analysis. I recall that after the Chernobyl / Pripiat accident, the International Atomic Agency did something similar for all functions in a Nuclear Plant (i.e. Supervisor, Technician, etc...). The caveat is that such studies quickly become abstract and its findings distant from and difficult to relate to existing procedures.

I also fear that such studies can easily be poisoned by special interests. FTO's, Pilot Unions, Aviation Medicine Specialists, all want to overload the task list far beyond what is necessary to safely navigate the skies.

But the overall idea is a valid one : by interviewing experienced private, commercial, airline transport pilots and pilot examiners, it should be possible to come up with a concise job and task analysis for each pilot level that would be supported by a broad range of pilots (a bit of a "community consensus model"), backed up by accident and incident reports or similar documentation to motivate the inclusion of a topic. It would immediately eliminate obsolete issues whilst introducing quite a few new things. It would also lead to a massive cost saving and enhance flight safety.

What severely handicaps such an effort at European level is the lack of uniformity in procedures and regulations at individual JAA member state level. The fact that all member states insist on their own niche implementation certainly doesn't help the exercise.

Anybody has a good idea on how to get such a project on the rails all the way to a successful implementation ?

[IO540]

I also fear that such studies can easily be poisoned by special interests. FTO's, Pilot Unions, Aviation Medicine Specialists, all want to overload the task list far beyond what is necessary to safely navigate the skies.That's always been the problem in the pre-Type Rating areas of flight training. Most of it is irrelevant in one way or another, or the TK is relevant and the flying isn't, or vice versa.

The reason is that none of the people involved in the service delivery have a business mandate to train pilots. Their business is selling TK or selling seat time.

If you are running say the RAF then you have to make sure the pilots can actually do the stuff.