What makes you deduce that
they are trying to get away from publishing a workable QB - either by not publishing a QB, or by creating a QB which is too huge to be of any use for revision.
?
Here are my comments about this "study" :
1. EASA is wasting our money handing it out to some Swiss consultants to collect some undergraduate level research and to repeat a few experiments of which the results would be obvious from the start.
2. Several important points are missed or wrongly interpreted by their "experts" :
- the use of a question database is an important guidance to let the applicant know what knowledge (depth) is required to comply with an "ability to demonstrate theoretical knowledge". The fact that the applicant familiarizes herself with this level of depth through studying of a question database should be welcomed.
- the classical example of the FAA "calculation questions" being answered in just a few seconds needs to be seen in the context of the learning process. While reviewing the following test problem on the FAA Commercial Exam :
(Refer to Figure 12 in FAA-CT-8080-1C)
GIVEN:
Pressure altitude 18,000 ft
Temperature -1°C
Power 2,200 RPM - 20'' MP
Best fuel economy usable fuel 344 lb
What is the approximate flight time available under the given conditions? (Allow for VFR day fuel reserve.)
A) 4 hours 50 minutes
B) 5 hours 20 minutes
C) 5 hours 59 minutes
an unobservant student would look at the power settings table and read for 2200 RPM @ 20" Manifold Pressure in the column for 20° above standard temperature : 59 PPH. Dividing 344 lbs by 59, substracting 0,5 hours of VFR reserve, she would obtain 5 hours and 20 minutes, the incorrect answer. The correct answer is C (because a note says that for best fuel economy at 70% power or less - which is the case here - one should operate at 6 PPH leaner than shown in this chart).
After having reviewed this question beforehand, the student would probably answer the question in the test in less than 30 seconds, because she had already reviewed it. However, thanks to her preparation, the learning objectives (computing endurance whilst considering all the fine print of the performance tables) is clearly achieved. No sensible student in aviation would learn the answer 5 hours 59 minutes by heart without at least attempting to understand where the answer is coming from.
The fact that memory helps a student sitting exams does not imply that the student does "not rise above the rote level". Even if one were to fear this, one could possibly extend the number of questions and label them clearly as "equivalent", so that the student could work through the type of question by just solving 3-5 typical problems (out of a total maybe 20 or so in the database).
- The researchers rehash some well known results on timely exponential decline in theoretical knowledge, which they apply to rote memory of multiple choice questions but which are just as well applicable to retention of any theory memorization process.
We all recall the horror of "Human performance and limitations" questions, but because of the irrelevance in practice, we loose that knowledge exponentially fast. The authors do remark that "They found that there is a trend for pilots to remember information that is relevant to their day-to-day operation of aircraft, and that irrelevant knowledge is preferentially forgotten".
But they then go on to argue that because rote memorization is a bad way of storing knowledge, everything should be put to work to avoid students using this methodology. The latter reasoning is unfounded. After all, there is no such thing as an "in depth study" of Air Law for instance. Studying a part-FCL text can be just a numbing as studying multiple choice questions about that same part FCL.
- an inordinate amount of pages is wasted on the statistical analysis of their new experiments with university undergraduates. Some of the stats is downright laughable, such as coming up with an empirical formula for test accuracy as a function of the question database size.
- after this and without much scientific evidence in support, the researchers the argue in favour of increasing the number of questions in the test database. Although they purport to be "aviation consultants", no analysis is made of the learning processes associated with pilot training.
3. Probably preaching in the desert again, but some recommendations could be :
- publish (and make freely available) all question databases
- avoid trick questions, unless they convey a specific and clear learning objective.
- avoid all subjects that are not directly relevant to flight safety
- publish (and make freely available) reference manuals that can serve as definitive reference for study
- make sure that the context of each question is clearly understood / outlined, even if this comes at the cost of making the question longer.
- formulate each question so that eliminating the incorrect answers and giving the correct achieves by itself a learning objective.
- do not use the theoretical knowledge exam as a eliminating factor. Rather, encourage students to use it to make the flight training experience more rewarding. Instead, use the flight examiner to verify up to speed practical knowledge.