Noel, God bless him had to work it through for them on a whiteboard
I presume you refer to Noel L, Gaunty ? Back in the olden days, Noel and I spent a couple of sessions in with CASA's alter ego of the day (probably still was DCA, then) and we (and several other Industry folks) were tasked with going through the exam bank to comment. Nothing wrong with questions at that stage, I suggest.
The main problem with exams has always been twofold .. on the one hand, the examiner needs to test a reasonable fraction of the syllabus and that is a hard ask with a time limited single examination. In addition, there needs to be a reasonable looksee at the student's ability to do sensible quasi-real world calculations.
My view has always been that a combination of multi-guess to look at the former combined with longhand calcs for the latter is a reasonable scenario. With the multi-guess one can put in distractors and close alternatives to sort the wheat from the chaff and provide a level of grading. The longhand exercises, though, need to concentrate on technique and logic rather than the actual answer .. providing that the student recognises if he/she has gone off on an inappropriate tangent and so identifies in the manuscript.
For exam results suddenly to skew dramatically raises concerns in several areas -
(a) it is a reasonable presumption that student ability and performance, on average, will be predictable.
(b) a significant exam result skewing suggests one of
(i) (a) didn't operate on the occasion for some reason
(ii) the particular exam(s) involved a significant step change in technical difficulty, whether up or down
(iii) the particular exam(s) involved a significant step change in the level of dissimulation, whether up or down
Do the exams need to be "practical". I suggest not, although they do need to be sensible. Requiring discrimination of answers to quite inappropriate accuracies probably is not quite necessary. As in all areas of mensuration, accuracy ought to be pitched at the reasonable requirements of what is being assessed.
Gaunty also makes the point that a commercial piloting career ought not to be the result of good attendance.
Some folks ..
(a) are particularly suited and sail through exams, endorsements and training programs without any undue difficult .. and then fly with a level of skill which makes the rest of us green with envy
(b) are reasonably suited and talented and, with a bit of work and effort, make the grade
(c) are a little behind the eight-ball to start with and, by virtue of extreme application and diligence, make the grade. I can recall (with great fondness) a couple of such folk whom I trained through their theory work and they persevered and made a good grade. One ended up a well regarded C&T chap for a respected carrier
(d) some, unfortunately, really need to look elsewhere for work
The exams, endorsement programs, line training, upgrade training programs and so forth are all part of the process which facilitates putting folks into one of the paddocks.
Does the system always get it right ? Of course not but, then, no system gets to be perfect.
The student needs to know the work and, also, have a familiarity with the style of questions and answers which the examiner is looking for.
Well do I recall my first looksee at ATPL/SCPL Flight Planning. I read the syllabus .. pretty straightforward ... swotted Worthington (a recommended standard text for the subject) ... pretty interesting and straightforward .. and then sat the exam .. pretty shattering.
Next time around I had had a look at some typical exam questions and solutions and the result was a tad different ...
A bit of a thick skin helps one to get through this aviation minefield, I suggest.