Computer Based Exams for JAA?

[Capt. Manuvar]

Why aren't CAA/JAA exams computer based (like the FAA ones) ?
Are there any plans to move to a computer based system?
Since questions are multiple choice, they can be marked instantly.
The exams can be done at anytime at an approved centre. and even though it might take some financial investment to set up, it will save the CAA/JAA ( and probably the candidates) a lot of money in the long run.
Students will be able to go through their training much quickly as they don't have to wait for specific dates to do their exams. I also believe marking is instant so you dont have to wait for months for your results.
Just my thoughts, what do others think?

[troposurfer]

cyberexams!......................the CAA!....................21st century!.......
.............

The CAA can't even manage to sort out a testing centre between Bedford and Glasgow, probably citing lack of suitable facilities.

Seems strange though that they could allow one to be set up in another country ie USA.

There are countless universities and colleges throughout the UK that would be suitable as venues for the ATPL exams. This would be even more feasible if the exams were cyber exams via the universities ADSL/broadband links back to CAA Towers.

I used this system in Australia at an approved testing centre. It worked fine, I got used to the format very quickly and got my results back within 1/2 and hour, printed and verified by CASA with a KDR (knowledge deficiency report). I passed!!
The KDR was then used as part of the oral pre-flight test brief and I had to brush up on those subjects before the flightest. The oral lasted over 1 hour and included subjects I had shown to have deficiencies in.

It seems like the former colonies beat us at sport time and (except maybe RWC! Go England!) again and have a totally different attitude to progress. The UK seems stuck in the past with a 'make do and mend' mentality eg public transport/health service/education. Though it seems a paradox that we have a system driven by the CAA, first implemented by the CAA and the 'envy' of the rest of the aviation world
....................................THE JAA.

Even the Aussies want to join it! (2005 I was told be some aussie mates!)

What a political and logistical triumph, 16 nations, dozens of languages, a huge question bank and proper theory training/courses. Come on CAA prove me wrong on the Exam venues and future technology!!!!


Tropo

[minus273]

Thought that the Aussies were moving to the FAA system? They have started to go that way with their airspace anyway.

As for the CAA and computer based exams that would mean that:

1. The exams theoretically would be cheaper.
2. People could do them when they wanted.
3. There would not be a centralisation of power at CAA HQ.

Answer:

1. Cheaper means that the CAA gets less of your money.
2. People doing them when they wanted means that the CAA
would lose control. (Which personally is what I think it is all about)
3. God forbid that this would happen.

-273

oh yes by the way that will be 52 pounds for that test.

[troposurfer]

The aussies seem to have adopted a similar system in terms of the naming of sections of airlaw on a par with the FAA system ie part 141 for FTOs etc etc but have now seemingly rejected the adoption of either the FAA or JAA system as their domestic legal framework and legislation differs significantly from both.

Their licence was in the past an ICAO licence but the 'differences' notified to ICAO meant that the Aussie licence was not being recognised in other territories. (As I found to my cost) The Aussies have made proposals to goverment to change their system and have already started by changing the structure of their professional exams and qualifications.

This could mean 'interesting' times for anyone hoping to convert their Aussie PPL/CPL after taking advantage of the lower training costs down under. The Aussies now want a fully ICAO equivalent licence adhering more closely to ICAO SARPs and therfore finding greater acceptance such as FAA/JAA.

And there were my Aussies mates ready to hop on over to Europe with their new shiny JAA licences and 2000hrs! Phew!

If anyone needs to check before they go.......

www.casa.gov.au

Foreign PPLs/CPLs are gonna get better recognition in Aus too.


Ps The CAA made £2.26 on each question in VFR/IFR comms, combined they took me less than 25mins, £104 for 25 mins 'work' not bad if you can get it!!!!! Beats being a BA Captain!!

Too right on the less power too wield back at CAA Towers.

Over and out.

[Keith.Williams.]

I have no wish to defend the CAA in this matter, but there is no point in exaggerating the situation. It does not take months to get the results under the current system. It takes between 7 and 10 working days from the end of the exam week. It just seems like months if you are one of the students waiting for the results.

Computerised exams are unlikely to provide any significant reduction in overall training time. Most students approaching their exams wish that they had a little more time to get ready. It is however true that those who fail exams usually want to retake then as soon as possible. It is also true that this urgency is often a mistake and students would do better to ensure that they are fully prepared before any resits.

The CAA have actually considered the use of computerised exams, but are a bit worried about the quality of their central question bank. Some students and instructors share this concern.

[troposurfer]

dear keith,


maybe it's just me, but I didn't see any of the other posts complaining about the 7-10 day wait for results.

The main gist is that we are using an antiquated system. Hardly an exaggeration. My major gripe is that I have to spend 4 hours travelling down the M1, 2 nights at £50-80 a night on hotels and another 4 hours back up the M1 just to sit a bl@@dy exam! This is on top of the extortionate £52 per paper and distance learning fees. Again hardly an exaggeration either!

This is the point being made!

It would be far cheaper and a more flexible system if cyberexams took place at venues other than the South East of England and could be arranged easily on a demand basis. One could then chose to visit exam centres on a time table that best suited the student at minimal travelling distance and expense, afterall most students are distance learning and are already paying large amounts for ATPL training.
Any system that makes it easier/cheaper for said student/ training organisation can ONLY BE A GOOD THING!

I can't see how anyone could complain about that!

Your comment........
____________________________________________________
The CAA have actually considered the use of computerised exams, but are a bit worried about the quality of their central question bank. Some students and instructors share this concern.
____________________________________________________

oh great so I am paying 52 quid for exams that even the CAA think are cr@p! Marvellous!

[Send Clowns]

As I understand it the planning for exactly this change is in process at the moment. The CAA is at least considering these options, though I do not know how far any plans have advanced.

[troposurfer]

Thanks Send Clowns,

It might seem like I keep banging on about this issue but it is very important to me. If a student takes the option of Distance Learning for the ATPL they are looking at approx £2000 for their course. If they don't live near the Groundschool they are also looking at hotel/B+B accomodation for 4-5 weeks as an additional cost on top of that.

Add on top of that say 2 trips to a testing centre for main exams and say 1 unfortunate trip back for a resit and associated hotels and travelling expense, and the 14 exams themselves.

What appeared at first to be a course for £2000 plus accom now turns into................

Distance Learning Course + accom/food+beer. say £2500-£3000

then Add..........

£728 for 14 atpls

£52 per re-sit(let's say only 1 needed)=£780

£50-80 per night for hotel at Gatwick,Glasgow,Oxford or Bedford approx another £400 inc food(3 trips)

And travel expenses to Gatwick/Glasgow/Oxford/Bedford for 3 trips £150

Days off work for travel/exams........?

The exams/additional exam expenses total around....£1350

as a percentage of your course this is 50%!!!!!

I don't know about you but £1350 is quite a lot of money!

It's good to know they are considering a change to the exam system and centres, could you keep us posted Send Clowns? Hopefully the CAA will make this a bit more public. It seems that only those 'close' to CAA personnel are aware of this kind of information..

Tropo.

[Mobieus1]

Make the exams cheaper?? Yeah right more like put them up. CAA excuse - pay for all the fancy software and computers of course :(

[Keith.Williams.]

Dear Troposurfer,

You are of course correct......it is just you.

If I may quote from the very first post in this string....

"I also believe marking is instant so you dont have to wait for months for your results".

As I stated above the typical timescale for marking is much shorter than this and there is nothing to be gained by exaggerating the problem.

You also appear to be assuming that computer testing will reduce costs and enable exams to be carried out in a wider range of venues. On what do you base this assumption?

If a student asks for a remark the CAA charge £20 just to push a sheet of paper through a computer data reader. Why should we imagine that they will charge less for a whole exam.

Shortly after my school started operating at Bournemouth I asked the CAA to establish an exam centre there. My argument was that the three schools in the area (EPTA, GTS and BCFT) could generate a reasonable number of students. Within three days the CAA rejected the idea on the basis that "It would not be cost effective". The small number of exam centres currently in use is due simply to the fact that the CAA do not want to create any more.

Even if they eventually introduce computer testing they are unlikely to allow the tests to be supervised by the schools. They might (just possibly) permit them to be carried out in the offices used by their flight examiners, but even this is far from certain.
In all probability the flight examiners would claim that they are too busy to carry the extra work load.

You have also grossly exaggerated my comments about the CAA concern over the quality of their question bank. My saying that they are concerned is not the same as saying "the CAA think their exams are cr@p". Their concern about using the current bank for computer exams is that this will generate an endless stream of appeals. They have no wish (and probably no resources) to deal with such a stream.

The more enlightened members of the CAA staff (yes there are one or two) recognise the fact that they have problems, but other members do not. Unfortunately the "we have no problems" brigade currently hold the reigns.

Like you I want to see a much more efficient and less costly system of exams. But there is little to be gained by exaggerating the problem and raising unrealistic expectations. If you have already started your course you can be assured that you will have taken all of your exams long before any computer testing system is put in place.

[minus273]

Hey all,

Well seems like a real can of worms has been opened on this one. I think that some people are taking it all a little more seriously then the gripe that was intended.

Personally I have done the US & Canadian system as well as the JAA.

In the US you sit in a booth with a computer and do your exams in a reasonable time period for a reasonable cost, the results are obtained in a matter of seconds not days.

(And on that point I think that the common saying of "it takes months" is understood and should not be blown out of proportion by someone trying to score points)

In Canada you trundle on down to your local TC office, where after giving them a call a couple of days ahead of time they ask whether you would like morning or afternoon and which exam you would like to take, it is still done on the card reader paper like the JAA exams not on computer, but when you are done you tell them, they take your paper and run it through the machine in front of you and give you the result.

Two other things that I will say about the FAA and Canada tests are that you get feedback on what you got wrong in the exam. You also get a chair and desk that you can actuallu use. Trying to flight plan on one of those JAA exam desks is disgraceful.

Now as far as I rememeber from instructing the only way one can learn is to know what one got wrong. And this does not happen with the JAA exams.

The JAA exams are also extremely costly 52quid, or another way when I sat the VFR comms paper it cost me 10quid a minute.

Well I hope that this thread can continue in a polite non-slagging match as I think that there are true greivances to be aired, also the fact that I would like the people coming through the system behind me to benefit from a change of process.

-273

[Send Clowns]

They are apparently looking at various options being studied, all of which would increase the number of venues and the number of sittings available while reducing the waiting times (possibly to minutes).

The costs were not mentioned, but may increase due to the expense involved as I am sure students will bear these, but as tropo pointed out this may be set against cost savings of travel and accomodation. It is good to see these issues being raised, and I hope hte CAA will get word. I certainly will mention it next time I see one of the fine people from the Belgrano!

I agree (of course!) that there should be a centre down here, Keith. In fact there was one, at least when I sat them in 2000, but even then it had not been available in living memory.

[FlyingForFun]

Kieth,

Everything you're saying sounds very plausible, although not encouraging. Except for your comment that: "Their concern about using the current bank for computer exams is that this will generate an endless stream of appeals." Why should candidates be any more likely to appeal after a computer-based exam as opposed to a paper-based one if the questions are the same? Am I missing something?

As for other comments, I agree that not having feedback is appalling. The exams are, presumably(!), examining things which I am expected to know - so if I don't know them (and let's face it, no one scores 100% on every exam so we all have holes in our knowledge) then for the CAA to deny me the opportunity to find out where those holes in my knowledge are is bordering on negligent.

FFF
-------------

[Send Clowns]

FFF

What Keith is alluding to is the current system not relying entirely on the reliability of the question bank. Each paper is looked at and any suspect questions that are spotted are removed. The current appeals are therefore only the ones that "get away" and are not noticed by the CAA themselves. This would not be possible if the computer was immediately presenting the exam to the student.

[Tinstaafl]

Don't agree with that, SC.

In any Authority based exam system I've seen the questions are NOT held on the client computer - they're accessed via phone line or internet to an Authority controlled server just prior to sitting the exam. This means that al questions can only those approved by the relevant Authority. If a question is innappropriate then that is purely the Authority's fault for releasing it to the data bank.

Whether a particular question is presented on paper OR on a monitor makes no difference to the vetting it needs & should receive prior to release to use.

Establishing computer based exam centres is easy to do AND relatively cheap. It makes it very viable for exams on demand at reduced cost.

A caveat though: If the examination system is contracted to a commercial organisation to administer then there must be several organisations approved, not just one. A serial monopoly doesn't improve the cost or service.

[Send Clowns]

Yes, Tinstaafl, the system would likely be as you describe. However, although bad questions should not be given out by either system and should have been removed, they are in the quesion bank. Currently questions are still being removed after production of the paper, which cannot be done with a computer-based exam. I agree that this current system is absurd, but remember the CAA are stuck with whatever over-arching system JAA has forced upon them, so it may not be their fault.

[Keith.Williams.]

-273

I'm note sure who you think is trying to score points, but I am certainly not. Troposurfer used a rather curious form of address which I felt warranted a similar response.

Your comment that "...the common saying of it takes months, is understood and should not be blown out of proportion..."ignores the unhelpful effect that such exaggerations will have on the CAA.

Many of those posting comments in this forum do so in the hope that the CAA will see them and take action. It is certainly true that some CAA staff read pprune. But if they see complaints about "months awaiting exam results" they will simply dismiss them out of hand on the basis that it is "just students being unreasonable again". The comment "the CAA think their own exams are crap" is also likely to have a less than constructive effect. The message here is that if you want your comments to be taken seriously then you must make serious comments.


FFF
The concern about getting an increased number of appeals with a computer based system originated from a CAA examiner, and he was probably correct.

At the moment the xams occur monthy with something in the region of 150 students participating. If we take for example the POF exam it currently holds about 45 questions. So the CAA produce a single paper of 45 questions per month. They examine this paper to try to detect and eliminate problem questions. The number of appeals raised by students and schools varies from month to month, but for a single subject it is pobably about 5 to 10 per month.

Before marking the papers the examiners consider each of these appeals and decide which to accept. Where an appeal is accepted they give all of the students the marks for that question and remove the question from the question bank. In slow time they then go back to the JAA Subject Expert Team and decide what to do about the problem question. In most months not more than 1 or two questions in any given subject are accepted as being defective.

From the CAA point of view this process represents a big work load, but at least it all occurs within a single week. They then have the rest of the month to get on with other tasks such as carrying out annual inspections of the schools.

Now consider the situation with computer based exams on demand. In order to prevent cheating it will be necessary to ensure that no two (or at least very few) exams are identical. This will require either the creation of a large number of stock exam papers or randomly generated papers for each student. The random generation of papers is obviously no problem for the computers, but will inevitably involve the use of a much bigger number of questions in each month.

This bigger number of questions will inevitably attract more appeals. And these appeals will arise on a daily basis instead of being concentrated in a single week. Worse still, if the results are to be published immediately, the appeals must be considered immediately. OK they could issue conditional results, but there would still be a need to process appeals in a very short time. From the CAA examiners' point of view this scenario is nightmare city.

All of these problems could of course be avoided if only the CAA could sort out their question banks. Sadly the fact is that many of the recently created questions are no better than earlier ones. So more problems are being stored up every time the question bank is expanded. For an example of this problem just look at the "variations of Vx and Vy with altitude changes" questions in the PERF exam. These questions were withdrawn early in 2001 but they still keep resufacing without any sign of improvement.

I would love to see the introduction of computer exams on demand, but there is no point in raising unrealistic expectations. They will certainly be brought in within the next century. They will probably be brought in with the next decade. But within the next year?........I very much doubt it.

[minus273]

Dear Mr Williams,

I think that even the CAA would understand the frustration of students in wanting to obtain their results as fast as possible.

You seem to also be trying to say that I have not contributed anything to this discussion. In that I have not posted anything serious to the discussion, apart from my first post which was a little in jest then everything else has been serious.

The fact that I have done two other systems of exams I believe allows me to offer an excellent view point as to how the CAA can improve theirs.

If you are going to use my quotes against me that is fine but make sure you use the correct ones. The quote against the time period of "Months" that is mine and was in backing up Troposurfer. I am also not sure where you get the idea that this expression is rather vague.

The other quote with regard to the CAAs exams being crap belonged to one of the other posters. And I think that if you check you will find that I never said that the CAA think that their own exams are crap. So please use the correct quotes for the correct person.

The last time I checked sarcasm or exaggeration was not a crime and this notice board is for people to publish their grievances as well as to try to help people understand what is going on in flying as not everyone is of the same experience level.

Anyways back to what the topic is about:

From my understanding of the US system it comprises of data banks for each level of exam and on top of that I believe that about 10% of the exam questions are non-published questions. So even though the data bank is publised to all 10% of questions are unkown so teh student to get a high score must have learnt the material rather than memorised it.

The procedure for the exam is: exam centre calls up the main centre and they in turn send the exam which is randomly generated from the data bank. This is held on the the exam centre computer and the student takes the exam which is then marked the second after they have completed the exam and wish it to be marked.

As for the ability to generate what is termed a valid exam in the USA, "One which tests the true ability of student and the material being examined" I believe that the CAA exams fall far short of this.

I am sorry that you believe that the CAA will not be getting a computer exam system.

Quote: "I would love to see the introduction of computer exams on demand, but there is no point in raising unrealistic expectations. They will certainly be brought in within the next century. They will probably be brought in with the next decade. But within the next year?........I very much doubt it."

But in your own words Quote:"The message here is that if you want your comments to be taken seriously then you must make serious comments."

So I am glad that you are allowed to use an expansion of time to make a point, but others are not.

-273

[troposurfer]

it really does pay to check your facts first!


To quote from one of your posts Keith.......................( If I may quote from your very first posts in this string........." I also believe marking is instant so you don't have to wait months for your results".)...............................................

please note! In capital letters, I DID NOT MAKE THAT COMMENT!

The fact is Keith, those particular words of wisdom were put forth by Captain Manuvar and NOT myself.

You misquote me!

You also claim that I "grossly exaggerated" your comments regarding the CAA question bank. Again, you are incorrect what I did do was to quote your post, word for word, ACCURATELY and then PASS MY OWN OPINIONS ON THE FORUM, as a payoff note at the end of my post. My words do not infer directly or otherwise that this is what you said or intended to say.

This forum is about opinions, they are like ar$eholes!

Everyone's got one.


It seems to me that we are more in agreement than disagreement as a whole.Nearly everyone on this particular subject has had something worthwhile to add. Keith, you yourself have enlightened me to a few things I hadn't considered. I have taken your comments on board.

You did however make a few points that I would like to reply to, but I will attempt to do so in a different tone.................


You make a valid point challenging the posters who support Cybertesting to show how it will reduce costs. Well I spent some time detailing this in an earlier post. The costs (to me personally) arise from travel and accomodation and the fact that each paper is £52 regardless of the duration of the exam or the number of questions. I could accept that to charge different amounts for each exams may be difficult to administer or justify by the CAA.

My remarks regarding Cyberexams and those of other Pprune posters have been based on EXPERIENCE and the overwhelming fact that for those of us that have used them is that they are..

1) Easy to use

2) instant marking

3) cheaper per exam

4) you don't have to travel far and wide to sit an exam(internet connections can be set up to a secure server from anywhere. This is not science fiction.

5) And the most important...You get Positive Feedback detailing your areas of weakness in a given subject. This MUST surely be better for safety and knowledge.

I agree with one of the earlier posters that it is almost negligent to allow students to pass exams without pointing out their deficiencies and advising if not forcing them to make up this deficit in knowledge. The whole purpose of the approved distance learning system is to stop those people who self teach feedback questions and don't know sqat about Pof,Perf etc etc. They just learn questions word for word. Those kind of people we do not need, taking 300 souls up to 40,000ft and not even knowing why an aircraft flies!

Keith you also made the point that the CAA rejected your request for a testing centre in Bournemouth "within 3 days" on the grounds that it was "not cost effective". I have worked in industry for the last 12 years in various roles including senior positions. I have never heard of any feasability study being undertaken, especially by a Govt department, inside 3 days. It sounds more like, just my opinion, your enquiry was dismissed outright. Without consideration. Although I do remember someone in Bournemouth telling me that they had a testing centre in Bournemouth a few years back but withdrew it. They may have based their decision on past experience. Who knows.

I make this point again though, that it seems strange that they have a testing centre in the US of A which I think IS supervised and run by 3 FTOs combined at a site in Florida.

I hope that bygones can be bygones and we can get back to the issue of exams.

By the way, I know I will be long gone before before one can sit at a nice shiny new computer and take our ATPLs at a testing centre in the somewhere in the North or nearby, but that doesn't make a blind bit of difference to me.
If in the future another WANABEE like me can save a bit of cash and do just a few more hours actually up in the air rather than filling the coffers at CAA Towers that would make my day!!!

Yours,


Tropo

[Keith.Williams.]

-273

There really is no point in getting over excited. It adds little to the debate and is preventing you from understanding my comments.

If I may take the liberty of quoting from your most recent post:

QUOTE 1
"I think that even the CAA would understand the frustration of students in wanting to obtain their results as fast as possible."

MY COMMENT
You are of course correct, but they also get very frustrated with what they see as the inability or unwillingness of students to understand their problems.

QUOTE 2
"You seem to also be trying to say that I have not contributed anything to this discussion. In that I have not posted anything serious to the discussion, apart from my first post which was a little in jest then everything else has been serious."

MY COMMENT
Nothing in any of my previous posts states that you have contributed nothing serious. I was merely making the point that if anyone wants to be taken seriously by the CAA they must make serious comments and avoid hyperbole.

QUOTE 3
"If you are going to use my quotes against me that is fine but make sure you use the correct ones. The quote against the time period of "Months" that is mine and was in backing up Troposurfer."

MY COMMENT
I did not state that these were your comments. My use of them was simply to illustrate the type of comments that will do nothing to impress the CAA.

QUOTE 4
" I am also not sure where you get the idea that this expression is rather vague."

MY COMMENT
Nothing in any of my posts suggested that the term was rather vague. I was simply pointing out the fact that such exaggeration is likely to be unhelpful if the objective is to influence the CAA.

QUOTE 5
"The other quote with regard to the CAAs exams being crap belonged to one of the other posters."

MY COMMENT
I agree, but as I have have never attributed this statement to you I fail to see your problem.

Now if you reread my previous contributions to this string you will see that they consistently support the introduction of computer testing, but describe some of the problems that the CAA believe they will face in introducing such a system.

TROPOSURFER
You appear to having the same problem as -273

If I may quote from your most recent post:

QUOTE 1
"it really does pay to check your facts first!

To quote from one of your posts Keith.......................( If I may quote from your very first posts in this string........." I also believe marking is instant so you don't have to wait months for your results".)...............................................

please note! In capital letters, I DID NOT MAKE THAT COMMENT!

The fact is Keith, those particular words of wisdom were put forth by Captain Manuvar and NOT myself.

You misquote me!"

MY COMMENT
I did not say that you made the statement about waiting months. In your previous post to me you had stated that no one had complained about waiting for results. I simply pointed out that the originator of this string had done so.

QUOTE 2
"You also claim that I "grossly exaggerated" your comments regarding the CAA question bank. Again, you are incorrect what I did do was to quote your post, word for word, ACCURATELY and then PASS MY OWN OPINIONS ON THE FORUM, as a payoff note at the end of my post. My words do not infer directly or otherwise that this is what you said or intended to say."

MY COMMENT
You quoted my statement that the examiners were concerned about the quality of the questions, then followed this with the comment about the CAA thinking their exams were crap. It seems to me that this is a clear exaggeration of the meaning my comments. It is of course possible that I have misinterpreted your intentions, in which case I readily appologise for doing so.

Now, what was this string about??????