This would be a vast improvement IMO in the way the exams are run, it would go a long way to ensuring candidates have actually understood the material. The current system is a joke in that as long as you play the game and bash the question bank you're practically guaranteed high marks, regardless of how well you actually understand the material.
I worked hard when I did my ATPL exams (no complaints there, I found the course stimulating and interesting and knew the more effort I put in the more prepared I would be for a career as a pilot) and achieved a final average I was pleased with. But there were others I knew that easily got higher marks simply by cramming in as much practise as possible on question banks and memorising questions/answers.
On one occasion at an exam centre, I overheard a group of candidates discussing how they were able to recognise questions in their exams from their QB practise and able to automatically pick the correct answer without really knowing where it came from (and these guys were cadets on a major UK airline sponsored/part-sponsored course).
The only issue with a free/open written question system would be the point Alex touched on. Judging by the standard of some of the questions I sat through in my exams, I'd have misgivings over the ability of some of those writing the questions in the first place, let alone the ability of CAAs' to accurately assess candidates' responses.