Last week we added a new exam to Examenzaal: Wet- en regelgeving in het werkveld , one of the five exam parts for the Dutch security guard diploma (Beveiliger 2). The exam body publishes the format (40 questions, 40 minutes, pass at 27 points) and a list of requirements. It offers no practice exams (only a few model items that show the exam software), and we don't use any of its questions.

So the questions are written by one model and checked by two others. This post is about the numbers from that pipeline, because they show something I didn't expect: the two checkers almost never disagreed about whether an answer was correct . They disagreed about whether the answer could be guessed from its wording .

The pipeline, in one paragraph

Per slot (40 slots over 42 requirements; which requirement each slot tests is our own split), a drafting model writes questions from the saved sources only (mostly legal texts: Wpbr, Rpbr, Arbowet, AVG, Sv, Sr and so on).…