Setting the standard

How does the MCC establish the pass score for an examination?

When an examination determines a professional’s future, a lot depends on the score needed to pass. It sets the standard for physicians in Canada. How is that pass score determined? Who makes that determination?

“There’s no right or wrong standard—it’s an informed judgment,” according to Dr. André De Champlain, Director of Psychometrics and Assessment Services at the Medical Council of Canada (MCC). The MCC ensures that the standard is arrived at by a rigorous process that uses best practices and methods accepted worldwide. “One of the things we do routinely is to assess what level determines competency and provide regulatory authorities with a standard that we can stand by.”

Panels set the standard

The process starts by recruiting panels of physicians from across Canada to take part in a standard-setting exercise. “The MCC does not set the standard; the standard is recommended by panels,” emphasized Dr. De Champlain. Such exercises are conducted every three to five years, to “reflect the rapid pace at which medicine and medical education evolve,” he said. Major changes to examinations can also trigger standard-setting.

To ensure panels are as representative as possible, they “are balanced by region, years of practice, specialties, gender, language and whether physicians were trained in Canada or abroad,” said Dr. Andrea Gotzmann, Senior Research Psychometrician at the MCC.

For all types of examinations, panellists are divided into two panels, who conduct the exercise at the same time. Their recommended pass scores are usually very close but rarely exactly the same. “This tells us how much the pass score would generalize to other panels if we replicated the exercise with another group of physicians,” said Dr. Gotzmann.

Borderline and bookmark methods

All panellists begin with a half day of training. But the method of determining the pass score differs depending on the examination. “The difference in methodology is due to the different modalities of the exams,” Dr. Gotzmann explained. For performance-based examinations (the National Assessment Collaboration Objective Structured Clinical Examination [OSCE] and the Medical Council of Canada Qualifying Examination [MCCQE] Part II), the MCC uses the “borderline group” method. For written examinations (the MCCQE Part I), it uses the “bookmark” method.

In the borderline method, panels review candidate behaviours and attitudes that characterize good and poor performance through videos of OSCEs during training and introducing each station. The emphasis is on the candidates who are “just passing” (“just qualified”) and the behaviour expected from such candidates. Panellists independently rate 50 score sheets across all stations from actual candidates, the MCC staff calculate a pass score and discuss their results with the rest of the panel. They then repeat the exercise, adjusting their ratings based on the feedback and discussion.

For the bookmark method, the emphasis is on deciding which items a “minimally proficient physician” would be able to answer, said Dr. Cecilia Alves, Research Psychometrician at the MCC. After trying a practice exam, participants individually review an examination with questions ordered by difficulty. They place a “bookmark” between the questions a minimally proficient candidate would probably answer correctly (with a two-thirds chance of answering correctly) and those they would probably answer incorrectly. Similar to the borderline method, panels discuss their results and then do a second round.

Endorsement

For both methods, the two panels recommend pass scores. The average of the pass scores between independent panels are reviewed and approved by the exam’s governance Committee. “The Committee is provided with everything the panels did,” said Dr. De Champlain, as well as data on how many candidates would pass or fail if the pass score were endorsed. The Committee discusses the recommendation—a task it takes seriously, since setting the standard is the Committee’s responsibility. “It always endorses a standard that is arrived at by a rigorous process. It’s extremely rare that the Committee will decide not to adopt the recommended standard.”

He said standard-setting can give both patients and candidates confidence in the process. “While the intent of the MCC examinations is to protect the safety of the public, candidates can take comfort in the fact that we use an extremely high-quality process to arrive at this score, and we involve their professors, peers and colleagues.”

__________

For detailed information on standard-setting methods, read technical reports on the exercises for MCCEE MCCQE Part I, MCCQE Part II, and NAC Examination.

For more like this:

A chair for all seasons