Ensuring exam scores are valid and reliable

Psychometricians make sure every Medical Council of Canada (MCC) exam “is really measuring what we want it to measure”

Compiling all the data from an exam and poring over them is not a job that would appeal to everyone, but it’s a passion for Maxim Morin – one of the MCC’s Psychometrics and Assessment Services psychometricians. Analyzing this data allows us to evaluate the reliability and validity of exam scores.

The dictionary definition of a psychometrician is a person skilled in psychological tests. “But that definition is not fully accurate,” says Mr. Morin. “It’s about the measurement of psychological attributes, but there are two branches — in the educational field and in the psychological field.” With a recognized expertise in educational psychometrics, MCC employs six psychometricians.

Quality assurance, and reassurance for test takers

Mr. Morin was in the process of completing a degree to become a physics teacher when he accepted a summer job reviewing literature in educational assessment and he was hooked.

Mr. Morin went on to complete a Master’s degree and is currently working towards earning a doctorate in educational assessment. “I always liked math, but as is very common for psychometricians, particularly applied statistics,” he says. “I like something a little more concrete.” Computer programming is another interest that Mr. Morin brings to his work. He has worked for the MCC remotely from his home in Blainville, Quebec, for the past three years.

Quality assurance, and reassurance for test takers

Psychometrics helps the MCC conduct quality assurance that an exam “is really measuring what we want it to measure,” says Mr. Morin.

“When we think of an exam, like the Medical Council of Canada Qualifying Examination Part I, we make a claim that it is meant to assess skills, knowledge, and attitudes of candidates who will begin a residency. Psychometrics provide data to bolster that claim and assures candidates that exams are fair and scores are reliable and valid.”

M. Maxim Morin,
Psychométricien, Psychométrie et services docimologiques, CMC

The organization is also concerned about fairness. “Let’s say there was a fire alarm in the test centre,” says Mr. Morin. “There will be a break in the timing of the exam. We need to check that the break did not have an effect on the test score of the candidate.”

The MCC also looks at the overall quality of the exam. As there are many test forms with different questions, psychometricians compare the difficulty of the questions within each exam to take steps so candidates see forms of comparable difficulty.

The future

Mr. Morin’s interest in computer programming has come to the fore in efforts to automate many of the labour-intensive processes for exams. Assembling the questions for 20 test forms is one thing he’d like to automate “based on the same principles and the same tools used by aviation companies to fly the best route for the planes.” Mr. Morin is also involved in the project to automate much of the exam marking using machine learning and natural-language processing (see Exam marking goes high-tech).

The aim is to continuously improve the exams and quickly deliver the results to candidates. “Psychometrics and test development go hand-in-hand,” states Mr. Morin. “I’ve been here for almost three years, and I haven’t seen any problems, but a lot of improvements.”