Setting the MCCQE Part II standard

Introducing a new Blueprint

In late October 2018 the first administration of the Medical Council of Canada Qualifying Examination (MCCQE) Part II was held, based on the new examination Blueprint. The new Blueprint, intended to ensure that critical core competencies, knowledge, skills and behaviours are being appropriately assessed, was developed following extensive practice analysis, input from subject matter experts, and consultation with external stakeholders.

Following the exam, a full and detailed standard-setting exercise was conducted in December 2018. “It is best practice to revisit a standard every 3 to 5 years or when there is a change to an exam,” explains Dr. Liane Patsula, Associate Director of Psychometrics and Assessment Services at the MCC.

Given the introduction of the new Blueprint, we are now measuring something different. Hence the need to reestablish the standard for minimal competence for entering independent practice to ensure that fair pass and fail decisions are made based on the new expectations of the exam.”

Dr. Liane Patsula,
Associate Director of Psychometrics and Assessment Services, MCC

A meticulous exercise

The MCCQE Part II standard-setting exercise is a rigorous process involving a group of panelists as well as MCC psychometricians and staff. “Twenty physicians, a sample representative of practicing Canadian physicians in terms of demographic variables such as gender, geographic region and medical specialty, were selected to participate in the exercise,” says Dr. Andrea Gotzmann, Senior Research Psychometrician at the MCC. “The three-day activity involved half a day of training about the new Blueprint, the exam structure, the selected standard-setting methods, etc.” After the exercise was completed and all necessary data was gathered, the MCC was able present a recommended pass score to the Central Examination Committee (CEC).

The MCC’s CEC convened in January to review and approve the pass score for the new exam and to verify the results for the October 2018 cohort. The CEC was provided detailed information that outlined the processes and procedures for the standard-setting exercise, and the impacts to candidates based on the recommended pass score from the standard-setting panel. The pass score of 138 on a scale ranging from 50 to 250 was unanimously endorsed by the members of the CEC.

“Since there was a significant change to our Blueprint, we introduced a new scale to avoid confusion between our exam based on the old Blueprint, and the new exam based on the new Blueprint.

We do not encourage score comparisons between the old and the new exams because they measure different competencies,” adds Dr. Patsula. However, if comparisons are needed, we encourage program directors to use the group mean and the standard deviation for a given exam session (see How your MCCQE Part II score can be used to assess relative performance).

A new exam, a new direction

Following the standard-setting exercise, the MCC also interviewed panelists to get their opinion on the new Blueprint and what the new exam means for candidates, regulators and the public who rely on the MCC’s assessments. “I think the new Blueprint is fair for the candidates, it’s more reflective of the reality they will experience in their offices, as opposed to focusing on specialties,” says Dr. Céline Bouchard, Associate Clinical Professor in Obstetrics/Gynecology at the Université Laval.

Dr. Achla Virmani, Family Physician at McGill University’s Department of General Medicine, adds that, “it was time for a transition. Having sat through many of the MCC exams in the past, this Blueprint’s focus has been narrowed down and places more emphasis on assessing key requirements the graduates need to have.”

When asked about the rigor of the standard-setting process, panelists had a lot to say. “I find it very reassuring, as a practicing physician, knowing that the quality and the standard are very high in terms of establishing a valid examination process,” states Colonel J.G. Kile, Director of Medical Policy and Clinical Services for the Canadian Armed Forces.

As for Dr. Wendey Proctor, Family Physician at West Lincoln Memorial Hospital, she mentions that “the process is very rigorous and does a very good job at making sure that the integrity of the profession is maintained here in Canada. Canadians should feel very secure and safe with the healthcare they are being provided based on the standards that are set here.”

With all the changes that were implemented to the MCCQE Part II, and the rigorous standard-setting exercise that took place, the MCC is ensuring continued excellence in physician assessment and fulfilling the objectives of its Blueprint project.