Collaboration through research

In October 2018, two research psychometricians from the Medical Council of Canada (MCC) travelled to Dubai to participate in the 13th International Conference on Medical Regulation. This conference is held biennially by the International Association of Medical Regulatory Authorities (IAMRA) to offer its members the opportunity to exchange and discuss ideas surrounding many topics, including health regulation and medical education. Dr. André De Champlain and Dr. Fang Tian presented the preliminary results of their research conducted in collaboration with the College of Physicians and Surgeons of Ontario (CPSO) and the College of Physicians & Surgeons of Alberta (CPSA).

Examination performance:
possible predictor of practice assessment outcome

Dr. Fang Tian, Senior Research Psychometrician at the MCC, presented a poster outlining the preliminary findings of the study she is currently undertaking in collaboration with other MCC psychometricians and CPSO researchers.

The purpose of this research is to examine the relationship between physician performance on the Medical Council of Canada Qualifying Examinations (MCCQEs) and their performance in practice as assessed through CPSO’s peer and practice assessment program. “This outcome-based research would help the MCC gather evidence to support the validity of its exams for licensure while helping CPSO to identify risk factors for physicians in practice,” says Dr. Tian.

“Preliminary findings suggest that a statistically significant relationship exists between the performance of physicians on the MCCQE Part II and their performance in practice on CPSO’s practice-based peer assessments,” confirms Dr. Tian. Specifically, candidates who passed the MCCQE Part II on their first attempt have a 76% probability of obtaining a satisfactory peer assessment outcome. While these preliminary results are promising, she cautions against overinterpreting these results as physician performance can be affected by many factors.

Following her poster presentation at the conference, Dr. Tian is looking forward to continuing to work with CPSO on this study. “This event also provided opportunities to network with colleagues from various Medical Regulatory Authorities (MRAs), which may lead to further collaborations to conduct outcome-based research.”

Examination performance:
possible predictor of patient complaints and prescribing patterns

A second poster presented at the conference by Dr. André De Champlain, Director of Psychometrics and Assessments Services at the MCC, addressed a different topic. Dr. De Champlain, other MCC psychometricians, and researchers from CPSA are presently focusing their attention on the possible relationship between MCC exam scores and patient complaints as well as physician opioid and benzodiazepine prescribing patterns. “The main use of our Qualifying Examinations is to provide the best evidence possible to MRAs that physicians are ready to enter both supervised and independent practice in Canada,” says Dr. De Champlain. “However, there has been mounting interest in using the scores for other purposes.” Both organizations are interested in looking at passing and failing scores on the first attempt on the MCCQE Part I and the MCCQE Part II, and determining whether those results are predictive of patient complaints and unsafe opioid as well as benzodiazepine prescribing habits.

The study’s preliminary findings indicate that candidates who fail the MCCQE Part I on the first attempt are the recipients of 27% more patient complaints than those who pass. Candidates who fail the MCCQE Part II on the first attempt are likely to prescribe opioids and benzodiazepines in high doses to 30% more patients, compared to physicians who pass the exam on the first attempt. “Those results are meaningful since they provide MRAs with information that might be useful to identify which physicians might need support at an early stage of their careers,” affirms Dr. De Champlain. Secondly, he adds that this research allows the MCC to collect validity evidence in defense of the use of licensing exams for purposes other than initial licensure.

Moving forward, the MCC and CPSA will begin work on the second part of this study, which will focus on assessing whether the MCCQE Part I and the MCCQE Part II initial results can help to predict different types of patient complaints. “We would also like to look more closely at the sample that was chosen to ensure that it represents the entire physician population in Alberta,” mentions Dr. De Champlain.