How are total scores and sub-scores calculated on the MCCQE Part I?
How is the pass score established?
How is a pass/fail decision made?
What will appear on the Statement of Results and Supplemental Feedback Report?
Enhancements to the SFR implemented in 2016
The Medical Council of Canada Qualifying Examination (MCCQE) Part I is scored independently of the MCCQE Part II. To be awarded the Licentiate of the Medical Council of Canada (LMCC), a candidate must pass both the MCCQE Part I and Part II.
The MCCQE Part I consists of 196 multiple-choice questions (MCQs) and 40-50 Clinical Decision-making (CDM) cases (approximately 80 questions). Of the 196 MCQs, 144 are scored questions and 52 are pilot questions that are being used for the first time. Following the statistical calibration of these 52 questions, decisions are made as to their use as scored questions. Typically, 80%-90% (42-47) of these MCQ questions are used as counting questions as well. Of the 80 CDM questions, approximately 65 are scored questions and the remaining are pilot questions not used in calculating candidates’ total score. While the non-scored questions do not count toward the total score, they are not identified as non-scored questions in the exam. Candidates are encouraged to do as well as they can on every question.
The MCCQE Part I, like all Medical Council of Canada (MCC) exams is a criterion-referenced exam. This means that candidates who meet or exceed the standard will pass the exam regardless of how well other candidates perform on it.
The total score is calculated by combining MCQs and CDMs. MCQs are scored 1 point for a correct answer and scored 0 for an incorrect answer. For CDMs, there is a maximum of 1 point per question. Some questions, such as a question whose answer includes four correct elements, allow candidates to receive partial marks (e.g., 0.25, 0.5, 0.75 or 1). The MCCQE Part I total score is reported on a scale that ranges from 50 to 950. There are eight sub-scores; seven discipline sub-scores (Family Medicine, Medicine, Obstetrics and Gynecology, Pediatrics, Population Health and the Ethical, Legal and Organizational aspects of Medicine, Psychiatry and Surgery) and one CDM sub-score. Discipline sub-scores include scores from MCQs only, with the exception of the Family Medicine sub-score that includes scores from MCQs and CDMs; the CDM sub-score includes scores from CDMs only.
The MCC conducts a standard setting exercise every three to five years to ensure that the standard and the pass score remain appropriate. Standard setting is a process used to define an acceptable level of performance and to establish a pass score.
In fall 2014, the MCC completed a rigorous standard setting exercise based on expert judgments from a panel of 17 physicians across the country that represented faculties of medicine, different specialties, and years of experience supervising students and residents. The Bookmark Method was used and has been successfully employed and defended with a number of large-scale exam programs around the world. Following the standard setting exercise, the recommended pass score was brought forward to, reviewed and approved by the Central Examination Committee (CEC) in June 2015. The technical report on the standard setting exercise for the MCCQE Part I provides additional information.
Using the spring 2015 results of all MCCQE Part I candidates, the new 50 to 950 scale was established to have a mean of 500 and a standard deviation of 100. Results from the spring 2015 and subsequent exam sessions will be reported using this new scale allowing us to compare candidate performance across sessions beginning with the spring 2015 session. For example, if a candidate’s score is 600, his or her ability level is one standard deviation above that of the reference group mean. On this new scale, the pass score that was recommended from the standard setting panel and approved by the CEC is 427. This pass score will remain in place until the next scheduled standard setting exercise in 2018.
Prior to 2015, the pass score for the MCCQE Part I was 390 on the old 50 to 950 scale. The new pass score (427) translates to 440 on this old scale.
A candidate’s final result (e.g., pass, fail) is determined by his or her total score and where it falls in relation to the examination pass score. A total score equal to or greater than the pass score is a pass and a total score less than the pass score is a fail. The candidate’s performance is judged in relation to the examination pass score and not judged on how well other individuals perform.
The SOR includes the candidate’s final result and total score, as well as the pass score. Additional information about discipline and CDM sub-scores and comparative information is provided on the SFR. A sample of the SOR and a sample of the SFR, containing mocked-up, random data, depict how information is presented to exam candidates. Both the SOR and the SFR are made available through the candidate’s physiciansapply.ca account.
Effective in 2016, the Supplemental Feedback Report (SFR), which is used by MCC to provide feedback to exam candidates, is being standardized across all examinations administered by MCC.
The 2016 enhancements to the SFR involves combining the display of sub-scores and quintiles into one figure and replacing quintiles with the mean sub-scores of first-time test takers who passed. The change is intended to enable candidates to better understand and gauge their performance on the specific test form of the examination that they have taken.