The TDM Examination goes computer-based

For the past several years, the Medical Council of Canada (MCC) has been making efforts to modernize its assessments and operations with a view to improving the candidate’s experience, the quality of exams, as well as the efficiency of exam administration. One of the most recent of such efforts is the transition of the Therapeutics Decision-Making (TDM) Examination from a paper-based test to a Computer-Based Test (CBT) that will be operational in October 2020.

The TDM Examination is a step in the selection process for entry into Practice-Ready Assessment (PRA) programs. It assesses the competence of International Medical Graduates at the level required of a family physician practising independently and safely in Canada. The content consists of patient-based scenarios representing common or important clinical presentations and problems that family physicians practising in Canada should be able to competently manage within various dimensions of medical care, such as illness prevention, acute illness and chronic illness.

An important transition

Currently, the MCC develops the content of the paper-based exam, prints the exam booklets and ships them to provincial PRA programs who then administer the exam themselves. The booklets are shipped back to the MCC for marking by physician markers.

In October 2020, the TDM Examination will be offered electronically in Prometric Test Centres across Canada. Like the MCC’s other CBT assessment, the Medical Council of Canada Qualifying Examination (MCCQE) Part I, the TDM Examination will be offered through an exam delivery platform called Surpass. Offering the exam in this way will help create a familiar and consistent exam-day experience, because TDM candidates will already have completed the MCCQE Part I.

Another important benefit of this computerization initiative is that it creates new research and development opportunities. “It will facilitate conducting research to inform test development and operations, and ultimately improve the efficiency of processing as well as the overall quality of this assessment tool,” says Dr. Fang Tian, Senior Research Psychometrician with the MCC’s Psychometrics and Assessment Services (PAS) team.

Research to improve assessment

According to Dr. Qi Guo, Research Psychometrician and lead researcher for the TDM Examination, there are three main areas of work that PAS will be concentrating on. The first is automated marking, which has already been developed for the MCCQE Part I and has been found to be accurate and efficient. Automated marking uses Natural Language Processing (NLP) to assign an answer key label to each candidate response. The second area of research is in Automated Item Generation (AIG), which uses NLP algorithms to automatically generate new exam content for future iterations of the TDM Examination, based on a content structure established by subject matter experts. AIG helps keep the content fresh for successive cohorts of applicants and helps maintain the exam content bank.

“I would like to clarify that all this automation doesn’t get rid of the human entirely,” Dr. Guo points out. “Qualified people are still required for marking and to validate the automatically generated items.” As impressive as this technology is, humans are, in fact, still involved at every step of the process. Automation simply allows for quicker and more streamlined operations.

And finally, unlike the paper-based test, the CBT will allow us to see, for example, the time it takes candidates to complete a given question. “If a question takes many candidates an unusually short amount of time to complete,” Dr. Guo explains, “this could suggest an issue.” Alternatively, if a question is found to take candidates an unusually long time to complete, this could mean the question is unclear and needs to be rewritten. After analysis, we can flag certain questions for content review. In short, this area of research is aimed at improving exam security and the quality of the exam content.

Although moving away from paper requires a high level of effort, this important transition to a computerized administration of the TDM Examination will lead to a higher level of efficiency, accuracy, exam security and ultimately, to an improved assessment.