Research in Clinical Assessment grant

To support medical assessment research, the MCC offers research grants to interested faculty members, staff members or graduate students of Canadian medical faculties.

Recipient: 2019-2020 - Dr. A. Siddiqui

Artificial Intelligence for automated evaluation of high-fidelity simulation in anesthesia

Investigator

A. Siddiqui, BHSc, MD, MEd, FRCPC

Co-investigators

T. Everett

Abstract

Introduction

As post-graduate medical education shifts towards competency-based medical education (CBME), a greater emphasis is placed on evaluation, feedback and assessment [1]. Under CBME, the increasing demand for numerous high-quality evaluations has placed a considerable burden on educators [1]. One mode of assessment that is commonly deployed in medical education is high-fidelity simulation. One example in anesthesiology is the Managing Emergencies in Pediatric Anesthesia (MEPA) course [2]. Through administration of the MEPA course, our group has developed a large database of recorded highfidelity simulations in the pre-determined MEPA curriculum. Big data, machine learning and artificial intelligence (AI) technologies have made significant advancements over the course of the last decade and provide opportunity for advancement in medical education. The purpose of this project is to create an automated evaluation algorithm using AI algorithms to evaluate anesthesiology trainees undertaking a standardized simulated operating room crisis (anaphylaxis). This grant application is in keeping with the MCC’s preferred research theme of “assessments along the continuum” and will provide a multidisciplinary approach towards evaluation in medical education by combining medical assessment with the computer science field of machine learning and AI.

Methods

This study will be completed in two stages. The purpose of stage 1 is to use the anaphylaxis video database to train and validate a machine-learning algorithm using previously recorded and human rater evaluated high-fidelity anaphylaxis simulation videos. The purpose of stage 2 is to use the AI-generated algorithm to score previously un-rated videos and compare those scores to those attributed by the human rater at the time. Agreement will be scored using intra-class correlation coefficients.

Outcomes

We anticipate that there will be strong agreement between AI computer-generated scores and expert faculty scores in the evaluation of anesthesiology trainees undertaking an operating room simulation scenario.