Research in Clinical Assessment grant

To support medical assessment research, the MCC offers research grants to interested faculty members, staff members or graduate students of Canadian medical faculties.

2023-2024 – Kathy Boutis

Performance-based competency standard setting in medical imaging diagnosis: Balancing performance expectations with patient safety and learner experience

Investigator

Kathy Boutis, MD, FRCPC, MSc

Co-investigators

M. Pakkal

M. Perez

J. Taylor

C. Hague

M. Pusic

Abstract

Background:

For clinical skills that can be measured against a reference standard, group consensus often derives a pragmatic numeric competency standard. However, there is limited consequence validity evidence on this approach, and critics are concerned that a pragmatic standard allows post-graduate (PGT) to miss a percentage of assessment items without attention to which items are being missed, potentially resulting in patient safety concerns. In contrast, demanding near perfect diagnostic performance may not be feasible given all the skills required in PG training. Thus, this research will examine the patient-safety/feasibility trade-offs that result from implementing different performance thresholds in a common clinical skill, medical image interpretation.

Objectives:

In a cohort of radiology PGTs who deliberately practice a diagnostic image to a performance standard, we will determine if there is a difference in clinically important diagnostic errors (proxy for patient safety consequences) if the performance standard is set at 75% accuracy, sensitivity, and specificity (Group 1; reference standard) versus 85% (Group 2) versus 95% (Group 3). We will also compare the groups for the following learner consequences: (1) feasibility by determining the proportion of participants that achieve the competency standard; and (2) acceptability.

Methods:

An online customized software platform will be developed to present 500 chest radiographs that enables participants to diagnose each radiograph and get feedback after every case encounter. Participants will be randomized to achieve one of the three aforementioned performance standards. Upon achieving the standard, participants will complete an acceptability questionnaire.

Significance:

The “right” competency standard remains relatively unknown and is often uniformed by data, and this research will provide the data to include patient-safety and learner considerations in selecting a competency standard. MCC grant-specific themes relevant to this submission are outcomes assessment and use of technology for assessment.