Research in Clinical Assessment grant

To support medical assessment research, the MCC offers research grants to interested faculty members, staff members or graduate students of Canadian medical faculties.

2015-2016 - Christina St-Onge

Which, Why and How are Item Analysis Guidelines Applied When Monitoring the Quality of Examinations? Interviews with Key Informants

Investigator

Christina St-Onge, PhD

Co-investigators

M. Young

L. Varpio

J.-S. Renaud

B.-A. Cummings

Abstract

Introduction:

The use of MCQs necessitates the monitoring of their quality (item difficulty and discrimination coefficients). Guidelines are available but were developed for use in large cohorts. However, within medical education programs examination cohorts (eg.: a clerkship rotation) are variable in size, and often as small as 10-12 students. Applying item guidelines to such small cohorts may be problematic depending on the choice and use of guidelines. We do not know which item analysis guidelines are being in the various medical education settings, nor why they are chosen or how they are applied. Meaning we know little about what contextual factors influence the use and application of item analysis guidelines.

Purpose:

Document MCQ quality monitoring practices of key informants in various medical education settings, why item analysis guidelines are chosen and how they are applied (strategies to consider the different contextual factors).

Methods:

We will conduct semi-structured interviews with key informants in various medical education settings in order to understand their choice of item analysis guidelines and their strategies (whether adaptive or contextually driven) when using the guidelines. A purposive sampling strategy will be employed to include key informants from different settings in medical education (such as UGME, PGME and certification) and will be complemented with a snowball sampling strategy. We will aim for a minimum of 15 participants. Analysis will rely on a qualitative description approach in the language that the session was conducted (English or French; since all team members are sufficiently bilingual).

Anticipated results:

We aim to be able to provide practitioner with a summary of the different item analysis guidelines being used (guidelines that exist/are currently used) and how they can be used cautiously (i.e., that factors should be considered when applying item analysis guidelines based on the current literature in addition to actual pragmatic practices.