2019-2020 - W. Tavares

Background

Inferences regarding competence and areas in need of correction are inherent in performance-based assessments, and yet existing strategies emphasize only behaviours without taking into consideration the potentially more meaningful cognitive framing (i.e., interpretive schemes) that drive candidate/clinician behaviours. This leaves observers/assessors to make inferences about and/or derive interpretations of the meaning of observed behaviours. However, the same behavior may result from appropriate and inappopriate cognitive framing, leaving this process uncertain and a possible source or error when extrapolating to whether the individual would perform appropriately in a different situation or how to improve the performance.

Purpose

To explore in what way direct observation combined with knowledge of candidate/clinician “interpretive schemes” influences raters’ perceptions of performance.

Methods

We designed an exploratory between-groups sequential mixed methods study. We will begin by exploring the degree and type of interpretive schemes that exist in response to clinical performance stimuli. Next, participants will be asked to observe and assess clinical performances in one of three randomized experimental conditions: (1) with interpretive schemes that are appropriate for the behaviours exhibited; (2) with interpretive schemes that are inappropriate for the behaviours exhibited; (3) with no interpretive schemes available (i.e., a control condition). Outcome measures include indicators of assessment quality (e.g., reliability) and feedback quality (i.e., agreement on topics/content to include). Post-task interviews will explore the role and relevance of interpretive schemes.

Implications

Knowledge of interpretive schemes may provide data that supplement and strengthen validity claims by improving what we know about the behaviours observed. This assumes knowledge of interpretive schemes will be helpful, not confusing, and that assessors will use and/or know how to use the information when formulating judgments or feedback. If access to interpretive schemes lead to improved outcomes, rater training and assessment processes may take advantage of existing strategies to access such data.