2016-2017 – Irene Ma

Background:

Competence in point of care ultrasound (POCUS) is an expectation for many residency training programs. There is a need for valid and reliable assessment for POCUS skills. Although easy to use, checklists may be less reliable and less sensitive to expertise levels compared to global ratings. However, modifying checklist scores based on essential items may result in an improved ability to identify competence, compared with the unmodified overall checklist score or minimum passing scores (MPS) set by traditional methods.

Purpose:

This study seeks to develop a POCUS checklist and compare the ability of the checklist to identify competence based on a) the overall checklist score, b) completion of all essential items, and c) achieving MPS.

Methods:

This study will be divided into three phases:

Phase 1: Checklist items and determination of importance of items (essential, important, marginal) will be established by consensus from an international panel of at least 15 POCUS experts, using the Delphi method, over 2-3 rounds of an online survey.

Phase 2: Standard setting of checklist scores will be undertaken by a minimum of 10 experts using the Angoff method.

Phase 3: Ten video-recorded performances of established competence and incompetence will be rated by a minimum of 30 experts using the checklist. The sensitivity and specificity of checklist score in identifying competence will be evaluated at various cut points with a Receiver Operating Characteristic analysis.

Diagnostic accuracy will be estimated by the Area Under the Curve. The agreement between achieving MPS (versus not) and competence will be evaluated using Kappa statistic. The agreement between completing all essential items on the checklist (versus not) and competence will also be evaluated using the Kappa statistic.

Implications:

Results of this study will provide insight into the role of essential items in competency assessment for POCUS.