Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta

Morten Engberg, Lars Lönn, Lars Konge, Søren Mikkelsen, Tal Hörer, Hans Lindgren, Edmund Søvik, Morten Bo Svendsen, Martin Frendø, Mikkel Taudorf, Lene Russell

Research output: Contribution to journalJournal articleResearchpeer-review

Abstract

BACKGROUND: Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE.

METHODS: This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard.

RESULTS: Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed.

CONCLUSION: Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts.

LEVEL OF EVIDENCE: Diagnostic test, no or poor gold standard, level V.

Original languageEnglish
JournalThe Journal of Trauma and Acute Care Surgery
Volume91
Issue number4
Pages (from-to)663-671
ISSN2163-0755
DOIs
Publication statusPublished - Oct 2021

Bibliographical note

Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.

Keywords

  • Resuscitative endovascular balloon occlusion of the aorta
  • clinical competence
  • simulation
  • skills
  • validation
  • validity evidence
  • vascular access, assessment tool

Fingerprint

Dive into the research topics of 'Reliable and valid assessment of procedural skills in resuscitative endovascular balloon occlusion of the aorta'. Together they form a unique fingerprint.

Cite this