Advanced life support cardiac arrest scenario test evaluation

Suitability of GPT-4o as an evaluator of cardiopulmonary resuscitation skills examinations

AIM

To assess the accuracy and reliability of GPT-4o for scoring examinees' performance on cardiopulmonary resuscitation (CPR) skills tests.

METHODS

This study included six experts certified to supervise the national medical licensing examination (three junior and three senior) who reviewed the CPR skills test videos across 103 examinees. All videos reviewed by the experts were subjected to automated assessment by GPT-4o. Both the experts and GPT-4o scored the videos across four sections: patient assessment, chest compressions, rescue breathing, and repeated operations. The experts subsequently rated GPT-4o's reliability on a 5-point Likert scale (1, completely unreliable; 5, completely reliable). GPT-4o's accuracy was evaluated using the intraclass correlation coefficient (for the first three sections) and Fleiss' Kappa (for the last section) to assess the agreement between its scores vs. those of the experts.

RESULTS

The mean accuracy scores for the patient assessment, chest compressions, rescue breathing, and repeated operation sections were 0.65, 0.58, 0.60, and 0.31, respectively, when comparing the GPT-4o's vs. junior experts' scores and 0.75, 0.65, 0.72, and 0.41, respectively, when comparing the GPT-4o's vs. senior experts' scores. For reliability, the median Likert scale scores were 4.00 (interquartile range [IQR] = 3.66-4.33, mean [standard deviation] = 3.95 [0.55]) and 4.33 (4.00-4.67, 4.29 [0.50]) for the junior and senior experts, respectively.

CONCLUSIONS

GPT-4o demonstrated a level of accuracy that was similar to that of senior experts in examining CPR skills examination videos. The results demonstrate the potential for deploying this large language model in medical examination settings.

The nationwide impact of COVID-19 on life support courses. A retrospective evaluation by Resuscitation Council UK

Aim

To determine the impact of the COVID-19 pandemic on Resuscitation Council UK Advanced Life Support (ALS) and Immediate Life Support (ILS) course numbers and outcomes.

Methods

We conducted a before-after study using course data from the Resuscitation Council UK Learning Management System between January 2018 and December 2021, using 23 March 2020 as the cut-off between pre- and post-pandemic periods. Demographics and outcomes were analysed using chi-squared tests and regression models.

Results

There were 90,265 ALS participants (51,464 pre-; 38,801 post-) and 368,140 ILS participants (225,628 pre-; 142,512 post-). There was a sharp decline in participants on ALS/ILS courses due to COVID-19. ALS participant numbers rebounded to exceed pre-pandemic levels, whereas ILS numbers recovered to a lesser degree with increased uptake of e-learning versions. Mean ALS course participants reduced from 20.0 to 14.8 post-pandemic (P < 0.001).Post-pandemic there were small but statistically significant decreases in ALS Cardiac Arrest Simulation Test pass rates (from 82.1 % to 80.1 % (OR = 0.90, 95 % CI = 0.86-0.94, P < 0.001)), ALS MCQ score (from 86.6 % to 86.0 % (mean difference = -0.35, 95 % CI -0.44 to -0.26, P < 0.001)), and overall ALS course results (from 95.2 %to 94.7 %, OR = 0.92, CI = 0.85-0.99, P = 0.023). ILS course outcomes were similar post-pandemic (from 99.4 % to 99.4 %, P = 0.037).

Conclusion

COVID-19 caused a sharp decline in the number of participants on ALS/ILS courses and an accelerated uptake of e-learning versions, with the average ALS course size reducing significantly. The small reduction in performance on ALS courses requires further research to clarify the contributing factors.

Demonstrating equivalence and non-inferiority of medical education concepts

CONTEXT

In medical education, there are often several didactic methods for teaching and learning a specific medical skill. For educators, there are often pragmatic reasons to decide for one or another of them, such as costs, infrastructural requirements, time expenditure or qualification of the teacher. However, a central aspect to consider is the learning outcome: Does a new method achieve a similar learning success as an established standard method? To answer this question, we need an appropriate method to assess comparability of learning outcomes.

METHODS

In this paper, we present two essential statistical concepts that can address the issue of comparability of learning outcomes: Equivalence and non-inferiority testing. We explain the ideas behind these concepts and illustrate them with an example data set. To clarify several concepts, we use theoretical examples from one selected field: the teaching and assessment of Basic Life Support (BLS).

CONCLUSIONS

Equivalence and non-inferiority tests can be powerful tools for comparing teaching and assessment methods. However, their correct application requires adequate knowledge about their strengths, pitfalls and application fields. The aim of this paper is to deliver this knowledge and to provide clinician researchers with a practical guidance to a successful application of these methods.

Feedback in advanced life support: A quality improvement initiative

AIM

To identify participant, course characteristics and centre factors associated with participant satisfaction and ALS outcomes.

METHODS

17,690 participants enrolled on ALS courses between 1st December 2017 and 30th November 2018. Participant, course and centre characteristics were explored in relation to course learning outcomes and participant experience. Learning outcomes were assessed through a post-course MCQ score and technical and non-technical skills through a cardiac arrest simulation test (CAS-Test). Successful completion of knowledge and skill-based assessments led to overall course success. Participant feedback was collected on a post-course questionnaire. Multivariable analyses identified variables associated with course outcomes and feedback. Adjusted funnel plots compared inter-course centre outcomes.

RESULTS

Mean post-course MCQ score was 86.7% (SD = 6.7). First attempt CAS-Test pass rate was 82.6% and overall course pass rate 94.4%. Participant characteristics explained the majority of variation between course centres. Characteristics associated with knowledge, skill and course outcomes were age, prior experience, pre-course MCQ score, course type, ethnicity, place of work, profession and seniority. Feedback scores were predicted by course type, pre-course MCQ, ethnicity, profession and seniority.

CONCLUSION

This is the first study to identify variables associated with both ALS feedback scores and assessment outcomes. It has demonstrated that both course outcomes and participant experience are similar across a large number of course centres. Identifying the demographic traits of participants who may struggle with ALS, may enable bespoke support from an earlier stage. Analysis of feedback scores and outcomes enables ongoing appraisal and targeted improvement of the Resuscitation Council UK ALS course.

Experience of emergency healthcare professionals in cardiopulmonary resuscitation and its relationship with self-efficacy: A qualitative approach

AIM

To gain a deeper knowledge of the phenomenon of cardiopulmonary resuscitation by capturing the meanings that professionals attach to the behaviour of reanimation and its relationship with Social Cognitive Theory.

METHOD

Qualitative research from a phenomenological approach focussing on field work with a focus group of professionals from Madrid (doctors, nurses and technicians), as well as a previous study. After the transcription of the data, it was coded, categories were formed and it was grouped into dimensions, interpreting the data according to the Social Cognitive Theory. Data was collected from other sources, experience of professionals and family members who had witnessed a cardiopulmonary resuscitation, until the saturation of the data.

RESULTS

The phenomena that affect the people involved in a cardiopulmonary resuscitation were explored holistically. The relevant aspects to feel able to perform cardiopulmonary resuscitation, as well as those that may hinder the behaviour were identified. Two dimensions were found, personal and situational factors, as well as several categories in each dimension related to Bandura's theory of self-efficacy.

CONCLUSIONS

Professionals' self-efficacy in a cardiopulmonary resuscitation does not always depend on the technical skills acquired. Personal and situational factors influence the professional being able to vary their behaviour cognitively and emotionally. This knowledge enables cardiopulmonary resuscitation competencies in a clinical situation to be improved and individualized training programmes to be designed in teaching practice.

e-Learning in Advanced Life Support-What factors influence assessment outcome?

AIM

To establish variables which are associated with favourable Advanced Life Support (ALS) course assessment outcomes, maximising learning effect.

METHOD

Between 1 January 2013 and 30 June 2014, 8218 individuals participated in a Resuscitation Council (UK) e-learning Advanced Life Support (e-ALS) course. Participants completed 5-8h of online e-learning prior to attending a one day face-to-face course. e-Learning access data were collected through the Learning Management System (LMS). All participants were assessed by a multiple choice questionnaire (MCQ) before and after the face-to-face aspect alongside a practical cardiac arrest simulation (CAS-Test). Participant demographics and assessment outcomes were analysed.

RESULTS

The mean post e-learning MCQ score was 83.7 (SD 7.3) and the mean post-course MCQ score was 87.7 (SD 7.9). The first attempt CAS-Test pass rate was 84.6% and overall pass rate 96.6%. Participants with previous ALS experience, ILS experience, or who were a core member of the resuscitation team performed better in the post-course MCQ, CAS-Test and overall assessment. Median time spent on the e-learning was 5.2h (IQR 3.7-7.1). There was a large range in the degree of access to e-learning content. Increased time spent accessing e-learning had no effect on the overall result (OR 0.98, P=0.367) on simulated learning outcome.

CONCLUSION

Clinical experience through membership of cardiac arrest teams and previous ILS or ALS training were independent predictors of performance on the ALS course whilst time spent accessing e-learning materials did not affect course outcomes. This supports the blended approach to e-ALS which allows participants to tailor their e-learning experience to their specific needs.

Impact of a novel, resource appropriate resuscitation curriculum on Nicaraguan resident physician's management of cardiac arrest

PURPOSE

Project Strengthening Emergency Medicine, Investing in Learners in Latin America (SEMILLA) created a novel, language and resource appropriate course for the resuscitation of cardiac arrest for Nicaraguan resident physicians. We hypothesized that participation in the Project SEMILLA resuscitation program would significantly improve the physician's management of simulated code scenarios.

METHODS

Thirteen Nicaraguan resident physicians were evaluated while managing simulated cardiac arrest scenarios before, immediately, and at 6 months after participating in the Project SEMILLA resuscitation program. This project was completed in 2014 in Leon, Nicaragua. The Cardiac Arrest Simulation Test (CASTest), a validated scoring system, was used to evaluate performance on a standardized simulated cardiac arrest scenario. Mixed effect logistic regression models were constructed to assess outcomes.

RESULTS

On the pre-course simulation exam, only 7.7% of subjects passed the test. Immediately post-course, the subjects achieved a 30.8% pass rate and at 6 months after the course, the pass rate was 46.2%. Compared with pre-test scores, the odds of passing the CASTest at 6 months after the course were 21.7 times higher (95% CI 4.2 to 112.8, P<0.001). Statistically significant improvement was also seen on the number of critical items completed (OR=3.75, 95% CI 2.71-5.19), total items completed (OR=4.55, 95% CI 3.4-6.11), and number of "excellent" scores on a Likert scale (OR=2.66, 95% CI 1.85-3.81).

CONCLUSIONS

Nicaraguan resident physicians demonstrate improved ability to manage simulated cardiac arrest scenarios after participation in the Project SEMILLA resuscitation course and retain these skills.

Advanced life support provider course in Italy: A 5-year nationwide study to identify the determinants of course success

INTRODUCTION

The advanced life support (ALS) provider course is the gold standard for teaching and assessing competence in advanced resuscitation. Outcomes over a 5-year period of European Resuscitation (ERC)/IRC ALS provider courses in Italy were investigated, and the factors associated with course success are described.

METHODS

In 2008, the Italian Resuscitation Council (IRC) created a database in which every ERC/IRC ALS course was recorded. Data from courses organized from 2008 to 2012 were analysed. The data included: candidate's age and degree (medical doctor (MD) or nurse), medical specialty of MD candidates, course outcomes, duration and reference guidelines, number of instructors and course director. Relationships between the course outcomes and the courses and candidates' characteristics were analysed using logistic regression.

RESULTS

A total of 13,624 candidates were evaluated from 871 courses. Among the candidates, 55% were MDs and 45% were nurses. Ninety-seven percent of candidates passed the final evaluation, while 3% failed. Candidates who passed were younger (37 [31-44] vs. 43 [37-50] years, p<0.0001) and had a greater pre-course resuscitation knowledge (multiple choice quiz (MCQ) score: 88 [83-93] vs. 80 [73-87], p<0.0001) compared to those who failed. The course pass rate was higher for MDs compared to nurses (98% vs. 95%, p<0.0001) and participants in emergency disciplines were most significantly associated with course success (χ(2) 71, p<0.0001). In the multivariate analysis, an older age (OR 0.926, 95%CI [0.915-0.937]) was independently associated with course failure, while being a MD (OR 3.021, 95%CI [2.212-4.132]), having a higher pre-course MCQ score (OR 1.033, 95%CI [1.026-1.040]) together with a higher candidate/instructor ratio (OR 1.314, 95%CI [1.067-1.618]), and having a longer course duration (OR 1.717, 95%CI [1.090-2.703]), were independently associated with success.

CONCLUSIONS

Younger age, professional background, and pre-course resuscitation knowledge are the most important predictors of ALS provider course success, together with higher candidate/instructor ratios and longer course durations.

E-learning in advanced life support--an evaluation by the Resuscitation Council (UK)

AIM

To descriptively analyse the outcomes following the national roll out of an e-Learning advanced life support course (e-ALS) compared to a conventional 2-day ALS course (c-ALS).

METHOD

Between 1st January 2013 and 30th June 2014, 27,170 candidates attended one of the 1350 Resuscitation Council (UK) ALS courses across the UK. 18,952 candidates were enrolled on a c-ALS course and 8218 on an e-ALS course. Candidates participating in the e-ALS course completed 6-8h of online e-Learning prior to attending the 1 day modified face-to-face course. Candidates participating in the c-ALS course undertook the Resuscitation Council (UK) 2-day face-to-face course. All candidates were assessed by a pre- and post-course MCQ and a practical cardiac arrest simulation (CAS-test). Demographic data were collected in addition to assessment outcomes.

RESULTS

Candidates on the e-ALS course had higher scores on the pre-course MCQ (83.7%, SD 7.3) compared to those on the c-ALS course (81.3%, SD 8.2, P<0.001). Similarly, they had slightly higher scores on the post-course MCQ (e-ALS 87.9%, SD 6.4 vs. c-ALS 87.4%, SD 6.5; P<0.001). The first attempt CAS-test pass rate on the e-ALS course was higher than the pass rate on the c-ALS course (84.6% vs. 83.6%; P=0.035). The overall pass rate was 96.6% on both the e-ALS and c-ALS courses (P=0.776).

CONCLUSION

The e-ALS course demonstrates equivalence to traditional face-to-face learning in equipping candidates with ALS skills when compared to the c-ALS course. Value is added when considering benefits such as increased candidate autonomy, cost-effectiveness, decreased instructor burden and improved standardisation of course material. Further dissemination of the e-ALS course should be encouraged.

Which factors predict candidate outcome in advanced life support courses? A preliminary observational study

PURPOSE

To identify factors associated with candidate outcome in the European Resuscitation Council (ERC) advanced life support (ALS) provider courses.

METHODS

Medical doctors participating as candidates to consecutive ALS courses organised by an ERC training centre in Italy were enrolled in this prospective cohort study. The association between the ALS course outcome and candidate demographics, professional background and pre-course knowledge measured by using the pre-course multiple choice quiz (MCQ) was investigated by using logistic regression.

RESULTS

A total of 283 candidates, median age 31 years, were evaluated. Among them, 269 (95.1%) passed the final evaluation and 14 (4.9%) failed. Candidates who passed were younger (median age 31 vs. 37.5 years; p = 0.006) and attained a higher pre-course MCQ score (median 84 vs. 72.5%; p < 0.0001). On multivariate analysis, a higher pre-course MCQ score (OR 1.18 [95%CI 1.09-1.28]) and a basic life support (BLS) certification (OR 5.00 [95%CI 1.12-22.42]) were independent predictors of candidate success, while older age was associated with a significantly higher risk of failing (OR 0.90 [95%CI 0.83-0.97]). Female candidates had higher pass rates (97.2 vs. 91.2%; p = 0.048); however, after correction for confounders gender was not significantly associated with candidate outcome. Neither candidate specialty nor site of work was a predictor of candidate success.

CONCLUSIONS

On ALS courses, younger age and a higher level of specific pre-course knowledge, as measured by both the pre-course MCQ and the presence of BLS certification, are the most important predictors of success. Candidate gender and professional background did not show a significant correlation with course outcome.