Designing simulation-based curriculum content for emergency medicine residents in France: a Delphi method

BACKGROUND

The value of simulation in emergency medicine is indisputable because it allows training and acquisition of many technical and non-technical skills (TS and NTS). In France, there are no curriculum regarding the use of simulation as a teaching tool during emergency medicine residency. The aim of this study was to design the content of a national simulation-based curriculum for emergency medicine residency programs.

METHODS

The Delphi method was conducted between March and June 2022. The questionnaire was divided into three sections: TS, NTS and clinical situations as starting points (SSPs). A panel of emergency physicians' experts on simulation education was established. An online survey was conducted in which they were asked to score, on a four-point Likert scale, the suitability of skills and SSPs to be taught through simulation courses during the emergency medicine residency. The questionnaire was revised between each round following comments or suggestions for additional items from the experts.

RESULTS

Sixty-six experts completed the Delphi process. The initial questionnaire included 64 TS, 37 NTS and 103 SSPs. The experts' comments led to the addition of 12 TS, 24 NTS and 6 SSPs. Consensus was obtained after three rounds. The experts selected 24 TS and 20 NTS to be taught as a priority through simulation during the emergency medicine residency, and 15 SSPs to be used in priority.

CONCLUSION

With a Delphi method, French experts in simulation-based emergency medicine education have selected 24 technical and 20 non-technical skills to be taught as a priority with simulation-based training to emergency medicine residents.

Testing the validity of three acute care assessment tools for assessing residents' performance during in situ simulation: the ACAT-SimSit study

BACKGROUND

The assessment of technical and nontechnical skills in emergency medicine requires reliable and usable tools. Three Acute Care Assessment Tools (ACATs) have been developed to assess medical learners in their management of cardiac arrest (ACAT-CA), coma (ACAT-coma) and acute respiratory failure (ACAT-ARF).

OBJECTIVE

This study aims to analyze the reliability and usability of the three ACATs when used for in situ (bedside) simulation.

METHODS

This prospective multicenter validation study tested ACATs using interprofessional in situ simulations in seven emergency departments and invited training residents to participate in them. Each session was rated by two independent raters using ACAT. Intraclass correlation coefficients (ICC) were used to assess interrater reliability, and Cronbach's alpha coefficient was used to assess internal consistency for each ACAT. The correlation between ACATs' scores and the learners' level of performance was also assessed. Finally, a questionnaire and two focus groups were used to assess the usability of the ACATs.

RESULTS

A total of 104 in situ simulation sessions, including 85 residents, were evaluated by 37 raters. The ICC for ACAT-CA, ACAT-coma and ACAT-ARF were 0.95 [95% confidence interval (CI), 0.93-0.98], 0.89 (95% CI, 0.77-0.95) and 0.92 (95%CI 0.83-0.96), respectively. The Cronbach's alphas were 0.79, 0.80 and 0.73, respectively. The ACAT-CA and ARF showed good construct validity, as third-year residents obtained significantly higher scores than first-year residents ( P  < 0.001; P  < 0.019). The raters supported the usability of the tools, even though they expressed concerns regarding the use of simulations in a summative way.

CONCLUSION

This study reported that the three ACATs showed good external validity and usability.

Usability and reproducibility of three tools to assess medical students and residents in emergency medicine

OBJECTIVES

It is critical to assess competency of medical students and residents in emergency medicine (EM) during undergraduate and graduate medical education. However, very few valid tools exist to assess both technical and nontechnical skills in the specific context of EM. Three Acute Care Assessment Tools (ACAT 1, 2, and 3) have been previously developed for three acute care conditions: cardiac arrest (1), coma (2), and acute respiratory failure (3). This study aimed to evaluate the reproducibility of the tools.

METHODS

The tool was tested using recorded videos of simulation sessions of fourth year medical students and first year residents in EM. Raters independently reviewed the videos two times in a 3-month interval, and interrater and intrarater reliability using intraclass correlation (ICC) was calculated. Secondary endpoints included the completeness rate and relevance of the ACAT.

RESULTS

Sixty-two sessions were recorded and 48 videos analyzed (18 for CA and 15 for both respiratory failure and coma. The learners were residents in 32 (66%) of videos. Interrater reliability was excellent (ICC >0.9 for all three contexts) and so was the intrarater reliability (>0.88), both upon first review (month 0, M0) and at 3 months (M3). The usability of the ACAT was good, with a completeness of the items that ranged from 96% to 100%. Only one item of the ACAT 1 had a relevance of 27%, as it could not be completed in 13 scenarios out of 18.

CONCLUSIONS

The results demonstrate educators can evaluate students similarly utilizing video recordings of simulated medical scenario. The excellent completeness of the rated items advocated for good usability. The three ACATs can be utilized to assess for completeness of predefined tasks in three acute care broad scenario in a competency-based medical education framework.