Reliability of simulation-based assessment for practicing physicians: performance is context-specific

Development of peer assessment rubrics in simulation-based learning for advanced cardiac life support skills among medical students

INTRODUCTION

Peer assessment can enhance understanding of the simulation-based learning (SBL) process and promote feedback, though research on its rubrics remains limited. This study assesses the validity and reliability of a peer assessment rubric and determines the appropriate number of items and raters needed for a reliable assessment in the advanced cardiac life support (ACLS) context.

METHODS

Ninety-five third-year medical students participated in the ACLS course and were assessed by two teachers (190 ratings) and three peers (285 ratings). Students rotated roles and were assessed once as a team leader on a ten-item rubric in three domains: electrocardiogram and ACLS skills, management and mechanisms, and affective domains. Messick's validity framework guided the collection of validity evidence.

RESULTS

Five sources of validity evidence were collected: (1) content: expert reviews and alpha, beta, and pilot tests for iterative content validation; (2) response process: achieved acceptable peer interrater reliability (intraclass correlation = 0.78, p = 0.001) and a Cronbach's alpha of 0.83; (3) internal structure: demonstrated reliability through generalizability theory, where one peer rater with ten items achieved sufficient reliability (Phi-coefficient = 0.76), and two raters enhanced reliability (Phi-coefficient = 0.85); construct validity was supported by confirmatory factor analysis. (4) Relations to other variables: Peer and teacher ratings were similar. However, peers rated higher in scenario management; further generalizability theory analysis indicated comparable reliability with the same number of teachers. (5) Consequences: Over 80% of students positively perceived peer assessment on a 5-point Likert scale survey.

CONCLUSION

This study confirms the validity and reliability of ACLS SBL rubrics while utilizing peers as raters. Rubrics can exhibit clear performance criteria, ensure uniform grading, provide targeted feedback, and promote peer assessment skills.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Nursing students' perceptions of identifying and managing sex trafficking cases: A focus group study

AIM

To explore nursing students' perceptions of the identification and medical care of sex trafficking victims and how to improve the care provided based on the identified barriers.

BACKGROUND

Despite the fact that more than 87% of sex trafficking victims establish contact with health professionals, their lack of awareness and training hinders their ability to assist victims properly.

METHODS

A descriptive qualitative study was carried out using 11 focus groups of nursing students after a simulated case of sex trafficking victim. The analysis of results was conducted through content analysis.

RESULTS

A total of 110 fourth-year nursing students participated in this study. After performing the data analysis, three main categories emerged: (i) making a health issue visible, (ii) identifying sex trafficking victims: resources based on evidence-based practice, and (iii) educational resources on sex trafficking in nursing degree.

CONCLUSIONS

Our results show that through clinical simulation, integrating content regarding the medical care of vulnerable groups can be useful for developing the skills necessary to provide effective care from a trauma-informed approach.

IMPLICATIONS FOR NURSING MANAGEMENT

Health institutions, senior charge nurses, and other health service managers should be aware of the current need for sex trafficking training, structural-level changes, and updated evidence-based guidelines and protocols with other service providers and law enforcement to provide high-quality care.