A Review of Currently Published Evaluation Instruments for Human Patient Simulation

Criteria to evaluate graduate nurse proficiencies in obtaining a health history and perform physical assessment in simulation-based education: A narrative review

BACKGROUND

Simulation is a technique being used increasingly in healthcare education which offers opportunities to evaluate nursing proficiencies. The use of valid and reliable instruments is recognised as the foundation for a robust assessment, however competency-based health assessment courses for graduate nurses can consequently become reductionist in measuring proficiencies.

OBJECTIVE

The specific review question was: In simulation-based education, what are the criteria that evaluate graduate nursing student's competence in obtaining a health history and performance of patient assessment?

METHODS

Eleven studies were included in the review. Papers were critically appraised with The Joanna Briggs Institute quasi-experimental studies checklist. Bloom's taxonomy was used to structure this narrative review.

RESULTS

Seven papers evaluated cognition through questionnaires and two papers used a Likert-scale to determine self-perceived knowledge. Six papers evaluated psychomotor skills with a behavioural checklist. Diversity of application was factored into the studies when testing affective skills. Three papers used Likert-scales to evaluate preparedness, six papers used Likert-scales to evaluate self-confidence and one used a Likert-scale to evaluate autonomy. Three papers used a checklist to evaluate professionalism. Four papers used faculty member/ standardised patient feedback.

CONCLUSION

Reductionist evaluation instruments create a barrier when evaluating competency. The limited validity and reliability of assessment instruments in simulation, as well as the lack of standardisation of affective skills assessment, presents a challenge in simulation research. Affective skills encompass attitudes, behaviours and communication abilities, which pose a significant challenge for standardised assessments due to their subjective nature. This review of the simulation literature highlights a lack of robustness in the evaluation of the affective domain. This paper proposes that simulation assessment instruments should include the standardisation of affective domain proficiencies such as: adaptation to patients' cognitive function, ability to interpret and synthesise relevant information, ability to demonstrate clinical judgement, readiness to act, recognition of professional limitations and faculty/standardised-simulated patient feedback. The incorporation of the affective domain in standardised assessment instruments is important to ensure comprehensive assessment of simulation particularly in the development of health history and physical assessment proficiencies. Attention to all of the domains in Blooms taxonomy during simulation assessment has the potential to better prepare professionals for the patient care setting.

A Mathematical Model Based on Stratifying the Severity of Medical Errors for Building Scenarios for Clinical Cases With Branching

Background There are no mathematical models or score systems available for assessing and creating clinical case simulations based on branching scenario scripts. Objective This study aimed to develop a mathematical model based on stratifying the severity of medical errors for building clinical cases with branching scenarios for clinical simulation. Methods This study was undertaken from August 2020 to August 2023. To build a mathematical model for building scenarios of clinical cases with branching, the classification of the seriousness of medication errors was used. A mathematical model was built for predicting and modeling the development of a clinical situation and as an assessment strategy. The study recruited a total of 34 participants, with 16 participants assigned to the branching scenarios without the mathematical model group and 18 participants assigned to the branching scenarios with the mathematical model group. Results A simple diagram of score based on stratification of the severity of medical errors and correct decisions in clinical practice for building interactive training scenarios with branching was proposed. According to this score system algorithm, each clinical decision-making step is scored points with plus or minus, from 0 to 10. The sum of the points for each block in the decision-making process is then added up. Each step in the overall clinical decision-making strategy is stratified by the proposed algorithm, and finally, the results of internal validation and implementation are presented. Conclusion A mathematical model and score system for building clinical case scenarios based on branching and classification of the seriousness of medication errors was developed. This system could help in the prediction and modeling of the development of events in particular clinical situations and the assessment of competency formation in medical simulation as well.

Preparing for practice, the effects of repeated immersive simulation on the knowledge and self-efficacy of undergraduate nursing students: A mixed methods study

AIM

The aim of this study was to compare the effects of two immersive simulation-based education instructional designs, immersive simulation with team deliberate practice and immersive repeated standard simulation, when delivered over the same time on the knowledge and self-efficacy of nursing students.

BACKGROUND

Implementing immersive simulation-based education is not without its resource challenges, making it prohibitive for simulation educators to include it in their curricula. Subsequently, there is a need to identify instructional designs that meet these challenges.

DESIGN

A two-stage mixed methods approach was used to compare the two instructional designs.

METHODS

In stage one, data were collected using questionnaires and differences estimated using analysis of covariance. In stage two, data were collected from two focus groups and analysed using a qualitative content analysis approach. Data were collected as part of a doctoral study completed in 2019 and was analysed for this study between 2022 and 2023. The justification for this study was that the identification of effective designs for immersive simulation remains a key research priority following the increase in allowable simulation hours by the Nursing and Midwifery Council.

RESULTS

In stage one, there was no statistical significance in the participant's knowledge or self-efficacy between the models. In stage two, four themes were identified: vulnerability, development of knowledge, development of self-efficacy and preparation for placement. In contrast to stage one, participants reported that the repeated nature of both designs reinforced their knowledge base increased their self-efficacy, reduced their anxiety levels, and helped them to prepare for placement.

CONCLUSION

The results inferred that both designs had a positive impact on the participants. Overall, participants reported that it helped them prepare for placements. Based on the findings, wherever possible, repeated immersive simulation-based education designs should be used and not a standalone immersive simulation-based education scenarios. If resources allow, this could be either a repeated scenario, or if there are resource constraints to use, over the same time, immersive simulation with team deliberate practice, or a similar model. Thus, giving a potential return on investment, one that supports simulation educators making those sensitive decisions regarding the inclusion of immersive simulation with team deliberate practice in their curriculum. Further research is needed into this area to ascertain the design features that maximise this impact and support a move away from standalone scenarios to an approach that uses repetitive immersive simulation.

Building Team Leader Competency in Senior Nursing Students

ABSTRACT

Nursing students have limited opportunities to build team leader skills during their clinical experiences, leaving a practice gap. Senior nursing students participated in a competency-based education unit and a simulation-based learning experience incorporating team leader skills. Presurvey and postsurvey data were collected using the Self-Confidence and Comfort with Utilizing a Standardized Handoff Communication Tool. Participants who completed both surveys were analyzed ( n = 85). Statistically significant improvements were found on 14 of the 19 survey items. Results suggest the effectiveness of using competency-based education and simulation-based learning experience to help nursing students build confidence and comfort in team leader skills.

Lasater clinical judgment rubric reliability for scoring clinical judgment after observing asynchronous simulation and feasibility/usability with learners

BACKGROUND

There is strong evidence supporting using the Lasater Clinical Judgment Rubric (LCJR) for scoring learners' clinical judgment during in-person simulation performance and clinical experience reflections. However, a gap exists for using LCJR to evaluate clinical judgment after observing asynchronous simulation.

OBJECTIVE

We aimed to determine the reliability, feasibility, and usability of LCJR for scoring learners' written reflections after observing expert-modeled asynchronous simulation videos.

DESIGN/SETTING/PARTICIPANTS

We used a one-group, descriptive design and sampled pre-licensure, junior-level bachelor's learners from the Southwestern United States.

METHODS

Participants observed eight expert-modeled asynchronous simulation videos over one semester and provided written responses to clinical judgment prompts. We scored clinical judgment using LCJR. We studied reliability by measuring internal consistency of 11 clinical judgment prompts and interrater reliability with two raters. This study also investigated feasibility and usability of the asynchronous simulation learning activity using descriptive statistics. Feasibility included time learners spent completing written responses and time raters spent evaluating written responses. Learners reported usability perceptions using an instructor-developed survey.

RESULTS

Sixty-three learners completed 504 written responses to clinical judgment prompts. Cohen's kappa ranged from 0.34 to 0.86 with a cumulative κ = 0.58. Gwet's AC ranged from 0.48 to 0.90, with a cumulative AC = 0.74. Cronbach's alpha was from 0.51 to 0.72. Learners spent on average 28.32 ± 12.99 min per expert-modeling video observation. Raters spent on average 4.85 ± 1.34 min evaluating written responses for each participant. Learners reported the asynchronous learning activity was usable.

CONCLUSIONS

Nurse educators can reliably use LCJR for scoring learners' clinical judgment after observing asynchronous expert-modeled simulation. Logistically, learners complete the reflective learning activity and faculty use LCJR to measure clinical judgment in feasible time. Further, participants perceived the asynchronous learning activity usable. Nurse educators should utilize this learning activity for evaluating and tracking observer clinical judgment development.

Development and psychometric testing of the Diagnostic Competency During Simulation-based (DCDS) learning tool

BACKGROUND

Despite diagnostic errors impacting an estimated 12 million people yearly in the United States, educational strategies that foster diagnostic performance among nurse practitioner (NP) students remain elusive. One possible solution is to focus explicitly on competencies fundamental for diagnostic excellence. Currently, no educational tools were found that comprehensively address individual diagnostic reasoning competencies during simulated-based learning experiences.

PURPOSE

Our research team developed and explored psychometric properties of the "Diagnostic Competency During Simulation-based (DCDS) Learning Tool."

METHOD

Items and domains were developed based on existing frameworks. Content validity was determined by a convenience sample of eight experts. Inter-rater reliability was determined by four faculty rating eight simulation scenarios.

RESULTS

Final individual competency domain scale content validity index (CVI) scores ranged between 0.9175 and 1.0; total scale CVI score was 0.98. The intra-class correlation coefficient (ICC) for the tool was 0.548 (p < 0.0001, 95 % confidence interval CI [0.482-0.612]).

CONCLUSIONS

Results suggest that the DCDS Learning Tool is relevant to diagnostic reasoning competencies and may be implemented with moderate reliability across varied simulation scenarios and performance levels. The DCDS tool expands the landscape of diagnostic reasoning assessment by providing NP educators with granular, actionable, competency-specific assessment measures to foster improvement.

Effect of High-Fidelity Simulation on Self-Satisfaction and Self-Confidence Among Nursing Students

Introduction

Newly graduating nurses should be able to enter clinical practice as safe, accurate, competent, and compassionate professionals in health care settings that necessitate them to be learners who can easily get familiar with a broad diversity of clinical practice settings.

Purposes

The purpose of this study was to evaluate the effect of high-fidelity simulation experiences on self-satisfaction and self-confidence among the pediatric nursing students.

Methods

A quasi-experimental design, post-test, was carried out through the recruitment of a convenience sample of 150 nursing students from Arab American University. The students were randomly assigned to the study (n  =  75) or control (n  =  75) groups. Data were collected using learner satisfaction and self-confidence scales.

Results

The results of the post-test revealed that there was a significant difference between both groups in self-satisfaction (t (148)  =  11.14, P < .001) and self-confidence (t (148)  =  13.04, P < .001). The mean score of self-satisfaction for the study group (22.45  ±  2.27) was higher than control group (16.80  ±  3.76). Also, the study group' mean self-confidence score (35.72  ±  3.49) was higher than the control group (27.28  ±  4.39).

Conclusion

High-fidelity simulation was confirmed to be an helpful tool to keep an effective and safe learning method for pediatric nursing students, thus increasing their satisfaction and confidence.

Psychometric properties of clinical competency assessment instruments for psychiatric nurses: A systematic review of literature

BACKGROUND

Facilitating the healing process of patients with psychiatric disorders depends on high-quality mental health care and expert psychiatric nurses. A valid tool is required to objectively evaluate the quality of performance and competency of psychiatric nurses. This systematic review aimed to investigate and critique the psychometric properties of some psychiatric nursing clinical competency assessment instruments with Consensus-based standards for the selection of health status measurement instruments (COSMIN) checklist and according to Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA).

MATERIALS AND METHODS

To retrieve published articles using Persian and English keywords "psychiatric nursing," "competence," "competency," "tool," "checklist," "scale," "questionnaire," "psychiatric mental health nurse," were searched in databases without time limitation. Then, psychometric properties of selected instruments were evaluated using the COSMIN checklist and reported according to the PRISMA statement.

RESULTS

Most of these tools did not entirely and desirably report psychometric properties. It is not designed as a special tool for postgraduate psychiatric nursing.

CONCLUSION

The methodology of existing instruments does not meet the COSMIN checklist criteria; therefore it needs to develop. To assess the competency of psychiatric nursing postgraduate, a tool tailored to the cultural and social context and with acceptable psychometric properties is necessary.

Remote Virtual Simulation With Prelicensure Nursing Students During Rapid Course Transition

The advent of COVID-19 required educational programs to rapidly transition courses to the remote environment. A postpartum hemorrhage simulation used within a traditional prelicensure bachelor of science in nursing program was rapidly transitioned to the remote virtual format to meet required social distancing guidelines. This quasi-experimental study examined student knowledge before and after participation in a remote virtual simulation (RVS) with a postanalysis of student confidence and satisfaction. RVS can increase student knowledge and provide adequate student satisfaction. However, hands-on learning appears to result in higher student confidence and satisfaction compared to RVS.

The impact of a 12-hour educational program on nurses' knowledge and attitudes regarding pain management: a quasi-experimental study

BACKGROUND

Proper pain assessment is fundamental to effective pain management. Training nursing staff is critical for improving pain assessment competence and patient clinical outcomes. However, there is a dearth of research examining interventions that can enhance nurses' knowledge and attitudes toward pain management, especially in Saudi Arabia. Thus, this study aimed to evaluate the effectiveness of a structured education program on nurses' knowledge and attitudes towards pain management.

METHODS

A quasi-experimental design was used. The study sample included 124 registered nurses working in intensive care or inpatient units in Saudi Arabia. Data were collected between March and September 2021 using a knowledge and attitudes survey regarding pain, satisfaction with and self-confidence in learning, and the learning self-efficacy scale for clinical skills.

RESULTS

Nurses showed moderate levels of knowledge and attitudes regarding pain before (M = 20.3, SD = 4.80) pain management education, which were significantly higher after the intervention (M = 22.2, SD = 5.09, t = 2.87, p < .01). Before the intervention, nurses with a baccalaureate degree had more knowledge and better attitudes regarding pain management than diploma nurses (t = 3.06, p < .01). However, there was no significant difference between the two groups after the intervention (p > .05), indicating that the education was effective in enhancing nurses' knowledge and attitudes, regardless of nursing education level. Nurses in this study had high mean scores for self-confidence in learning (M = 35.6, SD = 4.68, range = 18-40), self-learning efficacy (M = 52.9, SD = 7.70, range = 25-60), and satisfaction with learning (M = 22.2, SD = 3.24, range: 10-25).

CONCLUSION

Regular pain education programs can improve nurses' knowledge and attitudes. Increasing the breadth and depth of educational courses, alongside appropriate training, competency-based assessment, and pain education programs, is also recommended. Future research should consider the subjectivity and individualized nature of nursing by including patient satisfaction surveys to measure the improvement in nurses' knowledge and attitudes from the patient perspective.

Assessing the Financial Sustainability of High-Fidelity and Virtual Reality Simulation for Nursing Education: A Retrospective Case Analysis

To stimulate classroom discussion and collaboration amid the COVID-19 pandemic, increasingly creative pedological methods for nursing education are necessary. Traditionally, high-fidelity simulation has been the standard for nursing education, but the use of virtual reality simulation is increasing. One of the major evaluative measures of simulation clinical training is the cost associated with each modality. In this retrospective case analysis, budget impact analysis methods were employed to compare high-fidelity simulation with virtual reality simulation. The components of each simulation pedagogy were compared in categorized cost buckets. Overall, virtual reality simulation education was determined to require 22% less time than high-fidelity simulation education. The cost associated with the virtual reality simulation was found to be 40% less expensive than the high-fidelity simulation. Our results demonstrate that virtual reality simulation is a financially advantageous, resource conscious pedagogical option for nursing education.

The development of a self-evaluation scale for simulation laboratory practices

BACKGROUND

Simulation is an important learning-teaching tool for integrating theory and practice in nursing education.

OBJECTIVE

The aim of this study was to develop and to conduct the validity and reliability of a self-evaluation scale for simulation laboratory practices (SES-SLP) with undergraduate nurses.

METHODS

The study sample consisted of 220 undergraduate nursing students. Data were collected using a Descriptive Characteristic Form (DCF) and the 23-item Self-Evaluation Scale for Simulation Laboratory Practices (SES-SLP).

RESULTS

Cronbach's alpha value of the scale was 0.94. The scale comprised of two subscales: the developing factor (19 items) and the challenging factor (4 items). All items showed a statistically significant correlation (p < 0.05).

CONCLUSIONS

The SES-SLP is a valid and reliable scale that can be used to evaluate students' learning experience for simulation laboratory practices.

Reliability and validity study of the Spanish adaptation of the "Student Satisfaction and Self-Confidence in Learning Scale" (SCLS)

The European Higher Education Area (EHEA) recommends the use of new educational methodologies and the evaluation of student satisfaction. Different instruments have been developed in Spain to evaluate different aspects such as clinical decisions and teamwork, however no instruments have been found that specifically evaluate student self-confidence and satisfaction during clinical simulation. The aim was to translate the Student Satisfaction and Self-Confidence in Learning Scale (SCLS) questionnaire into Spanish and analyse its reliability and validity and understand the level of satisfaction and self-confidence of nursing students with respect to learning in clinical simulations. The study was carried out in two phases: (1) adaptation of the questionnaire into Spanish. (2) Cross-sectional study in a sample of 489 nursing students. The reliability and exploratory and confirmatory factorial analyses were performed. To analyse the relationship of the scale scores with the socio-demographic variables, the Fisher Student T-test or the ANOVA was used. The scale demonstrated high internal consistency reliability for the total scale and each of its dimensions. Cronbach’s alpha was 0.88 (0.83 to 0.81) for each of the dimensions. The exploratory and confirmatory factor analysis showed that both the one-dimensional and two-dimensional models were acceptable. The results showed average scores above 4 for both dimensions. The SCLS-Spanish translation demonstrated evidence of its validity and reliability for use to understand the level of satisfaction and self-confidence of nursing students in clinical simulation. Clinical simulations help students to increase their levels of confidence and satisfaction, enabling them to face real scenarios in clinical practice.

Structured simulations improves students' knowledge acquisition and perceptions of teaching effectiveness: A quasi-experimental study

AIMS

To determine the effect of adding structured simulation to a traditional classroom lecture format on students' knowledge acquisition and their evaluation of faculty teaching effectiveness, compared to traditional classroom lectures alone.

BACKGROUND

There is a critical lack of high-quality clinical placements/experiences in nursing education, particularly in clinical specialty populations, such as paediatrics. Simulation has been shown to help students practice in a safe environment. However, less is known about the outcomes of embedding simulation in didactic class sessions or classroom lectures. Additionally, data on the impact of simulation as a teaching pedagogy on faculty teaching effectiveness is limited.

DESIGN

Post-test, nonequivalent control group quasi-experimental study.

METHODS

Students in two prelicensure nursing programs participated in a 10-week term paediatric nursing course taught by the same faculty member on two different campuses. The students at one campus, designated as the control group (n = 43), received four hours of traditional, lecture-only, instructions. The group at a second campus, the intervention group (n = 44), received the same lectures with added structured simulations. Students' knowledge acquisition and their evaluation of faculty teaching effectiveness were measured in both groups using the same standardised assessment measures. The CONSORT checklist was followed.

RESULTS

Students' knowledge acquisition scores and their evaluation scores of faculty teaching effectiveness were significantly higher in the intervention (simulation) group compared to the control group. A significantly higher number of students in the simulation group scored above the national average Nursing Care of Children score as compared to students in the control group.

CONCLUSIONS

Integrating simulation with didactic instruction enhanced students' knowledge acquisition and improved their opinion of faculty teaching effectiveness.

RELEVANCE TO CLINICAL PRACTICE

Augmenting lecture with simulation may provide students with learning experiences that they may not have during clinical rotation due to a lack of paediatric clinical placement sites and differences between sites.

Collaborative clinical simulation in cardiologic emergency scenarios for medical students. An exploratory study on model applicability and assessment instruments

Aims: This paper evaluates the feasibility of piloting the collaborative clinical simulation (CCS) model and its assessment instruments applicability for measuring interpersonal, collaborative, and clinical competencies in cardiologic emergency scenarios for medical students. The CCS model is a structured learning model for the acquisition and assessment of clinical competencies through small groups working collaboratively to design and perform in simulated environments supported by technology. Methods: Fifty-five students were allocated in five sessions (one weekly session) conducted with the CCS model within the course Cardiovascular Diseases. The applied practice aimed at the diagnosis and treatment of tachyarrhythmias in a simulated emergency department. In addition to the theoretical classes four weeks before the simulation sessions, students were sent a study guide that summarized the Guide to the European Society of Cardiology. For each simulation session, one clinical simulation instructor, one cardiologist teacher, and the principal investigator participated. Students were divided into three groups (3-5 students) for each-session. They designed, performed, role-played, and debriefed three different diagnoses. Three instruments to assess each group's performance were applied: peer assessment used by groups, performance assessment, created and applied by the cardiologist teacher, and individual satisfaction questionnaire for students. Results: The applicability of the CCS model was satisfactory for both students and teachers. The assessment instruments' internal reliability was good, as was internal consistency with a Cronbach Alpha of 0.7, 0.4, and 0.8 for each section (Interpersonal, Clinical, and Collaborative competencies, respectively). The performance group's evaluation was 0.8 for the two competencies assessed (Tachyarrhythmia and Electrical Cardioversion) and 0.8 for the satisfaction questionnaire's reliability. Conclusions: The CCS model for teaching emergency tachyarrhythmias to medical students was applicable and well accepted. The internal reliability of the assessment instruments was considered satisfactory by measuring satisfaction and performance in the exploratory study.

Development of a novel global rating scale for objective structured assessment of technical skills in an emergency medical simulation training

BACKGROUND

Medical simulation trainings lead to an improvement in patient care by increasing technical and non-technical skills, procedural confidence and medical knowledge. For structured simulation-based trainings, objective assessment tools are needed to evaluate the performance during simulation and the learning progress. In surgical education, objective structured assessment of technical skills (OSATS) are widely used and validated. However, in emergency medicine and anesthesia there is a lack of validated assessment tools for technical skills. Thus, the aim of the present study was to develop and validate a novel Global Rating Scale (GRS) for emergency medical simulation trainings.

METHODS

Following the development of the GRS, 12 teams of different experience in emergency medicine (4th year medical students, paramedics, emergency physicians) were involved in a pre-hospital emergency medicine simulation scenario and assessed by four independent raters. Subsequently, interrater reliability and construct validity of the GRS were analyzed. Moreover, the results of the GRS were cross-checked with a task specific check list. Data are presented as median (minimum; maximum).

RESULTS

The GRS consists of ten items each scored on a 5-point Likert scale yielding a maximum of 50 points. The median score achieved by novice teams was 22.75 points (17;30), while experts scored 39.00 points (32;47). The GRS overall scores significantly discriminated between student-guided teams and expert teams of emergency physicians (p = 0.005). Interrater reliability for the GRS was high with a Kendall's coefficient of concordance W ranging from 0.64 to 0.90 in 9 of 10 items and 0.88 in the overall score.

CONCLUSION

The GRS represents a promising novel tool to objectively assess technical skills in simulation training with high construct validity and interrater reliability in this pilot study.

Simulation design and students' satisfaction with home-based simulation learning in oral health therapy

OBJECTIVES

The COVID-19 pandemic presents a challenging situation for dental education, with school closures worldwide. While practical sessions are important for the acquisition of clinical skills, few studies have examined the role of home-based simulation learning (HBSL). This study aims to identify key aspects of simulation design and factors associated with students' satisfaction with HBSL.

METHODS

During a "circuit-breaker" period in Singapore, the Nanyang Polytechnic School of Health & Social Sciences (Oral Health Therapy) distributed portable manikins and instruments to students and staff for use at home. Synchronous teaching sessions with real-time feedback were carried out over the Zoom teleconferencing platform. A cross-sectional survey of students was conducted after 3 weeks of the HBSL program. Multiple linear regression analyses were carried out to assess the association between satisfaction with domains of simulation design and overall satisfaction with HBSL.

RESULTS

The top three factors of importance to students in HBSL were feedback and guided reflection, support, and objectives and information. Final-year students expressed higher levels of overall satisfaction and self-confidence following HBSL than first-year students. Satisfaction with the objectives (β = 0.42, 95% CI 0.26-0.57), support (β = 0.27, 95% CI 0.11-0.42), and problem-solving (β = 0.28, 95% CI 0.10-0.46) domains in the Simulation Design Scale were positively associated with overall student satisfaction and self-confidence in learning.

CONCLUSION

The findings can aid educators in the design of future HBSL programs, with a focus on areas of simulation design that are of importance to students.

Speaking Up Against Hierarchy: A Simulation Geared Towards Nursing Students

Background

As simulation science continues to advance, the focus previously put on scenario creation and debriefing must now be applied to other components of the learning experience. There is a need to examine the effectiveness of pre-simulation activities and how they relate to the overall simulation experience and learning outcomes. However, few randomized controlled trials have been conducted comparing different approaches in the pre-simulation preparatory phase and the impact on learning outcomes.

Methods

A randomized controlled trial was conducted with undergraduate nursing students (n=83) who were randomized to a traditional paper case study (control group) or an interactive pre-simulation activity (intervention group). The use of the two-challenge rule and Satisfaction and Self Confidence in Learning (SSL) was evaluated.

Results

The proportion of students who utilized the two-challenge rule in the intervention group was significantly higher than the control group. Results from the two independent-samples Wilcoxon-Mann-Whitney test showed a significant difference in the median of the total score of the SSL W=2.5, p <0.001, satisfaction W=6.0, p <0.001, and self-confidence W=68.0, p <0.001 in learning between third-year nursing students in the control and intervention groups.

Conclusion

Our results showed significant differences in the use of the two-challenge rule by students who completed an interactive pre-simulation activity (intervention group) compared to those who completed the paper case study (control group). Additionally, students in the intervention group were more self-confident and satisfied with the entire simulation intervention than the control group. From a pedagogical perspective, this study also emphasizes the need to ground simulations in theory. Moreover, there is value in using progressive frameworks, i.e., revised Medical Research Council (2014) in simulation design and research to ensure high quality. More studies are required to examine the right dosage and type of pre-simulation activity and impact on learning outcomes.

Evaluation of Simulation Outcomes

Simulation has been used in nursing education and training since Florence Nightingale's era. Over the past 20 years, simulation learning experiences (SLEs) have been used with increasing frequently to educate healthcare professionals, develop and increase the expertise of practicing professionals, and gain competency in key interprofessional skills. This chapter provides a brief overview of simulation evaluation history, beginning in the late 1990s, and the initial focus on learner self-report data. Using Kirkpatrick's Levels of Evaluation as an organizing model, four types of SLE evaluation are reviewed as well as suggestions for future research.

Interactive Clinical Avatar Use in Pharmacist Preregistration Training: Design and Review

BACKGROUND

Virtual patients are interactive computer-based simulations that are being increasingly used in modern health care education. They have been identified as tools that can provide experiential learning and assessment in a standardized and safe environment. However, the study of high-fidelity virtual patients such as interactive clinical avatars within pharmacy is limited.

OBJECTIVE

The aim of this paper is to describe the design and review of three interactive clinical avatar simulations as part of pharmacist preregistration training.

METHODS

A multistep design approach was taken to create interactive clinical avatar simulations on the topics of emergency hormonal contraception (EHC), calculation of renal function, and childhood illnesses. All case studies were reviewed by registered pharmacists to establish content and face validity. The EHC case study and data collection questionnaire were also reviewed by a purposive sample of preregistration trainees and newly qualified pharmacists. The questionnaire used Likert ranking statements and open-ended questions to obtain users' feedback on the design, usability, and usefulness of the interactive clinical avatars as learning tools. Descriptive statistics and content analysis were undertaken on the data.

RESULTS

Ten preregistration trainees and newly qualified pharmacists reviewed the EHC interactive clinical avatars and data collection questionnaire. The data collection questionnaire was associated with a Cronbach alpha=.95, demonstrating good reliability. All three interactive clinical avatar simulations were reported as usable and appropriately designed for preregistration training. Users perceived they were developing skills and knowledge from the simulations. The high-fidelity nature of the avatars and relevance of the simulations to real-life practice were reported as aspects that encouraged the application of theory to practice. Improvements were suggested to ensure the simulations were more user-friendly.

CONCLUSIONS

The design and creation of the interactive clinical avatar simulations was successful. The multistep review process ensured validity and reliability of the simulations and data collection questionnaire. The in-depth explanation of the design process and provision of a questionnaire may help widen the use and evaluation of interactive clinical avatars or other simulation tools in pharmacy education. The interactive clinical avatars were reported as novel learning tools that promoted experiential learning and allowed users to feel like they were engaging in real-life scenarios, thus developing transferable knowledge and skills. This may be potentially beneficial for many health care training courses as a way to provide standardized experiences promoting active learning and reflection.

Psychometric testing of the Norwegian version of the Simulation Design Scale, the Educational Practices Questionnaire and the Student Satisfaction and Self-Confidence in Learning Scale in nursing education

Introduction

Simulation-based learning is a well-established technique in nursing education. However, there is a need for reliable and validated evaluation tools across both national boundaries and cultural conditions. Such evaluation tools may contribute in identifying areas for improvement in simulation-based learning from the nursing students' perspective.

Objectives

The aim of this study was to test three widely used American questionnaires - the Simulation Design Scale, the Educational Practices Questionnaire, and the Student Satisfaction and Self-Confidence in Learning Scale, for psychometric properties among Norwegian undergraduate nursing students.

Methods

A descriptive cross-sectional study was conducted at a university simulation center in southern part of Norway. A total of 105 undergraduate nursing students participated, giving a response rate of 77%. An exploratory factor analysis was used to examine construct validity. Cronbach's alpha was applied in order to establish the questionnaires' internal consistency.

Results

The exploratory factor analyses displayed the same number of extracted factors as the number of subscales in each of the original American questionnaires. However, the item-factor structure differed from the original item-subscales. The Cronbach's alpha was > 0.7 for all three questionnaires, indicating acceptable internal consistency.

Conclusion

Psychometric testing of the Norwegian versions of the three questionnaires, the Simulation Design Scale, the Educational Practices Questionnaire, and the Student Satisfaction and Self-Confidence in Learning Scale, could be used as valid instruments for nursing students to evaluate important aspects of simulation-based learning. This also makes it easier to compare evaluation results of SBL across languages and cultural boundaries. However, to confirm the construct validity of the factors extracted in this study, further multi-site studies are needed to perform a confirmatory factor analysis in a new, large sample.

Development and Evaluation of an Algorithm-Corresponding Instrument for Nursing Simulation

BACKGROUND

This study aimed to develop and validate an assessment instrument for students engaging with an algorithm-based simulation scenario addressing emergency measures for high-risk newborns with apnea in a neonatal intensive care unit.

METHOD

The study was conducted in two phases of development and evaluation of the algorithm-corresponding instrument. One hundred sixty-nine senior nursing students from two universities in South Korea were evaluated using the developed instrument.

RESULTS

The developed and validated instrument consisted of three dimensions (assessment points, nursing skills, and communication) measured through 13 items. The exploratory factor analysis revealed three factors of the instrument, and the confirmatory factor analysis demonstrated a better model fit for a three-factor instrument model than for other models.

CONCLUSION

The developed algorithm-corresponding assessment instrument is suitable for assessing the clinical decision-making ability of nursing students in a simulation scenario. [J Nurs Educ. 2020;59(11):617-626.].

Standardizing nurse practitioner student evaluation during high-stakes clinical examinations

Nurse practitioners (NPs) require clinical competence in advanced health assessment skills to differentiate between normal, variations of normal, and abnormal patient findings. High-stakes clinical examinations, using live hands-on simulation scenarios and standardized patients (SPs) or other human role players, are accepted teaching and evaluation tools for NP students. Providing objective, valid, and reliable feedback to students during high-stakes clinical examinations is of considerable value for ongoing skill development. The study examined opportunities to improve the quality of student evaluation in simulation testing modes. A purposive sample of 17 video recordings of health students' comprehensive examination of an SP or physical examination teaching associate (PETA) from a nursing graduate level health assessment course was evaluated. Using a standardized rubric, students were scored live and after a comprehensive examination of a SP/PETA and via a secure web-based video platform by the faculty and an independent reviewer. Evaluator group examination score comparisons revealed that distributions of examination scores within evaluator groups were not similar. Median examination scores were significantly different between groups; faculty median examination scores significantly higher than SPs/PETAs. Efficiency of student evaluation may be increased by improving reviewer training, reducing checklist length, and adopting electronic scoring. Development of an exemplary teaching video providing explanation and detail for expected student skill performance will allow reviewers to practice and improve competence in reliable scoring, reduce time and effort of scorers, and increase accuracy of scoring.

Teaching Crisis Resource Management Skills to Nurses Using Simulation

BACKGROUND

Education programs teaching crisis resource management (CRM) skills (problem solving, situational awareness, resource utilization, communication, and leadership) have been shown to positively affect learner competence in handling crisis events. As part of an education program, a high-fidelity simulation program was used as a learning intervention to teach these skills to practicing nurses.

METHOD

In this repeated-measures observational study, 11 RNs were evaluated at four time points, measuring the effect of an education program on observed performance of CRM skills. Performance was measured using the Ottawa Global Rating Scale and a checklist tool.

RESULTS

Statistically significant changes in mean scores occurred between times one and two, and nonstatistically significant improvement occurred in means overall.

CONCLUSION

This study adds evidence of the effectiveness of high-fidelity simulation education and highlights the need for further research. [J Contin Educ Nurs. 2020;51(6):257-266.].

Using Simulation to Evaluate Clinical Performance and Reasoning in Adult-Geriatric Acute Care Nurse Practitioner Students

BACKGROUND

The use of simulation-based learning experiences (SBLE) with nurse practitioner (NP) students is largely unexplored. The purpose of this pilot project was to develop, implement, and evaluate a simulated patient (SP)-SBLE to improve confidence and evaluate clinical performance and reasoning in Adult-Geriatric Acute Care NP (AG-ACNP) students.

METHOD

A mixed-methods design was used to evaluate an SP-SBLE on AG-ACNP student clinical performance and reasoning.

RESULTS

Self-confidence scores and themes from reflections indicated that students perceived this as a valuable experience that enhanced learning and confidence.

CONCLUSION

Results supported the use of SP-SBLEs to improve confidence in clinical performance and reasoning with AG-ACNP students. [J Nurs Educ. 2019;58(10):599-603.].

Checklist design and implementation: critical considerations to improve patient safety for low-frequency, high-risk patient events

Purpose: This pilot project describes the development and implementation of two specialised aviation-style checklist designs for a low-frequency high-risk patient population in a cardiac intensive care unit. The effect of the checklist design as well as the implementation strategies on patient outcomes and adherence to best practice guidelines were also explored. The long-term objective was to improve adherence to accepted processes of care by establishing the checklists as standard practice thereby improving patient safety and outcomes.

Methods: During this project, 10specialised crisis checklists using two specific aviation-style designs were developed. A quasiexperimental prospective pre-post repeated measure design including surveys along with repetitive simulations were used to evaluate self-confidence and self-efficacy over time as well as the perceived utility, ease of use, fit into workflow and benefits of the checklists use to patients. Performance, patient outcomes and manikin outcomes were also used to evaluate the effectiveness of the crisis checklists on provider behaviours and patient outcomes.

Results: Overall self-confidence and self-confidence related to skills and knowledge while not significant demonstrated clinically relevant improvements that were sustained over time. Perceptions of the checklists were positive with consistent utilisation sustained over time. More importantly, use of the checklists demonstrated a reduction in errors both in the simulated and clinical setting.

Conclusion: Recommendations from this study consist of key considerations for development and implementation of checklists including: utilisation of stakeholders in the development phase; implementation in real and simulated environments; and ongoing reinforcement and training to sustain use.

Validation of the Simulation Effectiveness Tool in Nursing Education

BACKGROUND

The Simulation Effectiveness Tool (SET) frequently is used to assess perceived learning and confidence in simulation. However, few studies have reported the validity of the tool. This study assessed the reliability and validity of the SET.

METHOD

This retrospective analysis evaluated the tool using 568 cases conducted at three nursing schools.

RESULTS

A two-factor model showed reasonable fit indices. The fit statistics for the two-factor structure were: χ², 152.98 (df = 53, p < .001); comparative fit index, 0.94; root mean square error of approximation, 0.05 (range, 0.04 to 0.06); and standardized root mean square residual, 0.04. In addition, weak convergence was identified between the confidence in the SET and responding in the Lasater rubric.

CONCLUSION

The psychometric properties of the study indicate the SET has demonstrated acceptable evidence of validity and reliability to measure simulation effectiveness in Korean nursing students. The use of this instrument for brief simulation education is recommended. [J Nurs Educ. 2020;59(4):186-193.].

A novel measurement instrument for pressure-injury risk assessment competence: Theoretical procedures, simulation, and psychometric quality

This study developed a measuring instrument of pressure-injury risk assessment competence for nursing education adopting theoretical procedures based on competence structuration to support the psychometric quality. The objective of this study is testing the psychometric properties of the instrument using simulation strategy and to determine the instrument's standardisation and normalisation. A methodological study was designed. The instrument was developed by a content-validated theoretical construct administered to 155 undergraduate nursing students from universities in the northeast and south of Brazil using simulation. The instrument was applied over three phases: before the lecture on pressure-injury risk assessment competence, after scenario of simulation experience, and after debriefing experience. The instrument comprised 32 items. The factorial analysis found that three domains explained 64.6% and 62.18% of the total variance in post-scenario and debriefing cases. The perfect and very high discrimination index of the instrument indicated minimal differences in measured latent trait levels. It also found that reliability was excellent (0.973 and 0.967). Moreover, an equation applicable to instruments using study's theoretical procedure was proposed. The instrument was found to be a valid, accurate, and reliable educational tool for pressure-injury risk assessment competence.

Reliability and validity of the simulation learning effectiveness inventory

BACKGROUND

Instruments developed to measure simulation learning outcomes need evidence of their reliability and validity for rigorous research. The purpose of this paper is to report psychometric properties of the English version of the Simulation Learning Effectiveness Inventory.

METHODS

Psychometric properties of the English version of the Simulation Learning Effectiveness Inventory, which included internal consistency reliability and construct validity with factor analysis, were examined in a sample of 132 undergraduate nursing students.

RESULTS

Cronbach's alpha coefficients were >0.70 for all subscales. There was evidence of convergent, discriminant, and known-group validity. The factor analysis resulted in some items being associated with different subscales than in the original Chinese version.

CONCLUSIONS

The English version of the Simulation Learning Effectiveness Inventory has evidence of reliability and validity. Additional psychometric studies may result in changes in some of the subscales.

Implementation of Mock Acute Care Advance Pharmacy Practice Experience Simulations and an Assessment Rubric

Objective. To implement a mock acute care advanced pharmacy practice experience series into the didactic training of second-year pharmacy students and validate an accompanying assessment rubric. Methods. Three 90-minute acute care patient simulation laboratory sessions were developed with input from clinical specialists, preceptors, students, and faculty members. An accompanying student evaluation rubric was also developed. The assessment rubric was validated using pairs of preceptor raters to determine inter-rater reliability, along with predictive validity on advanced pharmacy practice experience (APPE) acute care scores. A student survey was also conducted. Results. The mock acute care APPEs were successfully implemented into the didactic curriculum. The assessment rubric had good inter-rater reliability and good predictive validity with acute care APPEs. Survey results indicated that students found the mock acute care APPE simulation laboratories useful. Conclusion. Other schools seeking to enhance their students' preparedness for and performance in acute care APPEs should consider implementing acute care APPE simulations in the didactic curriculum.

Ready, Set, Go: How Patients and Caregivers Are Prepared for Self-Management of an Implantable Ventricular Assist Device

Since the first use of an implantable ventricular assist device (VAD) nearly 3 decades ago, many VAD Centers in the United States expect patients and caregivers to manage the device and related care following hospital discharge. Despite this customary practice and the rapid advancements in VAD technology, no research data about the patient and caregiver preparations exist for self-management of VAD. This study explored the process of preparing patients and caregivers for VAD care before hospital discharge along with their perceived experiences of the discharge process. Using an exploratory research design, we collected the data with self-administered demographics and VAD Hospital Discharge Survey questionnaires. We analyzed the data provided by 102 patients and 116 caregivers from different regions in the United States. Patients' mean age was 51.4 ± 13.8 years; they were predominantly White (78%); male (66%); and duration of device between 2 and 74 months. Caregivers were predominantly White (80%); female (81%); and had a mean of age 48.9 ± 12.7 years. Participants reported that their VAD care preparations, a crucial component of hospital discharge, were extensive and intensive processes comprising education, training, competency validation, and provision of resources for VAD self-management in home settings. Participants were satisfied with the process and felt prepared for discharge. Further research is needed to rectify the study limitations, advance the science of VAD self-management, and develop evidenced-based VAD self-management guidelines that are aimed at optimizing longevity of the VAD, patients' health, and quality of life.

Beyond the clinical team: evaluating the human factors-oriented training of non-clinical professionals working in healthcare contexts

Background

As clinical simulation has evolved, it is increasingly used to educate staff who work in healthcare contexts (e.g. hospital administrators) or frequently encounter clinical populations as part of their work (e.g. police officers) but are not healthcare professionals. This is in recognition of the important role such individuals play in the patients' experience of healthcare, frequently being a patients' first point of contact with health services. The aim of the training is to improve the ability of the team to communicate and co-ordinate their actions, but there is no validated instrument to evaluate the human factors learning of non-clinical staff. Our aim was to develop, pilot and evaluate an adapted version of the Human Factors Skills for Healthcare Instrument, for non-clinical professionals.

Method

The 18-item instrument was developed reflecting the human factors skills of situation awareness, decision making, communication, teamwork, leadership, care and compassion and stress and fatigue management. The instrument was piloted pre- and post-training with non-healthcare professionals (n = 188) attending mental health simulation training within an 11-month period (June 2017-April 2018). Trainees were hospital/primary care administrators (n = 53, 28%), police officers (n = 112, 59%), probation officers (n = 13, 7%) and social workers (n = 10, 5%). Most participants were female (n = 110, 59%) and from White ethnic backgrounds (n = 144, 77%).

Results

Six items were removed, five were not sufficiently sensitive to change (d < .3) and one showed poor reliability. The remaining 12 items revealed a Cronbach's alpha of .93. An exploratory factor analysis revealed a one-factor solution, which explained 58.3% of the variance. The final 12-item instrument was sensitive to change post-training (p < .0001) with large effect sizes (d > .7). Cluster analysis revealed that participants with lower pre-training scores showed the greatest improvement.

Discussion

The Human Factors Skills for Healthcare Instrument-Auxiliary version (HuFSHI-A) provides a reliable and valid instrument for the evaluation of human factors skills learning following training of non-clinical populations working in healthcare contexts. Although this instrument has been developed and evaluated with training courses specifically focusing on mental health topics, HuFSHI-A is applicable for any training where teamwork and co-ordination between clinical and non-clinical professionals is considered.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Modeling Framework Used to Analyze and Describe Junctional Tourniquet Skills

Medical educators have acknowledged the importance of simulation training in developing procedural skills. While simulation training in other disciplines has benefitted from evaluations of users' skill acquisition, the majority of medical training simulators continue to be developed from overly simplified descriptions of procedures, such as techniques prescribed by existing instructional material. Our objective was to use a modeling framework to characterize the skill of various users in applying junctional tourniquets in order to design an effective training simulator. We recorded 46 medical first responders performing training exercises applying a junctional tourniquet and used coded video and sensor data to identify the hierarchy of actions they performed in the process. The model provides several insights into trainee performance, such as the way in which advanced users perform more tasks in parallel, or areas where advanced users employ situational awareness to identify ways they can deviate from recommended protocol to improve outcomes. The model successfully identifies variations in tourniquet application technique that correlates with improvement on clinically relevant metrics including application speed, pressure applied, and tourniquet placement stability. This methodology can improve medical training simulations by indicating changes during the course of learning a new task, such as helpful deviations from instructional protocol.

Effects of high-fidelity simulation based on life-threatening clinical condition scenarios on learning outcomes of undergraduate and postgraduate nursing students: a systematic review and meta-analysis

OBJECTIVE

The purpose was to analyse the effectiveness of high-fidelity patient simulation (HFPS) based on life-threatening clinical condition scenarios on undergraduate and postgraduate nursing students' learning outcomes.

DESIGN

A systematic review and meta-analysis were conducted based on the Cochrane Handbook for Systematic Reviews of Interventions and its reporting was checked against the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist.

DATA SOURCES

PubMed, Scopus, CINAHL with Full Text, Wiley Online Library and Web of Science were searched until July 2017. Author contact, reference and citation lists were checked to obtain additional references.

STUDY SELECTION

To be included, available full-texts had to be published in English, French, Spanish or Italian and (a) involved undergraduate or postgraduate nursing students performing HFPS based on life-threatening clinical condition scenarios, (b) contained control groups not tested on the HFPS before the intervention, (c) contained data measuring learning outcomes such as performance, knowledge, self-confidence, self-efficacy or satisfaction measured just after the simulation session and (d) reported data for meta-analytic synthesis.

REVIEW METHOD

Three independent raters screened the retrieved studies using a coding protocol to extract data in accordance with inclusion criteria.

SYNTHESIS METHOD

For each study, outcome data were synthesised using meta-analytic procedures based on random-effect model and computing effect sizes by Cohen's d with a 95% CI.

RESULTS

Thirty-three studies were included. HFPS sessions showed significantly larger effects sizes for knowledge (d=0.49, 95% CI [0.17 to 0.81]) and performance (d=0.50, 95% CI [0.19 to 0.81]) when compared with any other teaching method. Significant heterogeneity among studies was detected.

CONCLUSIONS

Compared with other teaching methods, HFPS revealed higher effects sizes on nursing students' knowledge and performance. Further studies are required to explore its effectiveness in improving nursing students' competence and patient outcomes.

Reliability and validity study of the Spanish adaptation of the "Creighton Simulation Evaluation Instrument (C-SEI)"

There are multiple advantages to using human patient simulation (HPS) as a teaching method for clinical nursing education. Valid, reliable tools that can be used when applying this teaching method are needed to evaluate nursing student skill acquisition. The aim of this study was to translate the Creighton Simulation Evaluation Instrument (C-SEI) into Spanish and to analyse the reliability and validity of the Spanish C-SEI version with nursing students. The study was conducted in two phases: (1) Adaptation of the instrument into Spanish. (2) Cross-sectional study in a sample of 249 nursing students who were evaluated by two observers. The psychometric properties were analysed in terms of reliability (internal consistency and inter-observer consistency) and construct validity using an exploratory factor analysis. Questionnaire internal consistency was 0.839 for the tool as a whole. Inter-observer concordance for the tool as a whole was 0.936 and greater than 0.80 for the majority of the items. The exploratory factor analysis showed a four-factor structure that explains 49.5% of the total variance. The results of this study show that the C-SEI-sp tool is a valid and reliable tool that is easy to apply in the monitoring of student performance in clinical simulation scenarios.

Using Scenario-Based Simulation to Address Affective Behaviors in Sonography Students

Simulation in health care education is considered an educational strategy in which a particular set of conditions is created or replicated to resemble authentic situations that are possible in real life. This research study records sonography student self-reflected confidence levels in three patient care scenarios using pre- and postsimulation surveys and debriefing strategies. Student confidence improved in (1) explaining the examination, (2) prepping the patient, and (3) performing the examination. In addition, students stated they had more confidence in what to do when a negative outcome is communicated to the patient. They also expressed greater appreciation for the importance of institutional policies that guide patient care. Research is ongoing to determine whether the simulation lab experience correlates to improved student performance in the actual clinical setting. This research occurred in a simulation lab but could be replicated by using high-fidelity sonography simulators and assigning an actor to interact with student learners.