An Updated Review of Published Simulation Evaluation Instruments

Instructor-led vs. peer-led debriefing in preoperative care simulation using standardized patients

BACKGROUND

Debriefing involves the active participation of learners to identify and close gaps in knowledge and skills. Varied debriefing methods are used in simulation, but no empirical studies have examined the effectiveness of peer-led debriefing in simulation using standardized patients. The purpose of this study was to compare the effects of two debriefing methods (instructor-led vs. peer-led) on nursing skills, knowledge, self-confidence, and quality of debriefing among undergraduate nursing students in South Korea.

METHOD

A nonequivalent control group pretest-posttest design was used. Fifty-seven third-year nursing students were randomly assigned to instructor-led (n = 26) or peer-led (n = 31) debriefing groups after a simulation of preoperative care. Structured questions and areas for discussion guided debriefing in both groups. Self-administered questionnaires were used to collect data on students' knowledge and self-confidence in providing preoperative care. Faculty evaluated students' nursing skills during pre- and post-simulation practice. Students evaluated the quality of the debriefings.

RESULTS

Nursing skills for preoperative care (p < .001) and the quality of debriefing (p < .001) were statistically higher in the instructor-led group compared to the peer-led group. There were no statistically significant differences in knowledge (p = .445) and self-confidence (p = .686). Knowledge and self-confidence from pre-test to posttest were improved in both groups.

CONCLUSION

The instructor-led debriefing showed improved nursing skills and higher quality debriefing. However, peer-led debriefing led by a non-trained peer also rendered positive results. Thus, peer-led debriefing may be considered a useful strategy for improving nursing students' self-confidence.

Kirkpatrick's Evaluation of Simulation and Debriefing in Health Care Education: A Systematic Review

BACKGROUND

Simulation is an integral component of health care education. Research suggests a positive relationship between simulation and learning outcomes. Kirkpatrick's framework is a four-level model based on the premise that learning resulting from training programs can be classified into four levels: reaction, learning, behavior, and results. Evaluation of educational impact provides valuable feedback to educators that may assist with development and improvement of teaching methods.

METHOD

This review is based on the PRISMA guidelines for conducting a systematic review. Inclusion criteria included articles (a) written in the English language, (b) published between 2000 and 2016, (c) describing a debriefing intervention after high-fidelity patient simulation, and (d) based in health care.

RESULTS

Thirteen studies met criteria for inclusion in the review.

CONCLUSION

Results indicated a paucity of studies at the highest levels of evaluation, indicating an area where future research is needed to assist with the development and improvement of simulation education. [J Nurs Educ. 2018;57(7):393-398.].

Confiabilidade e validade da Lasater Clinical Judgment Rubric – Brazilian Version

Resumo Objetivo Avaliar a confiabilidade e validade da Lasater Clinical Judgment Rubric – Brazilian Version em uma amostra de estudantes de enfermagem. Métodos Pesquisa com delineamento metodológico, realizado em uma instituição pública de ensino superior da região sudeste do Brasil. Preliminarmente, foi analisada a clareza dos comportamentos descritos em cada nível das 11 dimensões do instrumento, por oito estudantes. A seguir, deu-se a coleta de dados relativos a avaliação das propriedades psicométricas com auto aplicação a 179 estudantes. Os participantes foram agrupados em iniciantes (1ª e 2ª séries n=115) e concluintes (3ª e 4ª séries n=64). Foram analisadas as propriedades psicométricas: validade discriminante, a fidedignidade e a dimensionalidade. Resultados A versão brasileira da rubrica diferenciou os dois grupos de estudantes (p-valor <0,05) nas 11 dimensões avaliadas. A estabilidade foi verificada pelo teste-reteste (ICC de 0,88). A consistência interna foi obtida para o instrumento global (alpha de Cronbach, de 0,889) e para as fases de reconhecimento (α=0,75), interpretação (α=0,64), resposta (α=0,78) e reflexão (α=0,63). Validade da dimensionalidade pela análise fatorial confirmatória (AFC) obteve resultados de confiabilidade composta (CC) acima de 0,7 e a variância média extraída (AVE) superiores a 0,5, em todas as fases. A validade discriminante do modelo fatorial pelo critério de Fornell-Larcker e pelas cargas cruzadas confirmaram a estrutura teórica da versão original da rubrica. Conclusão A avaliação das propriedades psicométricas da versão brasileira da rubrica mostrou evidências de confiabilidade e validade de construto do instrumento para medir o desenvolvimento do julgamento clínico do estudante de enfermagem.

Cultural adaptation and validation of the Lasater Clinical Judgment Rubric in nursing students in Spain

BACKGROUND

The clinical judgment and decision-making abilities of nurses can influence many health outcomes, hence the importance of addressing these qualities in university studies. In this respect, clinical simulation is a commonly employed teaching method. The evaluation of simulation activities requires standardised instruments, such as the Lasater Clinical Judgment Rubric, which is widely used for this purpose, although a culturally adapted and validated version in Spain is not available.

AIMS

To obtain a Spanish culturally adapted and validated version of the rubric for undergraduate students of nursing.

DESIGN, PARTICIPANTS AND SETTING

Cultural adaptation and psychometric validation study carried out with undergraduate nursing students in the simulation laboratories at the University of Málaga (Spain).

METHODS

A process of translation/back-translation and cultural adaptation was carried out in accordance with international standards. The rubric was empirically evaluated in standardised scenarios with high and medium-fidelity simulators. Each student took part in two different simulation sessions, led by two instructors. In each simulation, the data were collected by two independent observers.

RESULTS

152 observations were obtained from 76 students. The interobserver reliability was high, with an intraclass correlation coefficient of 0.93 (95% CI 0.92-0.95) (p = 0.0001) and Cronbach's alpha of 0.93. According to the confirmatory factor analysis, the fit of the model was satisfactory in all indices, with a χ²/df value of 1.08, GFI 0.96, TLI 0.99, NFI 0.97 and RMSEA 0.24 (90% CI 0.000-0.066).

CONCLUSIONS

The rubric obtained is culturally adapted to the Spanish educational context, and is valid and reliable for nursing students. Further prospective studies should be undertaken to evaluate the responsiveness, potential for transfer to clinical practice and cost-benefit ratios of different simulation designs.

Development and Psychometric Evaluation of the Facilitator Competency Rubric

AIM

The purpose of the study was to evaluate the psychometric properties of the Facilitator Competency Rubric (FCR).

BACKGROUND

The FCR was developed to fill a deficit in tools available to holistically evaluate competency of simulation facilitators.

METHOD

The FCR has five constructs: preparation, prebriefing, facilitation, debriefing, and evaluation. Benner's novice-to-expert theory was used to categorize subcomponents. Expert panel validation was sought. Participants from four baccalaureate nursing programs completed 107 observations of facilitators using the FCR.

RESULTS

Interrater reliability as assessed using generalizability theory was good. G coefficients ranged from good to excellent (.80 to .99). FCR items appeared to be well constructed. Term, day of the week, time of day, and simulation type were significant predictors of the FCR global score.

CONCLUSION

The FCR is a valid, reliable tool that can be used for formative or summative evaluation of simulation facilitators. Scores can guide and prioritize faculty development.

Using Generalizability Theory to Inform Optimal Design for a Nursing Performance Assessment

The promotion of competency of nurses and other health-care professionals is a goal shared by many stakeholders. In nursing, observation-based assessments are often better suited than paper-and-pencil tests for assessing many clinical abilities. Unfortunately, few instruments for simulation-based assessment of competency have been published that have undergone stringent reliability and validity evaluation. Reliability analyses typically involve some measure of rater agreement, but other sources of measurement error that affect reliability should also be considered. The purpose of this study is three-fold. First, using extant data collected from 18 nurses evaluated on 3 Scenarios by 3 Raters, we utilize generalizability (G) theory to examine the psychometric characteristics of the Nursing Performance Profile, a simulation-based instrument for assessing nursing competency. Results corroborated findings of previous studies of simulation-based assessments showing that obtaining desired score reliability requires substantially greater numbers of scenarios and/or raters. Second, we provide an illustrative exemplar of how G theory can be used to understand the relative magnitudes of sources of error variance-such as scenarios, raters, and items-and their interactions. Finally, we offer general recommendations for the design and psychometric study of simulation-based assessments in health-care contexts.

Learning theories and tools for the assessment of core nursing competencies in simulation: A theoretical review

AIM

To identify the theories used to explain learning in simulation and to examine how these theories guided the assessment of learning outcomes related to core competencies in undergraduate nursing students.

BACKGROUND

Nurse educators face the challenge of making explicit the outcomes of competency-based education, especially when competencies are conceptualized as holistic and context dependent.

DESIGN

Theoretical review.

DATA SOURCES

Research papers (N = 182) published between 1999-2015 describing simulation in nursing education.

REVIEW METHODS

Two members of the research team extracted data from the papers, including theories used to explain how simulation could engender learning and tools used to assess simulation outcomes. Contingency tables were created to examine the associations between theories, outcomes and tools.

RESULTS

Some papers (N = 79) did not provide an explicit theory. The 103 remaining papers identified one or more learning or teaching theories; the most frequent were the National League for Nursing/Jeffries Simulation Framework, Kolb's theory of experiential learning and Bandura's social cognitive theory and concept of self-efficacy. Students' perceptions of simulation, knowledge and self-confidence were the most frequently assessed, mainly via scales designed for the study where they were used. Core competencies were mostly assessed with an observational approach.

CONCLUSION

This review highlighted the fact that few studies examined the use of simulation in nursing education through learning theories and via assessment of core competencies. It also identified observational tools used to assess competencies in action, as holistic and context-dependent constructs.

Effect of Nurse-Led Review Plus Simulation on Obstetric/Perinatal Nurses' Self-Assessed Knowledge and Confidence

Simulation may help both novice and experienced clinicians maintain competence in managing high-risk, low-frequency obstetric and perinatal complications and emergencies. Therefore, we designed a pre-/posttest study to determine whether a day of nurse-led lecture plus low-fidelity simulation would increase registered nurses' self-assessed knowledge and confidence in managing five high-risk obstetric/perinatal situations. The Nursing Management of OB/Perinatal Complications & Emergencies (NursOB) scale was distributed to 67 labor/birth and postpartum nurses before and after a simulation training day. Preliminary findings supported validity and reliability of the NursOB scale, but nurses' knowledge and confidence did not improve after the simulation (p < .05). Anecdotally, nurses' interest in competence reviews was reinvigorated, and we gained practical knowledge in simulation delivery. Future simulations could enhance outcome measures, improve drills, and establish criterion-related validity of the NursOB scale. More research is warranted.

A systematic review of the effectiveness of simulation-based education on satisfaction and learning outcomes in nurse practitioner programs

BACKGROUND

High-fidelity simulation (HFS) is becoming an integral component in healthcare education programs. There is considerable evidence demonstrating the effectiveness of HFS on satisfaction and learning outcomes within undergraduate nursing programs; however, there are few studies that have investigated its use and effectiveness within nurse practitioner (NP) programs.

OBJECTIVE

To synthesize the best available evidence about the effectiveness of HFS within NP education programs worldwide. The specific review question was: what is the effect of HFS on learner satisfaction, knowledge, attitudes, and skill performance in NP education?

METHODS

Joanna Briggs Institute systematic review methodology was utilized. The following databases were searched: MEDLINE, CINAHL, EMBASE, Epistemonikos, PROSPERO, HealthSTAR, AMED, Cochrane, Global Health and PsycINFO. Studies were included if they were quantitative in nature and reported on any aspect HFS within a NP program.

RESULTS

Ten studies were included in the review. All studies were conducted in the United States and published between 2007 and 2014. Outcomes explored included: knowledge, attitudes, skills and satisfaction. The majority of studies compared HFS to online learning or traditional classroom lecture. Most study scenarios featured high acuity, low frequency events within acute care settings; only two studies utilized scenarios simulated within primary care.

CONCLUSIONS

There is limited evidence supporting the use of HFS within NP programs. In general, HFS increases students' knowledge and confidence, and students are more satisfied with simulation-based teaching in comparison to other methods. Future studies should explore the effectiveness of simulation training within NP programs in reducing the theory to practice gap, and evaluate knowledge retention, transferability to real patient situations, and impact of simulation on patient outcomes.

The validity of a professional competence tool for physiotherapy students in simulation-based clinical education: a Rasch analysis

BACKGROUND

Despite the recent widespread adoption of simulation in clinical education in physiotherapy, there is a lack of validated tools for assessment in this setting. The Assessment of Physiotherapy Practice (APP) is a comprehensive tool used in clinical placement settings in Australia to measure professional competence of physiotherapy students. The aim of the study was to evaluate the validity of the APP for student assessment in simulation settings.

METHODS

A total of 1260 APPs were collected, 971 from students in simulation and 289 from students in clinical placements. Rasch analysis was used to examine the construct validity of the APP tool in three different simulation assessment formats: longitudinal assessment over 1 week of simulation; longitudinal assessment over 2 weeks; and a short-form (25 min) assessment of a single simulation scenario. Comparison with APPs from 5 week clinical placements in hospital and clinic-based settings were also conducted.

RESULTS

The APP demonstrated acceptable fit to the expectations of the Rasch model for the 1 and 2 week clinical simulations, exhibiting unidimensional properties that were able to distinguish different levels of student performance. For the short-form simulation, nine of the 20 items recorded greater than 25 % of scores as 'not-assessed' by clinical educators which impacted on the suitability of the APP tool in this simulation format.

CONCLUSION

The APP was a valid assessment tool when used in longitudinal simulation formats. A revised APP may be required for assessment in short-form simulation scenarios.

Writing a Simulation Scenario: A Step-By-Step Guide

Simulation is becoming a widely used method of helping nurses learn and maintain competency in the clinical area for both staff educators in clinical settings and nursing faculty in academic settings. Designing an effective simulation experience requires thoughtful planning, knowledge of educational principles, and knowledge of best practices in both simulation and clinical practice. An evidence-based strategy for writing a simulation scenario for nurses and other health care providers in any setting is described. A step-by-step process is outlined that incorporates best practices. Examples and suggestions are provided to help readers create quality simulation experiences.

Development of an assessment instrument to evaluate performance of the skill of decontamination

BACKGROUND AND PURPOSE

Nurses must competently demonstrate psychomotor skills. Few reliable and valid instruments are available for psychomotor evaluation for disaster skills, including the skill of decontamination.

OBJECTIVES

The purpose of this study was to develop and refine an instrument to measure the skill of decontamination.

DESIGN

A seven step instrument development design was implemented that included content validity and reliability as well as inter-rater reliability analysis.

SETTING AND PARTICIPANTS

A convenience sample of approximately 140 participants was drawn from two colleges of nursing at two large academic universities. The sample included senior nursing students in either their community or final practicum nursing course.

METHODS

Based on a sample of 140 students who participated in a decontamination training experience using virtual reality simulation, a seven step established process for assessment of reliability and validity was implemented to develop a checklist for the skill of decontamination.

RESULTS

The final instrument statistics: Content Validity Index for the overall instrument score was 0.94, Internal consistency coefficient=0.607(KR-20) and Inter-rater reliability=0.9114.

CONCLUSIONS

This instrument provides a reliable and valid assessment of nurses' competency in performing the skill of decontamination offering a template for educators to develop similar tools.

INACSL Standards of Best Practice for Simulation: Past, Present, and Future

AIM

To describe the historical evolution of the International Nursing Association for Clinical Simulation and Learning's (INACSL) Standards of Best Practice: Simulation.

BACKGROUND

The establishment of simulation standards began as a concerted effort by the INACSL Board of Directors in 2010 to provide best practices to design, conduct, and evaluate simulation activities in order to advance the science of simulation as a teaching methodology.

METHOD

A comprehensive review of the evolution of INACSL Standards of Best Practice: Simulation was conducted using journal publications, the INACSL website, INACSL member survey, and reports from members of the INACSL Standards Committee.

RESULTS

The initial seven standards, published in 2011, were reviewed and revised in 2013. Two new standards were published in 2015. The standards will continue to evolve as the science of simulation advances.

CONCLUSION

As the use of simulation-based experiences increases, the INACSL Standards of Best Practice: Simulation are foundational to standardizing language, behaviors, and curricular design for facilitators and learners.

Advancing renal education: hybrid simulation, using simulated patients to enhance realism in haemodialysis education

BACKGROUND

Simulation is a well-established and proven teaching method, yet its use in renal education is not widely reported. Criticisms of simulation-based teaching include limited realism and a lack of authentic patient interaction.

METHOD

This paper discusses the benefits and challenges of high-fidelity simulation and suggests hybrid simulation as a complementary model to existing simulation programmes.

CONCLUSION

Through the use of a simulated patient, hybrid simulation can improve the authenticity of renal simulation-based education while simultaneously teaching and assessing technologically enframed caring.