Kirkpatrick evaluation model for in-service training on cardiopulmonary resuscitation

Background:

There are several evaluation models that can be used to evaluate the effect of in-service training; one of them is the Kirkpatrick model. The aim of the present study is to assess the in-service training of cardiopulmonary resuscitation (CPR) for nurses based on the Kirkpatrick's model.

Materials and Methods:

This study is a cross-sectional study based on the Kirkpatrick's model in which the efficacy of in-service training of CPR to nurses was assessed in the Shahadaye Lenjan Hospital in Isfahan province in 2014. 80 nurses and Nurse's aides participated in the study after providing informed consent. The in-service training course was evaluated in reaction, learning, behavior, and results level of the Kirkpatrick model. Data were collected through a researcher-made questionnaire.

Results:

The mean age of the participants was 35 ± 8.5 years. The effectiveness score obtained in the reaction level (first level in the Kirkpatrick model) was 4.2 ± 0.32. The effectiveness score in the second level of model or the learning level was 4.70 ± 0.09, which is statistically significant (P < 0.001). The effectiveness score at the third and fourth level were 4.1 ± 0.34 and 4.3 ± 0.12, respectively. Total effectiveness score was 4.35.

Conclusions:

The results of this study showed that CPR in-service training has a favorable effect on all four levels of the Kirkpatrick model for nurses and nurse's aides.

Teaching evidence-based medicine with electronic databases for preclinical education

To evaluate the efficacy of an educational module on evidence-based medicine (EBM) assisted with electronic medical databases (EMDs) for preclinical education, medical students (n = 111) were matriculated in a program consisted of 16 2-h sessions on EBM plus hands-on experience on EMDs in a problem-based learning-type format. Students were required to make an oral presentation on designated clinical scenarios before and after the sessions, without prior notice, as an indicator of performance. In addition, questionnaires focusing on behavioral changes, awareness, and confidence of mastering EBM were administered before and after the sessions to assess the attitudinal and behavioral impact of the intervention on the participants. We found evidence of better postprogram performance in utilizing EBM-relevant concepts and resources when the enrolled medical students were giving oral presentations. Moreover, the participants reported increased awareness of EBM and, behaviorally, increased utilization of EBM-relevant resources provided by libraries. Also, they reported improvement on appropriately using EBM-relevant resources, and 99% of the participants reported strong confidence in practicing EBM. In conclusion, modules on EBM implemented with EMDs benefitted medical students in scenario-oriented PBL tutorials. Improvements in awareness, behavior, confidence, and performance in mastering EBM were noted.

Evaluation of a continuing professional development training program for physicians and physician assistants in hospitals in Laos based on the Kirkpatrick model

PURPOSE

Medical professionals from Korea and Laos have been working together to develop a continuing professional development training program covering the major clinical fields of primary care. This study aimed to evaluate the effectiveness of the program from 2013 to 2014 using the Kirkpatrick model.

METHODS

A questionnaire was used to evaluate the reaction of the trainees, and the trainers assessed the level of trainees' performance at the beginning and the end of each clinical section. The transfer (behavioral change) of the trainees was evaluated through the review of medical records written by the trainees before and after the training program.

RESULTS

The trainees were satisfied with the training program, for which the average score was 4.48 out of 5.0. The average score of the trainees' performance at the beginning was 2.39 out of 5.0, and rose to 3.88 at the end of each section. The average score of the medical records written before the training was 2.92 out of 5.0, and it rose to 3.34 after the training. The number of patient visits to the district hospitals increased.

CONCLUSION

The continuing professional development training program, which was planned and implemented with the full engagement and responsibility of Lao health professionals, proved to be effective.

Implementation research training for learners in low- and middle-income countries: evaluating behaviour change after participating in a massive open online course

BACKGROUND

Implementation research (IR) can play a critical role in the delivery of disease control interventions, particularly in low- and middle-income countries (LMICs). The growing demand for IR training has led to the development of a range of training programmes and university courses, the majority of which can not be accessed by learners in LMICs. This article reports on the evaluation of the massive open online course (MOOC) developed by the Special Programme for Research and Training in Tropical Diseases hosted by WHO on the topic of IR with a focus on infectious diseases of poverty. This study followed the Kirkpatrick Model to evaluate training programmes with a specific focus on post-training changes in behaviour.

METHODS

MOOC participants were invited to take part in an anonymous online survey examining their knowledge of IR and how they applied it in their professional practice approximately 1-1.5 years after completing their course. The survey contained 43 open-ended, multiple choice and Likert-type questions. Descriptive statistics were calculated for the quantitative data and responses to the open-ended questions were thematically coded.

RESULTS

A total of 748 MOOC participants responded to the survey. The demographic profile of the survey respondents aligned with that of the MOOC participants, with nearly 70% of respondents originating from Africa. Responses to the quantitative and open-ended survey questions revealed that respondents' knowledge of IR had improved to a large extent as a result of the MOOC, and that they used the knowledge and skills gained in their professional lives frequently and had consequently changed their professional behaviour. Respondents most often cited the problem-solving aspect of IR as a substantial area of behavioral change influenced by participating in the MOOC.

CONCLUSIONS

These findings indicate that the MOOC was successful in targeting learners from LMICs, in strengthening their IR knowledge and contributing to their ability to apply it in their professional practice. The utility of MOOCs for providing IR training to learners in LMICs, where implementation challenges are encountered often, makes this platform an ideal standalone learning tool or one that could be combined with other training formats.

Study design and protocol for a comprehensive evaluation of a UK massive open online course (MOOC) on quality improvement in healthcare

INTRODUCTION

Massive open online courses (MOOCs) offer a flexible approach to online and distance learning, and are growing in popularity. Several MOOCs are now available, to help learners build on their knowledge in a number of healthcare topics. More research is needed to determine the effectiveness of MOOCs as an online education tool, and explore their long-term impact on learners' professional practice. We present a protocol describing the design of comprehensive, mixed-methods evaluation of a MOOC, 'QualityImprovement (QI) inHealthcare: the Case for Change', which aims to improve learner's knowledge and understanding of QI approaches in healthcare, and to increase their confidence in participating, and possibly leading QI projects.

METHODS AND ANALYSIS

A pre-post study design using quantitative and qualitative methods will be used to evaluate the QI MOOC. Different elements of the RE-AIM (reach, effectiveness and maintenance) and Kirkpatrick (reaction, learning and behaviour) models will be used to guide the evaluation. All learners who register for the course will be invited to participate in the QI MOOC evaluation study. Those who consent will be asked to complete a presurvey to assess baseline QI knowledge (self-report and objective) and perceived confidence in engaging in QI activities. On completion of the course, participants will complete a postsurvey measuring again knowledge and perceived confidence. Feedback on the course content and how it can be improved. A subset of participants will be invited to take part in a follow-up qualitative interview, 3 months after taking the course, to explore in depth how the MOOC impacted their behaviour in practice.

ETHICS AND DISSEMINATION

The study has been approved by the University of Bath Human Research Ethics Committee (reference: 2958). Study findings will be published in peer-reviewed journals, and disseminated at conference and departmental presentations, and more widely using social media, microblogging sites and periodicals aimed at healthcare professionals.

The evaluation of a home-based paediatric nursing service: concept and design development using the Kirkpatrick model

Background

A new paediatric hospital-in-the-home nursing service required evaluation.

Aims

To determine whether the education and training provided for nursing staff employed in the service was effective.

Methods

This paper presents the way in which a training evaluation model supported the design and evaluation of a training programme for registered nurses working in an out-of-hospital, home-based nursing service for paediatric patients.

Results

The Kirkpatrick model provides a framework for evaluating the effectiveness of workforce training for any industry including healthcare (Kirkpatrick, 2009).

Conclusions

That the Kirkpatrick model is an appropriate framework to evaluate a nursing training programme, but it is imperative to evaluate all levels of the model to be able to ascertain the success of the training and the impact on clinical practice.

A pilot study on the simulation-based training for Ethiopia skilled birth attendants

This simulation-based Ethiopia Midwife Train-the-Trainer program for skilled birth attendants is one part of the China-UK Global Health Support Program (GHSP) activities and an approach to improving the Ethiopia trainers' midwifery teaching skills. This pilot study aimed to evaluate the effectiveness of a simulation-based training for Ethiopia skilled birth attendants. This program was divided into three stages. In Stage 1, Fudan team trained 5 trainers from Ethiopia. In Stage 2, Fudan team supervised and evaluated the 1st stage Ethiopia trainers to train their peers becoming the trainers in Fudan University. In Stage 3, Fudan team evaluated the training program implementation in established Advanced Midwifery Practice Center in Tirunesh Beijing General Hospital. The participatory observational method was used for process evaluation and the Kirkpatrick model was used for effectiveness evaluation. The Ethiopia staff's teaching skills for simulation-based midwifery training were strengthened and their abilities to deal with various complicated obstetric situations were improved by group explanation, questions, rehearsal, and step-by-step interactive teaching (p < .01). The mean score of total mastery degree was 3.48 ± 0.16 by self-assessment and 3.63 ± 0.16 by peer-assessment in stage 2, and 3.27 ± 0.24 by self-assessment and 3.38 ± 0.34 by peer-assessment in stage 3.The GHSP-OP4-V01 Ethiopia Midwife Train-the-Trainer Program has received positive response in Ethiopia by in-depth interview and questionnaire survey. This study demonstrated that the simulation-based training is an effective approach for midwifery Train-the-Trainer program in Ethiopia.

Evaluation of the Care Pathway in the Context of the Dispensing of Emicizumab (Hemlibra) in Community Pharmacies in France: Protocol for a Cross-sectional Study Based on the Kirkpatrick Model

BACKGROUND

Commercialized since 2019, emicizumab (Hemlibra) was available only in French hospital pharmacies for prophylaxis of hemophilia A with or without inhibitors. Since June 15, 2021, patients can choose between a hospital and community pharmacy. These changes in the care pathway have important organizational consequences for patients, their relatives, and health professionals. Two training programs are available for community pharmacists: the "HEMOPHAR" training program proposed by the national reference center for hemophilia and the Roche training program proposed by the laboratory that markets the product.

OBJECTIVE

The PASODOBLEDEMI study aims to evaluate the direct impact of the training programs provided to community pharmacists in the context of the dispensing of emicizumab, and to evaluate patients' satisfaction with their treatment whether they choose dispensation from a community pharmacy or retained dispensation from the hospital pharmacy.

METHODS

We designed a cross-sectional study based on the 4-level Kirkpatrick evaluation model: the immediate reaction of community pharmacists following training (Reaction), the knowledge acquired during the training (Learning), the professional practice of community pharmacists during dispensing of the product (Behavior), and patients' satisfaction related to the treatment whether it is dispensed from a hospital or from a community pharmacy (Results).

RESULTS

Considering that single outcome measures cannot adequately reflect the complexity of this new organization, the Kirkpatrick evaluation model provides 4 distinct outcomes: the immediate reaction after the HEMOPHAR training program, the level of knowledge acquired after the HEMOPHAR training program, the impact of training on professional practice, and patient satisfaction with access to emicizumab. We developed specialized questionnaires for each of the 4 levels of the Kirkpatrick evaluation model. All community pharmacists involved in dispensing emicizumab, whether they have followed the HEMOPHAR or the Roche training program or neither, were eligible for inclusion. All patients with severe hemophilia A were eligible, irrespective of inhibitor use, age, treatment with emicizumab, and whether they chose dispensation from a community pharmacy or retained dispensation from a hospital pharmacy.

CONCLUSIONS

The new organization for dispensing emicizumab to patients with hemophilia A in French community pharmacies must be accompanied by optimal safety and quality conditions due to the risk of serious and urgent bleeding situations in the management of rare bleeding diseases. The elaboration of the PASODOBLEDEMI protocol has already a positive impact with the commitment of all health professionals, physicians, hospital and community pharmacists, and the patient community. The results will be disseminated among the French authorities and will enable, if necessary, proposing this access model to other rare diseases.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05449197, https://clinicaltrials.gov/ct2/show/NCT05449197?term=NCT05449197; ClinicalTrials.gov NCT05450640, https://clinicaltrials.gov/ct2/show/NCT05450640?term=NCT05450640.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/43091.

Evaluating the effect of an in-service training workshop on ICD-10 coding instructions of pregnancy, childbirth and the puerperium for clinical coders

The quality of the data coded based on the 10^th revision of the International Classification of Diseases (ICD-10) can be improved by providing continuous education and promoting the clinical coders’ knowledge and skills. Due to the significance of maternal health in promoting the health of society, the present study evaluated the effects of an in-service training workshop on ICD-10 coding instructions of pregnancy, childbirth, and the puerperium for clinical coders. This applied evaluation study was conducted to evaluate the effects of a coding instructions training course focusing on the 15^th chapter of the ICD-10. The statistical population comprised 45 clinical coders working in the hospitals. The data were collected by a researcher-made questionnaire scored on a five-point Likert scale at the reaction level and by pretest and posttest questionnaires at the learning level. The data were then analyzed by descriptive statistics at the reaction level and by a paired-samples t-test at the learning level. The participants’ satisfaction with the training course was 94.7% on average at the reaction level. At the learning level, the results of the paired-samples t-test showed a significant difference between the means of scores before and after the training course (p=0.000). The training course led to satisfaction and enhanced the capabilities of the clinical coders with regard to coding the 15^th chapter of ICD-10. Clinical Coders must receive training on the new changes and guidelines in the other chapters of ICD-10 based on its most recent revision and employ them in the workplace.

Evaluation of a drug and alcohol safety education program in aviation using interrupted time series and the Kirkpatrick framework

Compliance with drug and alcohol regulations are required by 14 CFR Part 120/ and 49 CFR Part 40. These regulations affect approximately 7200 aviation-related companies and their associated services. Consequences for noncompliance can include loss of revenue from imposition of civil penalties, suspension, or revocation of the company's certificate to conduct business. Front End Analysis (FEA) was conducted to determine specific performance problems and provide five tailored educational interventions to address them. Program evaluation was conducted using Interrupted Time Series (ITS) modeling. Results showed significant (p < .05) decreases in nonconformities across all five models with small to moderate effect sizes. Based on the relative effects, values for reductions in civil penalty costs between 16% and 47%, were predicted. Actual sanction reductions from the pre-to-post-intervention periods were confirmed to be 24.21%. The study supported the efficacy of the ITS approach for implementation of level four Kirkpatrick evaluation.

Applying the Kirkpatrick model to evaluate dental students' experience of learning about antibiotics use and resistance

INTRODUCTION

"Antibiotic resistance" is of main concern in global health and that it could hinder the achievement of sustainable development goals. One of the reported contributing factors is the irrational prescribing behaviour of healthcare professionals including dentists. Efforts to design and evaluate effective educational programmes for undergraduate dental students about appropriate prescribing behaviour during their early educational years could mitigate the risk of antibiotic resistance.

MATERIALS AND METHODS

A total of 322 students participated in the study. Their knowledge of antibiotics and antibiotic resistance was assessed. Students received a specially designed, interactive educational programme. The success of the programme was assessed after the educational sessions and 2 years later using Kirkpatrick's four-level evaluation model.

RESULTS

This study is the first study to use Kirkpatrick model to evaluate an educational programme for dental students. The results of the first level of evaluation showed an overall satisfaction score of 82.1%. The second level revealed an increase in the percentage of correct answers after the educational sessions from 68.3% to 80%, and significant agreement with responsible antibiotics usage (p-value = .020, Effect size = 0.121). The third level showed that the percentage of correct answers 2 years later was 87.5%. The fourth level confirmed the success of the programme as 88.9% of participants reported using the knowledge gained from the programme when prescribing antibiotics.

CONCLUSIONS

Our study emphasises the success of the used educational programme and highlights the need for educational interventions in the under graduate dental curriculum.

Leveraging gains from African Center for Integrated Laboratory Training to combat HIV epidemic in sub-Saharan Africa

Background

In sub-Saharan Africa, there is dearth of trained laboratorians and strengthened laboratory systems to provide adequate and quality laboratory services for enhanced HIV control. In response to this challenge, in 2007, the African Centre for Integrated Laboratory Training (ACILT) was established in South Africa with a mission to train staffs from countries with high burdens of diseases in skills needed to strengthen sustainable laboratory systems. This study was undertaken to assess the transference of newly gained knowledge and skills to other laboratory staff, and to identify enabling and obstructive factors to their implementation.

Methods

We used Kirkpatrick model to determine training effectiveness by assessing the transference of newly gained knowledge and skills to participant’s work environment, along with measuring enabling and obstructive factors. In addition to regular course evaluations at ACILT (pre and post training), in 2015 we sent e-questionnaires to 867 participants in 43 countries for course participation between 2008 and 2014. Diagnostics courses included Viral Load, and systems strengthening included strategic planning and Biosafety and Biosecurity. SAS v9.44 and Excel were used to analyze retrospective de-identified data collected at six months pre and post-training.

Results

Of the 867 participants, 203 (23.4%) responded and reported average improvements in accuracy and timeliness in Viral Load programs and to systems strengthening. For Viral Load testing, frequency of corrective action for unsatisfactory proficiency scores improved from 57 to 91%, testing error rates reduced from 12.9% to 4.9%; 88% responders contributed to the first national strategic plan development and 91% developed strategies to mitigate biosafety risks in their institutions. Key enabling factors were team and management support, and key obstructive factors included insufficient resources and staff’s resistance to change.

Conclusions

Training at ACILT had a documented positive impact on strengthening the laboratory capacity and laboratory workforce and substantial cost savings. ACILT’s investment produced a multiplier effect whereby national laboratory systems, personnel and leadership reaped training benefits. This laboratory training centre with a global clientele contributed to improve existing laboratory services, systems and networks for the HIV epidemic and is now being leveraged for COVID-19 testing that has infected 41,332,899 people globally.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-020-06005-8.

Addressing the gap in health data management skills: an online self-guided course for researchers and health professionals

BACKGROUND

The healthcare sector is rapidly evolving with the rise of digital technology and data-driven decision-making. However, traditional medical education has yet to fully integrate training on managing health-related information, resulting in a significant skills gap among medical and research professionals. This gap is pronounced in low- and middle-income countries (LMICs), where data privacy concerns and inadequate infrastructure hinder efforts to utilise and share health data.

AIMS

To address this gap, we developed an online, modular course aimed at providing foundational skills on capturing, storing and sharing health data.

METHODS

The course was developed using the ADDIE(Analyze, Design, Develop, Implement, Evaluate) instructional design model. A needs assessment workshop involving 25 global health proffesionals identified key training gaps which informed the curriculum's development. A multidisciplinary team from six institutions developed the modules. The course was piloted in a face-to-face setting with 37 participants and later adapted for online delivery via the Global Health Network platform. We evaluated the course using Level 1 of Kirkpatrick's model for training evaluation.

RESULTS

Six foundational modules were developed: Introduction to Data Management, Data Quality, Data Repositories, Ethics of Data Sharing, Data Governance, and Costing for Data Management. Between December 2020 and April 2024, 6,384 individuals from 90 countries completed the course. Of these, 32% were from Africa, 15% from Asia, 16% from South/Central America and the Caribbean, and 24% from Europe. Summative evaluations, based on voluntary post-module surveys, demonstrated high relevance to participants' learning needs (96.6%) and strong intentions to apply the skills gained (88.3%). Key motivators for enrollment included the course's free access, relevance to professional or academic needs, and trust in the organizations and authors behind the content.

CONCLUSIONS

The high enrolment and broad geographical reach demonstrates the potential of online training as a cost-effective tool to equip health practitioners and researchers with data literacy skills. Future evaluations will assess its impact on participants' knowledge, behavior, and data-sharing and reuse practices.

Construction and application of standardized training effect evaluation system for new nurses in operating room

Background

This study aims to develop and validate a Structured Training Effectiveness Evaluation (STEE) tool based on the Kirkpatrick model for newly graduated registered nurses in the operating room in China.

Methods

The first phase will involve focus group and individual interviews with nursing educators and newly graduated registered nurses selected using purposive sampling. The data will be analyzed thematically to identify key components necessary to develop the STEE tool. The second phase will develop and validate the STEE tool through a panel of experts using the Delphi method. The item weights will be determined with the analytic hierarchy process technique. The third phase will involve implementation and evaluation of the STEE tool with an exploratory, nonexperimental, and comparative analysis. Descriptive and inferential statistical analyses will be performed with SPSS version 23.

Results

The STEE tool for newly graduated registered nurses in the operating room will be useful for evaluating training effectiveness during standardized training. The results obtained with this tool will clarify the effectiveness of training, thereby helping transform nursing students into competent nurses.

Conclusion

In this way, this study will provide practical guidance for improving standardized training programs and help newly graduated nurses manage their transition to the clinical work environment and remain in their posts.

Employing students' evaluations and tutors' perceptions to evaluate a faculty development program on problem-based learning at the Faculty of Medicine, King Abdulaziz University

BACKGROUND

Faculty development programs are crucial for promoting continuous learning, enhancing teaching effectiveness, and encouraging professional growth among medical educators. Problem-based learning was introduced as a teaching strategy in our Faculty of Medicine in 2007. Thereafter, several rounds of a faculty development program were conducted to help teachers recognize their role as facilitators and assess areas for improvement.

METHODS

We conducted a mixed-methods study with a sample of 284 third-year medical students answering a questionnaire and 21 faculty members participating in focus groups. A validated 13-item questionnaire was used to investigate the students' evaluation of their tutors' performance in problem-based learning. Three sessions were then conducted with faculty members involved in problem-based learning to gain in-depth insights into their experiences and perspectives.

RESULTS

The mean performance ranking for tutors awarded by the students was above halfway. There was a significant positive correlation between tutors' performance ranking and all five of the learning approaches examined herein: constructive/active learning, self-directed learning, contextual learning, collaborative learning, and intra-personal behavior (p < 0.05). The data from the focus groups were analyzed under five broad themes: tutors' insights into their strengths and weaknesses, challenges in conducting problem-based learning, tutors' ways of preparing for problem-based learning, feedback, and suggestions for improving problem-based learning workshops.

CONCLUSIONS

This study recommends improvements and future directions for advanced program evaluation. Faculty development programs can be tailored to effectively address students and faculty members' goals and needs, which can benefit the teaching and learning process and foster a culture of continuous improvement and professional growth.

Supporting migrants and refugees with posttraumatic stress disorder: development, pilot implementation, and pilot evaluation of a continuing interprofessional education for healthcare providers

BACKGROUND

Refugees and migrants face an increased risk of developing posttraumatic stress disorder (PTSD). Adequate care can be insufficient due to language barriers, cultural differences, and knowledge deficits of health service providers. Therefore, professional associations requested that healthcare providers to be educated to provide culturally sensitive care. An evidence-based educational intervention in the form of a continuing interprofessional education (CIPE) for healthcare providers on the topic of PTSD in migrants and refugees was developed, pilot-implemented, and evaluated according to the first two levels of the Kirkpatrick evaluation model (reaction and learning).

METHODS

The development of a curriculum for the CIPE intervention was based on a narrative literature review. Its content was validated by experts (N = 17) in an online survey and analyzed using both the Content Validity Index and a thematic analysis. The evaluation of the CIPE intervention was performed by conducting a pilot study with a quasi-experimental single group, using a pre-posttest design. In total, there were 39 participants distributed among three pilot courses. We collected and analyzed data on satisfaction, knowledge, and feasibility.

RESULTS

The curriculum for a half-day course, consisting of 8 modules, showed almost excellent content validity (S-CVI = 0.92). In the pilot-implementation phase, participants were "very satisfied" with the pilot courses and a positive effect on their knowledge was detected. No correlation between satisfaction and knowledge gain was found.

CONCLUSIONS

The CIPE intervention can be considered feasible and seems promising in its effects on satisfaction and knowledge. The insights gained in this study can be used to adapt and optimize the educational intervention, whereby the feedback from course attendees is particularly useful. Future studies need to further examine the effects in larger samples and more robust study designs.

Learning Outcomes of e-Learning in Psychotherapy Training and Comparison With Conventional Training Methods: Systematic Review

BACKGROUND

Mental disorders pose a major public health problem in most western countries. The demand for services for common mental health disorders has been on the rise despite the widespread accessibility of medication. Especially, the supply and demand for evidence-based psychotherapy do not align. Large-scale increase of modern psychotherapy is difficult with current methods of training which are often expensive, time consuming, and dependent on a small number of top-level professionals as trainers. E-learning has been proposed to enhance psychotherapy training accessibility, quality, and scalability.

OBJECTIVE

This systematic review aims to provide an overview of the current evidence regarding e-learning in psychotherapy training. In particular, the review examines the usability, acceptability, and learning outcomes associated with e-learning. Learning outcomes are assessed in different modalities including trainee experiences, knowledge acquisition, skill acquisition, and application of trained content in daily practice. Furthermore, the equivalence of web-based training and conventional training methods is evaluated.

METHODS

Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, a search from Ovid, MEDLINE, PsycINFO, and Scopus databases between 2008 and June 2022 was conducted. Inclusion criteria required studies to describe e-learning systems for psychotherapy training and assess acceptability, feasibility, or learning outcomes. The risk of bias was evaluated for both randomized and nonrandomized studies. Learning outcomes were categorized using the Kirkpatrick model. Effect sizes comparing e-learning and traditional methods were calculated.

RESULTS

The search yielded 3380 publications, of which 34 fulfilled the inclusion criteria. Positive learning outcomes are generally associated with various e-learning programs in psychotherapy training including trainee satisfaction, knowledge, and skill acquisition, and in application of trained content in clinical practice. Learning outcomes generally show equivalence between e-learning and conventional training methods. The overall effect size, indicating this disparity, was 0.01, suggesting no significant difference. This literature displays a high level of heterogeneity in e-learning solutions and assessment methods.

CONCLUSIONS

e-Learning seems to have good potential to enhance psychotherapy training by increasing access, scalability, and cost-effectiveness while maintaining quality in terms of learning outcomes. Results are congruent with findings related to e-learning in health education in general where e-learning as a pedagogy is linked to an opportunity to carry out learner-centric practices. Recommendations for conducting psychotherapy training programs in blended settings supported by activating learning methods are presented. However, due to the heterogeneity and limitations in the existing literature, further research is necessary to replicate these findings and to establish global standards for e-learning, as well as for the assessment of training outcomes in psychotherapy education. Research is especially needed on the effects of training on patient outcomes and optimal ways to combine e-learning and conventional training methods in blended learning settings.

[Examination of the effects of a role-playing training program for the improvement of planning and management skills of public health nurses]

Objectives　The Japan Health Insurance Association (JHIA) conducts training in each branch facility to improve the ability of public health nurses (PHNs). The headquarters of the PHNs of JHIA and the researchers conducted a training program for leaders of PHNs at each JHIA branch. The goal of the program was to create a training plan using role-play to acquire facilitation skills. This study aimed to examine the effects of training.Methods　The study was designed in accordance with the Instructional Designs. The training goals were as follows: (1) understanding the role of facilitation and the facilitator in the debriefing session after the role-play, (2) understanding facilitation techniques, (3) being confident in performing as facilitators in the debriefing session; and (4) conducting the debriefing sessions using facilitation skills. The evaluation of the training was based on the Kirkpatrick model from the viewpoint of confidence in using facilitation, knowledge of facilitation, and conduct of training and utilization of facilitation technology. Questionnaire evaluations were conducted three times before the training, immediately after the training, and three months after the training. In August 2016, 4.5 hours of training were conducted in one day.Results　There were 79 participants in the training group. The mean points of knowledge and confidence were 2.6-3.6 before training, 6.3-7.9 after training, and 6.0-6.9 at 3 months after training. The participants rated their interest in three questions of the training as high as 8.1-8.6. In addition, 64.6% of participants held a role-play session at each branch within three months of the initial training. In the role-play session planned by the participants, the practitioners implemented the explanation of the purpose and the rules at each branch 96.1% and 98.0%, respectively. Participants who had attended facilitation training prior to our program scored higher points of knowledge and confidence before and after three months. Three months after the training, 79 participants responded to the question of the role of the facilitator in role-playing. The descriptions were categorized into "opinions on the basics and planning of role-play training" and "opinions on the roles in retrospectives."Conclusion　The participants evaluated the training contents and the materials used as appropriate, and their knowledge and confidence in facilitation improved after the training. To maintain and improve facilitation skills, the educational system needs to implement training using role-playing at each branch consistently.

Impact of women's feedback on midwifery student learning: A thematic analysis of students' reflections

PROBLEM

Little is known about the educational impact of providing routine, online feedback from women on midwifery student learning and clinical practice.

BACKGROUND

Feedback on students' clinical performance has historically been provided by lecturers and clinical supervisors. Women's feedback is not routinely collected or evaluated for impact on student learning.

AIM

To evaluate the impact of women's feedback about continuity of care experiences with a midwifery student on learning and practice.

DESIGN

Descriptive, exploratory qualitative study.

METHODS

All second-and third-year Bachelor of Midwifery students undertaking clinical placement between February and June 2022 at one Australian university, submitted formative, guided written reflections on de-identified women's feedback they received through their ePortfolio. Data were analysed using reflexive thematic analysis.

FINDINGS

Forty-four of the 69 eligible students (64%) submitted reflections on feedback received. Three themes emerged: 1) Confidence boosting, 2) Deeply integrating Midwifery Metavalues, and 3) Enhancing commitment to continuity. Three subthemes: connection, future practice and advocacy were identified. Women's feedback positively impacts student learning and places the woman in the educational feedback loop.

CONCLUSION

This study is an international first evaluating the impact of feedback from women on midwifery students' learning. Students reported greater confidence in their clinical practice, a deeper understanding of their midwifery philosophy, and an intention to advocate for, and work in, midwifery continuity models after graduation. Routine feedback about women's experiences should be embedded into midwifery education programs.

Impact Assessment of Patient Experience Capacity-Building Program Using Kirkpatrick Model for Program Evaluation at a Regional Healthcare System

Introduction

The patient experience (PX) is central to improving the quality of healthcare services. We launched a PX capacity- and capability-building program at the Armed Forces Hospitals Taif Region, which is integral to our regional healthcare cluster transformation plans and is an initial step toward developing a culture of improvement in human experience in healthcare.

Methods

A multidisciplinary PX committee recruited five frontline interprofessional PX heads, one from each of our regional healthcare hospitals. The Kirkpatrick model for program evaluation was used to assess the impact on four key levels: reaction, learning, behavior, and results. A pre-program competency assessment was conducted to evaluate the level of expertise across various PX competencies, and a program curriculum was developed accordingly. Participants underwent an intensive workshop-based PX capacity-building training program. A post-program competency assessment was performed along with a post-program survey. The PX-related activities led by interprofessional frontline PX heads at their respective hospitals' post-programs were tracked. The impact on the regional PX mean score across various settings, including inpatient, outpatient, and emergency settings, was measured using Press Ganey PX surveys. Our work is reported in accordance with the SQUIRE-EDU guidelines of the EQUATOR network.

Results

The PX capacity-building program led to a significant improvement in participants' expertise across various PX competencies. Significant improvements beyond the strategic targets were observed in the PX mean score in inpatient departments pre-program (83.31) vs. post-program (86.34), with a p-value of < 0.001 across the regional healthcare system.

Conclusion

The PX capacity-building program is a first step toward major cultural change amid the healthcare cluster transformation in our regional healthcare system. The Kirkpatrick model helps evaluate the impact of PX capacity- and capability-building training programs comprehensively through an organizational approach. Sustainable improvements in PX over a long period through a capacity-building program alone remain challenging.

The effect of the Educational Scholar Program as a longitudinal faculty development program on the capability of educators as scholars

INTRODUCTION

The Educational Scholar Program (ESP) is designed and implemented as a longitudinal and institution-based faculty development program. The present study aimed to assess the effect of the ESP on educators' capabilities to undertake SoTL activities associated with their scholar role.

METHODS

This study was conducted from 2017 to 2022. The participants (n = 64) were educators in six schools of Shahid Sadoughi University of Medical Sciences. The ESP was a faculty development program that consisted of training and project-based stages. The educators experienced small-group learning, self-directed learning, and reflective assignments in the training stage. In the second stage, the educators completed a SoTL (Scholarship of Teaching and Learning) project. Learner-related outcomes based on Kirkpatrick model was assessed. The reaction of educators (satisfaction, active participation in the ESP, and the perception of mentoring sessions) was assessed by three questionnaires (Reaction level). The educators' learning was evaluated by modified essay questions and their project reports (Learning and Behavior levels). Outputs of the ESP including journal publications, abstracts presented at meetings or congresses, grant funding, awards in educational festivals, promotions, projects with ongoing implementation following the ESP, and conducting further SoTL projects after ESP were assessed quantitatively over two years after participating in the ESP (Results level). Data were summarized by descriptive statistics (mean, percentage, SD, 95% Confidence Interval (CI)). Cut-off scores of the instruments was calculated with a standard setting method which introduced by Cohen-Schotanus and Van DerVleuten. Data analyzed by One-sample t-test.

RESULTS

Sixty-four of 72 (89%) educators completed the ESP. The mean (CI) satisfaction score of educators was 42 (CI: 26.92-58.28), the active participation was 92 (CI: 80.24-103.76). The scores of the mentoring assessment from the perspective of the educators were reported at 90 (CI: 78.24- 101.76). The mean (95%CI) learning scores in the essay examination were 88 (CI: 70.36- 105.64), and project assessment were 90 (CI: 78.24- 101.76). The results showed the educators' scores in reaction and learning significantly higher than the cut-off scores. (P < 0.05). Most projects were conducted in curriculum development and assessment/evaluation domains. The number of projects with ongoing implementation over the two years following the ESP and the acquisition of grants was higher than other outputs in the results level.

CONCLUSION

The ESP, as an institute-based longitudinal program, enhanced the learner-related outcomes (in four levels of reaction, learning, behavior, and results). The creation of practical learning and supportive mechanisms influenced on the results. The outcomes of ESP indicated that the educators prepared to conduct SoTL activities in their educational community.

VTE Prevention Ability Among Community Nurses in the Medical Consortium Based on the Kirkpatrick Mode Evaluation of the Effects of Training

PURPOSE

To explore the application of the Kirkpatrick model in VTE prevention training among community nurses in the medical consortium.

METHOD

A team of experts was established to formulate a VTE prevention ability training program for community nurses in the medical consortium. According to a literature review, the results of a questionnaire survey and the results of on-site supervision surveys of nurses in five community health service centers in the medical consortium. The Kirkpatrick model was applied to train 117 nurses in five community service centers, and the effects on the four dimensions (i.e., the reaction layer, learning layer, behavior layer, and results layer) were observed.

RESULTS

A total of 100% of nurses' reported satisfaction with the training, and the attendance rate was ≥98%. After the training, the theoretical knowledge of VTE prevention and the results of the skills assessment of community nurses in the medical consortium were significantly greater than the pretraining levels (p < 0.001). Three months after training, the nurses' VTE prevention execution score significantly improved compared with the pretraining scores (p < 0.001). The compliance rate and standard rate of ankle pump movement significantly improved compared with those before training (p < 0.001).

CONCLUSION

Training based on the offset model can effectively improve VTE prevention knowledge among community nurses in the medical consortium, thereby enhancing VTE prevention among patients and at improving the compliance and standardization of patients' self-prevention. These findings provide a reference for future in-service training programs targeting VTE prevention among nurses in community hospitals.

The Asian Sarcoma Consortium Sarcoma Preceptorship Program: A Program Evaluation Study Utilizing the Kirkpatrick Model (Level 1 and 2)

Background

Sarcomas, a heterogenous group of malignancies consisting of tumors, are rarely encountered in most general oncology clinics. As a result, most general oncologists have very little experience in dealing with these tumors and resulting in poorer treatment and survival outcomes. Having successful collaborative efforts in research, the Asian Sarcoma Consortium (ASC) pioneered a Sarcoma Preceptorship Program with its main goal of increasing awareness of the multidisciplinary approach in the management of sarcomas.

Objective

The objective of this study is to evaluate the ASC Sarcoma Preceptorship Program using the first 2 levels of the Kirkpatrick Evaluation Model: Level 1) Reaction - through using satisfaction surveys; and Level 2) Knowledge - focusing on the amount of information gained from the activity evaluated by test-retest method.

Methods

The 2-day preceptorship program held in Singapore utilized educational activities such as didactic lectures, multidisciplinary case discussions, and case presentations. The program was evaluated using: 1) pre-test and post-test; 2) satisfaction survey from ASC; and 3) satisfaction survey made by the sponsor, Novartis Pharma.

Results

The preceptorship program enrolled 30 participants, most from the Asia-Pacific region. The overall results from the satisfaction surveys were generally optimistic, with results showing high levels of satisfaction with regard to the time allotment, scope of topics, and how helpful each session was. Test results showed that participants scored lower in post-test as compared to the pre-test. Computation of the Coefficient of Test reliability showed that although the pre-test was highly reliable, this was not true of the post-test.

Conclusions

Based on the Level 1 evaluation, the program was successful in terms of both participant and faculty satisfaction. However, certain areas for improvement were identified and recommendations were made in order to effect improvements for future iterations of this preceptorship.

A Clinically Immersive Medical Innovation Program for US MD Students: Curricular Description and Program Outcomes

Background

There is growing need for physician-innovators to address the mounting challenges within the US healthcare system. Despite this, there remains a significant gap in understanding of the efficacy of innovation programs for US MD candidates. We present initial program outcomes of a novel, clinically immersive medical innovation program offered to MD candidates at the David Geffen School of Medicine (DGSOM) at UCLA.

Methods

A novel clinically immersive medical innovation curriculum was developed based on existing and reputable medical innovation frameworks and tailored for medical students. Curricular topics broadly included clinical ethnography, interviewing techniques, mind mapping, needs formulation and prioritization, quality improvement, intellectual property, reimbursement pathways, solution landscaping and prioritization, regulatory processes. The program was trialed during an unscheduled summer with voluntary enrollees from DGSOM Class of 2024. The traditional four-level Kirkpatrick model was employed to assess program outcomes.

Results

Program outcomes were positive on all four Kirkpatrick levels. Students rated enjoyment at 9.5/10 for lectures and 9.1/10 for clinical immersion. Student-perceived confidence in key skills increased by 43%, and 75% of faculty directly perceived improvement in ethnographic skills. Students were highly engaged in both didactics and clinical immersion, discovering on average 2.6 faculty-verified needs per week. Faculty largely felt their students discovered important unmet clinical needs and added value to their clinical practice.

Conclusion

We developed and trialed a novel clinically immersive medical innovation curriculum tailored for medical students. This program achieved positive outcomes on all four levels of the Kirkpatrick model. Our findings have driven the local adoption of this program into our institution's medical school curriculum. We hope that the program efficacy demonstrated herein catalyzes more institutions to trial similar medical innovation programs.

Evaluation of the effectiveness of rehabilitation nursing training for clinical nurses based on the Kirkpatrick model

BACKGROUND

According to the "National Nursing Development Plan (2021-2025)" issued by the National Health Commission of China, it was crucial to address the limited research available on rehabilitation nursing training. By 2025, it was expected that at least 90% of urgently needed professional nurses, including rehabilitation nursing, would participate in training. Therefore, it was necessary to explore the effects of rehabilitation nursing training for clinical nurses and provide reference suggestions for rehabilitation nursing training.

METHODS

The Kirkpatrick model was used as an evaluation tool to conduct a questionnaire survey of 79 clinical nurses who participated in the 5th term of rehabilitation nursing training in Shandong Province. Paired t tests, factor analysis, and qualitative interviews were used to analyze the participants' responses, learning styles, behaviors, and results. Four levels were used to evaluate training effectiveness.

RESULTS

The overall satisfaction rate of the trainees was high. There were significant differences in theoretical scores, operational examination scores and related nursing abilities before and after training; these scores before training were significantly lower than scores after training (P < 0.05). Four themes were extracted from the semi structured interviews: improving relevant capabilities, optimizing training, improving safety management awareness and crisis awareness, and expanding career development space.

CONCLUSIONS

Rehabilitation nursing training significantly improved the rehabilitation skills and knowledge of clinical nurses. Therefore, it was recommended that the methods and programs used in this study be adopted for the rehabilitation nursing training of clinical nurses in all specialty areas of nursing to improve their nursing skills.