The Kirkpatrick model for training evaluation: bibliometric analysis after 60 years (1959–2020)

Blended learning compared to traditional learning for the acquisition of competencies in oral surgery by dental students: A randomized controlled trial

OBJECTIVE

To determine whether blended learning results in better educational outcomes compared to traditional learning in the acquisition of oral surgery technical skills for 4th-year undergraduate dental students.

MATERIALS AND METHODS

Seventy-three students participated in this two-arm parallel randomized controlled trial. Only students in the blended learning group had access to the online preparation platform for oral surgery practical work (PW) on a pig's jaw and to the debriefing. Kirkpatrick's four-level model was used to assess the educational outcomes directly after (levels 1 and 2) and 6 months later, after the start of the students' clinical activity (levels 3 and 4).

RESULTS

For level 1, higher global satisfaction scores were found for students in the blended learning compared to the traditional learning group (p = .002). For level 2, blended learning resulted in an increase in knowledge score (p < .01), comparable to that observed in the traditional learning group. For level 3, students in the blended group made more progress in 6 months than those in the traditional group in terms of feeling able to assess and perform anaesthesia (p = .040) and surgical tooth extraction (p = .043). No difference in level 4 was found for the 6-month clinical surgical activity between groups, but students in the blended group felt more able to assess and perform the surgical management of a failed extraction requiring bone removal (p = .044).

CONCLUSION

Blended learning for oral surgery PW had a positive impact on three of the four Kirkpatrick levels (level 1, 3 and 4). Efforts should focus on the procedures that are perceived as the most difficult.

Exploring self-experience practices in dementia care: A scoping review

BACKGROUND

Recognised as essential for high-quality dementia service, person-centred care aims to understand and respect the unique needs of each individual. Self-experience practices may offer caregivers an opportunity to acquire knowledge, empathy, and skills related to person-centred care, especially through recreating experiences similar to dementia. Given the need to enhance the understanding of self-experience practices in dementia care, a more comprehensive investigation of these training interventions for (future) caregivers is needed.

METHODS

We conducted a scoping review to map the evidence on the use of self-experience practices in dementia training. We systematically searched Cochrane Library, MEDLINE via PubMed, CINAHL, and Web of Science. We also searched for grey literature, as well as registry entries, and conducted backward citation tracking of included reviews. We analysed data on intervention characteristics, factors influencing the implementation, and learning outcomes based on Kirkpatrick's model.

RESULTS

We included 44 reports across 30 intervention programmes. The majority of reports (91%) were published from 2016 onwards, with 32% originating from the USA and 25% from the UK. We identified passive, interactive, immersive, and multicomponent self-experience interventions in dementia education and training. Learning outcomes based on Kirkpatrick's model were fairly distributed across all identified modalities. Both consumers and providers emphasised aspects related to the development and implementation of practices, particularly organisational-related considerations such as temporal and spatial planning of trainings.

CONCLUSIONS

Our review highlights diverse interventions incorporating self-experience practices, with an increasing role for technological tools. While self-experience interventions engage participants, the impact on individuals with dementia and organisational levels remain largely unreported. Our overview, informed by current literature, underscores unique considerations and challenges associated with dementia-related self-experience practices. Implementing and evaluating complex training interventions using self-experience practices should consider ethical aspects.

TRIAL REGISTRATION

Registry: Registered within the Open Science Framework (available at https://osf.io/fycxa/).

Educating for supported decision making and shared decision making: a scoping review of educational design and outcomes for education and training interventions

PURPOSE

To characterise existing knowledge about the design and learning outcomes of education and training programs for supported or shared decision making.

MATERIALS AND METHODS

A scoping review was performed to identify academic and grey literature, published between January 2006 and February 2022, that reported on the design and/or learning outcomes of supported or shared decision making education or training programs. Eligible literature was mapped across domains of educational design and Kirkpatrick's hierarchy of learning effectiveness, and then qualitatively synthesised using cross-case analysis.

RESULTS

A total of 33 articles were identified (n = 7 for supported decision making and n = 26 for shared decision making) that provided education or training to supporters of persons with mental illness or substance use disorders (n = 14), dementia or neurocognitive disorders (n = 6), cognitive disability (n = 5), mixed populations (n = 1), and those receiving end-of-life care (n = 7). In their design, most programs sought specific changes in practice (behaviour) via experiential learning. Reported educational outcomes also focused on supporter behaviour, with limited evidence for how changes in learner attitudes, skills, or knowledge might be contributing to changes in supporter behaviour.

CONCLUSIONS

Future education and training would benefit from a closer engagement with theories of teaching and learning, particularly those oriented towards co-design.

Evaluating the impact of interprofessional training wards on patient satisfaction and clinical outcomes: a mixed-methods analysis

Introduction

Interprofessional teamwork is pivotal in modern healthcare, prompting the establishment of interprofessional training wards since 1996. While these wards serve as hubs for optimizing healthcare professional collaboration and communication, research into patient outcomes remains notably sparse and geographically limited, predominantly examining patient satisfaction and sparingly exploring other metrics like mortality or self-discharge rates. This study seeks to bridge this gap, comparing patient outcomes in interprofessional training wards and conventional wards under the hypothesis that the former offers no disadvantage to patient outcomes.

Materials and methods

We explored patient outcomes within an interprofessional student ward called A-STAR at a University Hospital from October 2019 to December 2022. Engaging with patients discharged between May 2021 and April 2022, we utilized digital and paper-based anonymous questionnaires, catering to patient preference, to gather pertinent data.

Results

Analysis of outcomes for 1,482 A-STAR (interprofessional student ward) and 5,752 conventional ward patients revealed noteworthy findings. A-STAR patients tended to be younger (59 vs. 61 years, p < 0.01) and more frequently male (73.5% vs. 70.4%, p = 0.025). Vital clinical outcomes, such as discharges against medical advice, complication-driven readmissions, and ICU transfers, were statistically similar between groups, as were mortality rates (1.2% vs. 1.3%, p = 0.468). A-STAR demonstrated high patient satisfaction, underscored by positive reflections on team competence, ward atmosphere, and responsiveness to concerns, emphasizing the value placed on interprofessional collaboration. Patient narratives commended team kindness, lucid explanations, and proactive involvement.

Discussion

This data collectively underscores the safety and reliability of patient care within training wards, affirming that patients can trust the care provided in these settings. Patients on the interprofessional ward demonstrated high satisfaction levels: 96.7% appreciated the atmosphere and conduct of ward rounds. In comparison, 98.3% were satisfied with the discussion and information about their treatment during their hospital stay.

Effectiveness of online faculty calibration: A randomized crossover study

PURPOSE

Dental hygiene faculty members must be able to provide evidence of skill calibration for clinical evaluation of students. The purpose of this study was to evaluate the effectiveness of online instructional videos as compared to in-person sessions for faculty calibration.

METHODS

A randomised crossover pretest/posttest research design was used to evaluate online and in-person faculty calibration activities. Fifteen faculty members from a baccalaureate dental hygiene programme were randomly assigned to an AB or BA sequence for calibration sessions on two different instruments. Following a 2-week washout period, the groups switched activity modalities. A pretest, posttest and retention test, administered 10 weeks following the activity, were administered to determine learning levels and the new and retained knowledge. A 7-point Likert scale questionnaire evaluated the reaction to and impact of the calibration activities. Descriptive statistics analysed demographic and Likert scale data. Paired samples t-tests were used to analyse the research questions (p ≤ 0.05).

RESULTS

Online calibration activities yielded higher posttest scores than in-person activities (p = 0.01). Findings related to feelings of confidence revealed that a greater percentage of participants agreed that online calibration activities increased their ability to evaluate student performance. Findings related to feelings of preparedness supported equal percentages of participants who agreed the online and in-person activities increased their ability to teach dental hygiene instrumentation. There was no significant difference between in-person and online retention test scores (p = 0.235).

CONCLUSION

Faculty members agreed that both online and in-person calibration activities were an effective use of their time and contributed to greater feelings of confidence and preparedness. However, the online calibration activities seemed to be more effective at increasing calibration on instrumentation. More research is needed to determine additional effective strategies for the online calibration of clinical faculty.

Teaching (remotely) to communicate (remotely) with relatives of patients during lockdown due to the COVID-19 pandemic

Aim

In 2020, due to the pandemic, the lack of specific knowledge on breaking bad news over the phone became apparent. This study aims at assessing the differences, or lack thereof, in satisfaction reported by participants in the different formats of a course in telephone communication for breaking bad news to families of patients, developed based on the previous experience of a team dedicated to this task during the peak of the pandemic.

Methods

Four courses were delivered, two in a fully streamed format and two in a blended format, part pre-recorded, part streamed. There were 41 attendants, mostly doctors, but also nurses, social workers, occupational therapists, and administrative staff who deal with families. Subsequently, a survey was conducted to assess the degree of satisfaction of the participants.

Results

Both formats scored very positively, with small advantages for the fully streamed format, mainly due to the difference in interaction activities.

Conclusions

The main conclusion is the need for these courses, which are demanded by the professionals themselves, without forgetting the benefit obtained from interprofessional education that enriches interaction and learning.

Innovation

The inclusion of administrative staff allows for a global vision of care for family members, which improves it.

The ‘Rich Pictures’ Method: Its Use and Value, and the Implications for HRD Research and Practice

Embracing new and innovative qualitative methods has helped researchers in a number of fields to access aspects of the lived experience that traditional methods cannot easily reach. This paper explores the use and value of one such method – ‘rich pictures’: a technique whose origins lie in ‘soft systems’ engineering but which has been successfully applied in a broader range of contexts in recent years including health, medicine and education. Despite its use in these disciplines, however, recent studies suggest that HRD research continues to rely on established methods and that uptake of visual methods – and ‘rich pictures’ in particular – is virtually non-existent. The aim of this paper therefore is to shed light on this underused method and encourage HRD researchers to recognize its potential for studying human development.

The Effectiveness of a Blended Learning-Based Life Design Course: Implications of Instruction and Application of Technology

Purpose

Due to the outbreak of the COVID-19 pandemic in 2021 in Taiwan, we have adapted the face-to-face Life Design course to a blended learning approach with educational technology to cope with the problem of cross-generational confusion and anxiety towards later life from learners.The objectives of this study are to.evaluate learners' reactions after attending the Life Design course including their level of satisfaction, engagement (Level 1), and the applicability of courses in their life.evaluate their learning outcomes after attending the Life Design course, such as their acquisition of knowledge, skills, attitudes, confidence, commitment (Level 2), and behavioral changes (Level 3).explore the factors that enable and prevent students from transferring their learning in this course to the action taking and making behavioral changes.explore how the application of educational technology can enhance the teaching and learning in the Life Design course.

Methods

This study used an action research method to solve two main problems we identified in practice: students' confusion about their future life and the shortcomings of traditional teaching methods, which cannot meet the learning needs of this type of course due to the requirement of intensive personal reflection and self-disclosure. Participants were 36 master's students who completed the Life Design course. Based on this course's design, implementation, and evaluation, we used the new Kirkpatrick Learning Assessment Model (Kirkpatrick J, Kirkpatrick WK. An introduction to the new world Kirkpatrick Model. Kirkpatrick Partners, 2021) to analyze the learning effectiveness on the Reaction, Learning, and Behavioral levels.

Results

To facilitate learners to overcome the cross-generational confusion of their life design and solve the shortage of face-to-face teaching methods, we took biographical learning as the core theme for this Life Design course and designed online and offline learning activities. The blended learning approach with educational technology allowed us to go beyond time and location constraints and provide a holistic and inseparable learning experience in both formats. The result of the evaluation shows that students who took the Life Design course were highly satisfied with the overall course design, topics, and the suitability of a blended learning approach, which motivated them to extend their learning outside the classroom and helped them benefit from a more trusted, personal and hybrid interaction with teachers and their peers both online and offline. On the learning level, students not only learned the correct knowledge of age perspectives, changed their views of career and personal development, and acquired skills for life design, but were also confident and committed to applying what they have learned in their future life. After the course, many students applied and integrated the learning into behavioral changes in their life. In terms of the difficulties and hindrances encountered in action taking, many students mentioned the lack of peers' support and constraints from their busy daily life. Many suggested providing extra support after the course with regular impulse, follow-ups, and individual feedback from teachers and peers in an online learning community. This indicates how educational technology can better support these elements in continuous learning and the transfer of learning.

Conclusion

Based on these results, we affirm that implementing this Life Design course with a blended learning approach is indeed better than a fully physical course. However, the focus of a blended learning approach should be on learners from a pedagogical perspective rather than technology.

Impact of an Aboriginal and Torres Strait Islander brief intervention training program on health staff participants' own health behaviours: smoking, nutrition and physical activity

ISSUE

Little research has been conducted on the impact of Aboriginal and Torres Strait Islander brief intervention training programs on health staff participants' own health behaviours. Through the Queensland B.strong program (2017-2020), brief intervention training in smoking cessation, nutrition and physical activity was provided to the Aboriginal and Torres Strait Islander health workforce and other health and community professionals. This study examined the program's impact on participants' own health behaviours.

METHODS

Data were collected through four surveys (pre-and post-training workshop, and 3-month and 6-month follow-up) of the 1,131 participants in B.strong training workshops from June 2017 to August 2019. Surveys included items on participants' own health behaviours. Pre- and post-workshop surveys were paper-based, and follow-up surveys were completed online. For the analysis of data reported in this paper, paired-samples t-tests were used to assess changes between pre-workshop and 3-month follow-up.

RESULTS

Statistically significant improvements were found between pre-workshop and 3-month follow-up in the number of serves of vegetables or legumes/beans eaten per day, the number of serves of fruit eaten per day, and in time spent in physical activity. However, there was: no statistically significant change in smoking status, with baseline rates being relatively low; a statistically significant increase in consumption of sugary drinks, and of take-away foods; a non-significant increase in consumption of snack foods; and no significant change in sedentary behaviour of participants.

CONCLUSION

While some positive changes in participants' own health behaviours in nutrition and physical activity were associated with the B.strong program, there was no change in their smoking behaviour.

SO WHAT

This study found that some improvements in participants' own health behaviours were associated with the B.strong program. This research may inform future Indigenous brief intervention training programs and health services on how to promote healthy behaviours for health staff themselves.