Evaluation in health professions education-Is measuring outcomes enough?

Reducing epilepsy diagnostic and treatment gaps: Standardized paediatric epilepsy training courses for health care professionals

AIM

To evaluate improvement in knowledge and clinical behaviour among healthcare professionals after attendance at paediatric epilepsy training (PET) courses.

METHOD

Since 2005, 1-day PET courses have taught evidence-based paediatric epilepsy management to doctors and nurses in low-, middle-, and high-income countries. A cohort study was performed of 7528 participants attending 252 1-day PET courses between 2005 and 2020 in 17 low-, middle-, and high-income countries, and which gathered data from participants immediately after the course and then 6 months later. Training outcomes were measured prospectively in three domains (reaction, learning, and behaviour) using a mixed-methods approach involving a feedback questionnaire, a knowledge quiz before and after the course, and a 6-month survey.

RESULTS

Ninety-eight per cent (7217 of 7395) of participants rated the course as excellent or good. Participants demonstrated knowledge gain, answering a significantly higher proportion of questions correctly after the course compared to before the course (88% [47 883 of 54 196], correct answers/all quiz answers, vs 75% [40 424 of 54 196]; p < 0.001). Most survey responders reported that the course had improved their epilepsy diagnosis and management (73% [311 of 425]), clinical service (68% [290 of 427]), and local epilepsy training (68% [290 of 427]).

INTERPRETATION

This was the largest evaluation of a global epilepsy training course. Participants reported high course satisfaction, showed knowledge gain, and described improvements in clinical behaviour 6 months later. PET supports the global reduction in the epilepsy 'treatment gap' as promoted by the World Health Organization.

Qualitative evaluation of a multidisciplinary master of cancer sciences: impacts on graduates and influencing curricular factors

BACKGROUND

Evaluations of continuing professional development programs typically focus on short-term knowledge and skill acquisition. There is a need for more comprehensive program evaluation methods that assess a broader range of impacts and can elicit how and why these outcomes occurred. We conducted a qualitative study to investigate the impacts of a multidisciplinary, online health professional postgraduate degree and to gain insights into the factors that led to these impacts.

METHODS

Participants were graduates of the University of Melbourne's Master of Cancer Sciences who could participate in an online interview. Semi-structured, qualitative interviews were conducted exploring a broad range of impacts, including changes in professional practice and career trajectory since graduation, and how the degree influenced these impacts. Data were analysed inductively.

RESULTS

Fifteen participants (female: 80%, 31-50 years old: 67%) from a range of professions were interviewed. A number of major themes were uncovered. Impacts on career trajectory included expanded career horizons (e.g. increased role diversity and complexity), and increased confidence in their professional identity. Impacts on professional practice included individual improvements in patient care and research, as well as changes in organisational practice. Factors identified as leading to these impacts were: (i) active, interactive and interprofessional learning; (ii) networking, informal mentoring, and role-modelling; and (iii) support at multiple levels.

CONCLUSION

This study provides preliminary evidence of the positive impact of a Master of Cancer Sciences on graduate career trajectory and professional practice. In addition, the inductive methodology enabled identification of the curricular features (both planned and emergent) that influenced these impacts, facilitating potential transferability of learnings to other teaching programs.

Training as imagined? A critical realist analysis of Scotland's internal medicine simulation programme

BACKGROUND

Evaluating the impact of simulation-based education (SBE) has prioritised demonstrating a causal link to improved patient outcomes. Recent calls herald a move away from looking for causation to understanding 'what else happened'. Inspired by Shorrock's varieties of human work from patient safety literature, this study draws on the concept of work-as-done versus work-as-imagined. Applying this to SBE recognises that some training impacts will be unexpected, and the realities of training will never be quite as imagined. This study takes a critical realist stance to explore the experience and consequences, intended and unintended, of the internal medicine training (IMT) simulation programme in Scotland, to better understand 'training-as-done'.

METHODS

Critical realism accepts that there is a reality to uncover but acknowledges that our knowledge of reality is inevitably our construction and cannot be truly objective. The IMT simulation programme involves three courses over a 3-year period: a 3-day boot camp, a skills day and a 2-day registrar-ready course. Following ethical approval, interviews were conducted with trainees who had completed all courses, as well as faculty and stakeholders both immersed in and distant from course delivery. Interviews were audio-recorded, transcribed verbatim and analysed using critical realist analysis, influenced by Shorrock's proxies for work-as-done.

RESULTS

Between July and December 2023, 24 interviews were conducted with ten trainees, eight faculty members and six stakeholders. Data described proxies for training-as-done within three broad categories: design, experience and impact. Proxies for training design included training-as-prescribed, training-as-desired and training-as-prioritised which compete to produce training-as-standardised. Experience included training-as-anticipated with pre-simulation anxiety and training-as-unintended with the valued opportunity for social comparison as well as a sense of identity and social cohesion. The impact reached beyond the individual trainee with faculty development and inspiration for other training ventures.

CONCLUSION

Our findings highlight unintended consequences of SBE such as social comparison and feeling 'valued as a trainee, valued as a person'. It sheds light on the fear of simulation, reinforcing the importance of psychological safety. A critical realist approach illuminated the 'bigger picture', revealing insights and underlying mechanisms that allow this study to present a new framework for conceptualising training evaluation.

'Good' evaluation: Methodological diversity may be empress, but sound methods remain queen

@Betty_EvalEd argues it's not enough to simply value evaluation methodology as the gauge that indicates when our “needle” is shifting. We must also monitor the quality & operations of the gauge itself.

A guide to outcome evaluation of simulation-based education programmes in low and middle-income countries

Evaluation is a vital part of any learning activity and is essential to optimize and improve educational programmes. It should be considered and prioritized prior to the implementation of any learning activity. However, comprehensive programme evaluation is rarely conducted, and there are numerous barriers to high-quality evaluation. This review provides a framework for conducting outcome evaluation of simulation-based education programmes in low and middle-income countries (LMICs). The basis of evaluation, including core ideas of theory, purpose and structure are outlined, followed by an examination of the levels and healthcare applications of the Kirkpatrick model of evaluation. Then, methods of conducting evaluation of simulation-based education in LMICs are discussed through the lens of a successful surgical simulation programme in Myanmar, a lower-middle-income country. The programme involved the evaluation of 11 courses over 4 years in Myanmar and demonstrated evaluation at the highest level of the Kirkpatrick model. Reviewing this programme provides a bridge between evaluation theory and practical implementation. A range of evaluation methods are outlined, including surveys, interviews, and clinical outcome measurement. The importance of a mixed-methods approach, enabling triangulation of quantitative and qualitative analysis, is highlighted, as are methods of analysing data, including statistical and thematic analysis. Finally, issues and challenges of conducting evaluation are considered, as well as strategies to overcome these barriers. Ultimately, this review informs readers about evaluation theory and methods, grounded in a practical application, to enable other educators in low-resource settings to evaluate their own activities.

Teaching and assessment of communication skills in dietetics: a scoping review

BACKGROUND

Communication is a core element of dietetic practice, and although communication skills are a prominent feature of dietetic curricula, research suggests a need for more consistent approaches. The evidence on how communication skills are taught and assessed in dietetics has not been synthesised leaving uncertainty about best practice. This scoping review aimed to examine and map the research literature relating to the teaching and assessment of communication skills in dietetics.

METHODS

The review was conducted in accordance with the Joanna Briggs Institute (JBI) methodology for scoping reviews. Five electronic databases, two theses databases and eight conference proceedings were systematically searched for research on how communication skills are taught and assessed in dietetics. The search had no geographical or time limits. Studies were independently screened by two authors, summarised thematically using the Kirkpatrick Model and narratively synthesised.

RESULTS

The 45 included studies were organised thematically into three categories: (i) the inclusion of communication skills education in dietetics, suggesting that dietitians have variable experiences; (ii) approaches to teaching communication skills in dietetics, characterised predominantly by a move away from didactic approaches in favour of experiential opportunities for learning such as simulation; and (iii) assessment of communication skills in dietetics, including the use of both dietetic-specific and generic tools.

CONCLUSIONS

The included studies demonstrate that although a variety of teaching strategies are used in the development of communication skills in dietetics, there is potential for more robust evaluation and more diverse approaches to support dietetic educators to prepare the dietetic workforce.

Fetal Cardiology Bioethics: An Innovative New Curriculum for Cardiology Trainees

Decision-making in fetal cardiology is fraught with ethical issues yet education in bioethics for trainees is limited or nonexistent. In this innovation report, we describe the development of a fetal cardiology bioethics curriculum designed to address this gap. The curriculum was developed to supplement the core curriculum for cardiology fellows and fetal cardiology subspecialty trainees. The series combines didactic and interactive teaching modalities and contains 5 key components: (1) introduction to bioethics and its role in fetal cardiology, (2) counseling and pathways for compassionate terminal care, (3) case vignette-based ethical analysis and discussion cases, (4) fetal counseling considerations for shared decision-making and recommendations, (5) facilitated communications role play. The curriculum was refined using session evaluations from end users. This report describes the innovative curriculum as a starting point for further incorporation and study of bioethical education in pediatric cardiology and fetal training programs.

Massive Hemoptysis Simulation Curriculum Improves Performance

BACKGROUND

Massive hemoptysis is a rare, high-acuity presentation, which requires the integration of both cognitive and procedural skills. Simulation has been recommended to improve preparation for high-acuity, low-occurrence procedures; however, the effect of a simulation curriculum for massive hemoptysis management has never been investigated.

RESEARCH QUESTION

Does simulation for hemoptysis management improve competence?

STUDY DESIGN AND METHODS

Kern's six steps for medical education curriculum design were used iteratively to develop a simulation curriculum for the management of massive hemoptysis. Pulmonary and critical care medicine fellows from the University of Colorado participated in a local needs assessment and a massive hemoptysis simulation curriculum. Using a manikin-based massive hemoptysis simulator developed for this curriculum, the simulation session used repetitive practice, clinical variation, a range of difficulties, and directed feedback in a group practice setting. Time to management and performance were assessed for each management attempt; competence was assessed using a combined metric of management-related priorities and global entrustment.

RESULTS

During the needs assessment, fellows viewed massive hemoptysis management skills as important, while expressing their current confidence as low. Nineteen fellows participated in a 90-min case-based hemoptysis simulation during which each was exposed to five different cases and acted as the primary manager for two cases. There was significant improvement in performance from the first to final simulation attempts measured by time to successful management (14.24 vs 10.26 min, P = .0067) and entrustment (Global Assessment Scale, 1 [should not perform] to 5 [independent]; 4.11 vs 4.61; P = .015). Fellow self-assessed knowledge and confidence in hemoptysis management and endobronchial blocker placement improved significantly after the simulation.

INTERPRETATION

Hemoptysis simulation experience improves fellow confidence and skill for management of this high-acuity, low-occurrence presentation.

To prove or improve? Examining how paradoxical tensions shape evaluation practices in accreditation contexts

INTRODUCTION

Although programme evaluation is increasingly routinised across the academic health sciences, there is scant research on the factors that shape the scope and quality of evaluation work in health professions education. Our research addresses this gap, by studying how the context in which evaluation is practised influences the type of evaluation that can be conducted. Focusing on the context of accreditation, we critically examine the types of paradoxical tensions that surface as evaluation-leads consider evaluation ideals or best practices in relation to contextual demands associated with accreditation seeking.

METHODS

Our methods were qualitative and situated within a critical realist paradigm. Study participants were 29 individuals with roles requiring responsibility and oversight on evaluation work. They worked across 4 regions, within 26 academic health science institutions. Data were collected using semi-structured interviews and analysed using framework and matrix analyses.

RESULTS

We identified three overarching themes: (i) absence of collective coherence about evaluation practice, (ii) disempowerment of expertise and (iii) tensions as routine practice. Examples of these latter tensions in evaluation work included (i) resourcing accreditation versus resourcing robust evaluation strategy (performing paradox), (ii) evaluation designs to secure accreditation versus design to spur renewal and transformation (performing-learning paradox) and (iii) public dissemination of evaluation findings versus restricted or selective access (publicising paradox). Sub-themes and illustrative data are presented.

DISCUSSION

Our study demonstrates how the high-stakes context of accreditation seeking surfaces tensions that can risk the quality and credibility of evaluation practices. To mitigate these risks, those who commission or execute evaluation work must be able to identify and reconcile these tensions. We propose strategies that may help optimise the quality of evaluation work alongside accreditation-seeking efforts. Critically, our research highlights the limitations of continually positioning evaluation purely as a method versus as a socio-technical practice that is highly vulnerable to contextual influences.

The rapid growth and expansion of Master of health professions education (MHPE) programs: A mixed-methods study of international curricular trends and guidelines for programs

PURPOSE

This study explores international trends and standards of Master's degree programs through a comprehensive environmental scan and focus group interviews to understand curricular structure, content, program director expectations, educational context, and future directions.

METHOD

Authors conducted a two-phase mixed-methods sequential explanatory design to conduct the environmental scan (phase 1), and subsequently conducting focus groups (phase 2) with program directors. A population list of Master's programs was used to generate a sampling frame, considering the geographic region (continent) and institution type (university, organization, public institution). Qualitative data were coded to analyze the breadth and depth of courses. Three one-hour virtual focus group interviews were conducted with ten program directors.

RESULTS

The population list of 159 Masters programs worldwide was used to create a sample for analysis in the environmental scan (n = 46 Masters programs), representing programs from North America, Europe, Australia, and South Africa. Most programs (39%) delivered their courses online, with 20% exclusively offering an in-person program. Focus group participants indicated expectations of graduates, context in which they learn, as well as future directions for improving health professions education graduate programs.

CONCLUSION

Program directors should consider programmatic aims, localized needs, and quality/standard of the program in designing Masters programs, with individualized growth opportunities for learners.

The Chief Residency in U.S. and Canadian Graduate Medical Education: A Scoping Review

PHENOMENON

Despite the nearly universal presence of chief residents within U.S. and Canadian residency programs and their critical importance in graduate medical education, to our knowledge, a comprehensive synthesis of publications about chief residency does not exist. An understanding of the current state of the literature can be helpful to program leadership to make evidence-based improvements to the chief residency and for medical education researchers to recognize and fill gaps in the literature.

APPROACH

We performed a scoping review of the literature about chief residency. We searched OVID Medline, PsycINFO, ERIC, and Web of Science databases through January 2023 for publications about chief residency. We included publications addressing chief residency in ACGME specialties in the U.S. and Canada and only those using the term "chief resident" to refer to additional responsibilities beyond the typical residency training. We excluded publications using chief residents as a convenience sample. We performed a topic analysis to identify common topics among studies.

FINDINGS

We identified 2,064 publications. We performed title and abstract screening on 1,306 and full text review on 208, resulting in 146 included studies. Roughly half of the publications represented the specialties of Internal Medicine (n = 37, 25.3%) and Psychiatry (n = 30, 20.5%). Topic analysis revealed six major topics: (1) selection of chief residents (2) qualities of chief residents (3) training of chief residents (4) roles of chief residents (5) benefits/challenges of chief residency (6) outcomes after chief residency.

INSIGHTS

After reviewing our topic analysis, we identified three key areas warranting increased attention with opportunity for future study: (1) addressing equity and bias in chief resident selection (2) establishment of structured expectations, mentorship, and training of chief residents and (3) increased attention to chief resident experience and career development, including potential downsides of the role.

Graduate Medical Education in Pathology: A Scoping Review

CONTEXT.—

Pathologists have produced a substantial body of literature on graduate medical education (GME). However, this body of literature is diverse and has not yet been characterized.

OBJECTIVE.—

To chart the concepts, research methods, and publication patterns of studies on GME in pathology.

DATA SOURCES.—

This was a systematic scoping review covering all literature produced since 1980 in the PubMed and Embase databases.

CONCLUSIONS.—

Research on GME in pathology is evenly dispersed across educational topics. This body of literature would benefit from research based on theory, stronger study designs, and studies that can provide evidence to support decisions on educational policies.

Technology-Enhanced Learning in Medical Education Collection: Latest Developments

Technology-enhanced learning (TEL) refers to learning activities and environments that are potentially improved or enhanced with information and communication technologies (Shen and Ho, 2020; Wasson and Kirschner, 2020). TEL may be implemented in face-to-face, distance/remote and blended or hybrid modes; in various environments such as online, classrooms, workplaces, communities, and other built and natural environments; include a range of learning designs and pedagogies/andragogies; involve synchronous and asynchronous interactions amongst students, teachers, workplace staff and clients, and/or community members; and delivered with the support of various technologies (Wasson and Kirschner, 2020). To date, the Technology-Enhanced Learning in Medical Education collection, part of MedEdPublish, has received submissions relating to several technologies to support learning, including web conferencing, web 2.0, e-textbooks, e-portfolios, software, generative artificial intelligence, simulation mannequins and wearables for point-of-view video, often in combination. Learning designs included flipped classroom with interactive case discussions (Imran et al., 2022), e-portfolios (Javed et al., 2023), didactic teaching followed by demonstrations of clinical skills on a simulation mannequin (Zwaiman et al., 2023), interdisciplinary case discussions to promote interprofessional learning (Major et al., 2023), patient panels to share narratives and perspectives (Papanagnou et al., 2023), and team-based learning (Lee & Wong, 2023). In the four papers that included evaluation, participant reaction (feedback on learning activities) and/or learning (self-reported through surveys, with pre- vs post-training comparisons or at different timepoints during learning) were reported, corresponding to levels 1 and 2 of the commonly used outcomes-focused Kirkpatrick model of evaluation (Allen et al., 2022). Two papers focused on the work of health professions educators, including conducting the nominal group technique, a qualitative research method, via web conferencing (Khurshid et al., 2023); and using ChatGPT to assist with various medical education tasks (Peacock et al., 2023).

Opportunities, Challenges, and Future Directions of Generative Artificial Intelligence in Medical Education: Scoping Review

BACKGROUND

Generative artificial intelligence (AI) technologies are increasingly being utilized across various fields, with considerable interest and concern regarding their potential application in medical education. These technologies, such as Chat GPT and Bard, can generate new content and have a wide range of possible applications.

OBJECTIVE

This study aimed to synthesize the potential opportunities and limitations of generative AI in medical education. It sought to identify prevalent themes within recent literature regarding potential applications and challenges of generative AI in medical education and use these to guide future areas for exploration.

METHODS

We conducted a scoping review, following the framework by Arksey and O'Malley, of English language articles published from 2022 onward that discussed generative AI in the context of medical education. A literature search was performed using PubMed, Web of Science, and Google Scholar databases. We screened articles for inclusion, extracted data from relevant studies, and completed a quantitative and qualitative synthesis of the data.

RESULTS

Thematic analysis revealed diverse potential applications for generative AI in medical education, including self-directed learning, simulation scenarios, and writing assistance. However, the literature also highlighted significant challenges, such as issues with academic integrity, data accuracy, and potential detriments to learning. Based on these themes and the current state of the literature, we propose the following 3 key areas for investigation: developing learners' skills to evaluate AI critically, rethinking assessment methodology, and studying human-AI interactions.

CONCLUSIONS

The integration of generative AI in medical education presents exciting opportunities, alongside considerable challenges. There is a need to develop new skills and competencies related to AI as well as thoughtful, nuanced approaches to examine the growing use of generative AI in medical education.

Evaluation of massive education in prison health: a perspective of health care for the person deprived of freedom in Brazil

Education, with an emphasis on prison health, has acted as a policy inducing changes in work processes, which the Brazilian National Health System (SUS) has used, and which is present in permanent health education, which promotes health care for people deprived of liberty. This article aims to present an analysis of the impacts of the strategy of massive education on prison health in Brazil from the perspective of health professionals and other actors operating in the Brazilian prison system. The data used in the study come from a questionnaire consisting of 37 questions applied nationwide between March and June 2022. Responses were collected from students who completed the course "Health Care for People Deprived of Freedom" of the learning pathway "Prison System", available in the Virtual Learning Environment of the Brazilian Health System (AVASUS). This course was offered nationally, whose adhesion (enrollment) occurred spontaneously, i.e., the course was not a mandatory. The data collected allowed us to analyze the impacts of massive education on prison health. The study also shows that the search for the course is made by several areas of knowledge, with a higher incidence in the health area, but also in other areas, such as humanities, which also work directly with the guarantee of the rights of people deprived of liberty, which are professionals in the areas of social work, psychology, and education. The analysis based on the data suggests that the massive education mediated by technology through the courses of the learning pathway, besides disseminating knowledge-following the action plan of the 2030 Agenda of the United Nations Educational, Scientific and Cultural Organization (UNESCO)-, are an effective tool to promote resilience in response to prison health and care demands of people deprived of liberty.

Effectiveness of a modified Balint group process on empathy and psychological skills employing Kirkpatrick's evaluation framework

Background

To evaluate, using Kirkpatrick's evaluation model, a modified Balint group (BG) by adding 5-10 min at the end of the session, where the facilitators will debrief the residents about critical themes mentioned in the session.

Methods

A quasi-experimental study with a mixed-method design was conducted among family medicine residents over 1 year, using focus groups and validated tools filled by residents and their corresponding patients. The residents' empathy through self and patient evaluation, psychological skills, and satisfaction with the educational activity were measured.

Results

The focus group revealed that the residents were aware of the change and considered it a closure to the encounter, helping decrease some uncertainty. Most of the residents did not consider the change helpful. Using validated instruments, BG was ineffective at improving the residents' empathy and psychological skills. There was a statistically significant main effect of time on Psychological Medicine Inventory (PMI) scores, F (1,13) = 7.709, p = 0.016.

Conclusion

Debriefing at the end of BG by the facilitators about key themes may help give the residents closure, decrease the uncertainty, and make them more aware of their feelings. Nevertheless, Balint groups are still not very well accepted by the residents, and they prefer direct feedback and support groups.

Evaluation of Continuing Professional Development for Physicians - Time for Change: A Scoping Review

Introduction

Evaluation of education interventions is essential for continuous improvement as it provides insights into how and why outcomes occur. Specifically, for physicians' continuing professional development (CPD) programs, which aim to upskill physicians in a range of practice-essential domains, evaluations are crucial to assure physicians' continuous development, enhanced patient care and safety. However, evaluations of health professions education (HPE) interventions tend to be outcomes focused, failing to capture how and why outcomes occur. This scoping review aimed to identify evaluation techniques used to evaluate CPD programs for physicians, and to determine how these techniques are being implemented as well as the their quality.

Methods

We searched PubMed, Embase, Web of Science, among others for English publications on evaluation of CPD programs for physicians, in the past decade. We used a data charting template to extract study details regarding the evaluation techniques and produced a checklist to assess the quality of the evaluations.

Results

101 studies were included; of which 91 studies did not use an evaluation framework. Our findings revealed shortcomings in the evaluations of CPD programs including lack of attention to: intervention processes; unintended outcomes and contextual factors; use of theory; evaluation framework use; and rationale for chosen evaluation method.

Discussion

Our findings highlighted major gaps in the evaluation techniques employed in physicians' CPD. Attention needs to be paid to evaluating both program processes and outcomes to illuminate how and why impacts are or are not occurring.

Comprehensive evaluation of a cutaneous T-cell lymphoma education webinar

BACKGROUND

Effective and timely referral, treatment and care of people with cutaneous T-cell lymphoma (CTCL) depend on clinical staff possessing highly specialised knowledge and skills. Because of the fragmented nature of the CTCL workforce, specialist education was delivered via a webinar.

AIM

The study aimed to comprehensively evaluate the webinar and test the validity of using an evaluation model for a one-off education event.

METHODS

The webinar was evaluated using Moore et al's conceptual model for evaluation of education. Data were collected using polling questions and post-webinar questionnaires and analysed using descriptive summaries and content analysis.

FINDINGS

Respondents agreed or strongly agreed that the webinar was an effective way to learn, enjoyable, relevant to their role and interesting. Learners also reported improvements in awareness, knowledge and understanding of CTCL, its referral and treatment.

CONCLUSION

Evaluating one-off education events using a conceptual model of evaluation for continuous medical education is recommended, with some adaptations to overcome limitations.

Taking the Pulse of the Current State of Simulation

Simulation in health-care professions has grown in the last few decades. We provide an overview of the history of simulation in other fields, the trajectory of simulation in health professions education, and research in medical education, including the learning theories and tools to assess and evaluate simulation programs. We also propose future directions for simulation and research in health professions education.

Dermatology residents as educators: a qualitative study of identity formation

BACKGROUND

One of the many identities a physician comes to form during their career is their identity as an educator. Exploring formation of this identity may enrich our understanding of how physicians make decisions related to their roles as educators, their behaviors, and how this ultimately influences the educational environment. It is the aim of this study to investigate educator identity formation of dermatology residents while early in their careers.

METHODS

Drawing on a social constructionist paradigm, we conducted a qualitative study, utilizing an interpretative approach. We examined longitudinal data over a 12-month period using dermatology residents' written reflections from their professional portfolios and semi-structured interviews. We collected this data as we progressed through and beyond a 4-month professional development program designed to encourage residents' growth as educators. Sixty residents in their second, third, or final year of residency programs located in Riyadh, Saudi Arabia were invited to take part in this study. Twenty residents participated with sixty written reflections and 20 semi-structured interviews. Qualitative data were analyzed using a thematic analysis approach.

RESULTS

Sixty written reflections and 20 semi-structured interviews were analyzed. Data was categorized according to themes corresponding to the original research questions. For the first research question regarding identity formation, themes included definitions of education, the process of education, and identity development. For the second research question, 1 theme entitled professional development program included, the following sub-themes: individual act, interpersonal activity, and an organizational undertaking, with many believing that residency programs should prepare residents for their educator roles. Participants also described newfound leadership ambitions of creating new dermatology fellowship programs as a result of taking part in the Resident-as-Educator program.

CONCLUSIONS

Our study provides insights on the dynamic formation of educator identities amongst dermatology residents. Investment in developing residents as educators through professional development programs may instigate transformational change on the individual physician level and profession's level.

Program Evaluation in Health Professions Education: An Innovative Approach Guided by Principles

PROBLEM

Program evaluation approaches that center the achievement of specific, measurable, achievable, realistic, and time-bound goals are common in health professions education (HPE) but can be challenging to articulate when evaluating emergent programs. Principles-focused evaluation is an alternative approach to program evaluation that centers on adherence to guiding principles, not achievement of goals. The authors describe their innovative application of principles-focused evaluation to an emergent HPE program.

APPROACH

The authors applied principles-focused evaluation to the Children's Hospital of Philadelphia Medical Education Collaboratory, a works-in-progress program for HPE scholarship. In September 2019, the authors drafted 3 guiding principles. In May 2021, they used feedback from Collaboratory attendees to revise the guiding principles: Advance Excellence , Build Bridges , and Cultivate Learning .

OUTCOMES

In July 2021, the authors queried participants about the extent to which their experience with the Collaboratory adhered to the revised guiding principles. Twenty of the 38 Collaboratory participants (53%) responded to the survey. Regarding the guiding principle Advance Excellence , 9 respondents (45%) reported that the Collaboratory facilitated engagement in scholarly conversation only by a small extent, and 8 (40%) reported it facilitated professional growth only by a small extent. Although some respondents expressed positive regard for the high degree of rigor promoted by the Collaboratory, others felt discouraged because this degree of rigor seemed unachievable. Regarding the guiding principle Build Bridges , 19 (95%) reported the Collaboratory welcomed perspectives within the group. Regarding the guiding principle Cultivate Learning , 19 (95%) indicated the Collaboratory welcomed perspectives within the group and across disciplines, and garnered collaboration.

NEXT STEPS

Next steps include improving adherence to the principle of Advancing Excellence , fostering a shared mental model of the Collaboratory's guiding principles, and applying a principles-focused approach to the evaluation of multi-site HPE programs.

"Train the Trainers" Program to Improve Knowledge, Attitudes and Perceptions About Organ Donation in the European Union and Neighbouring Countries: Pre- and Post- Data Analysis of the EUDONORGAN Project

EUDONORGAN, a European Union-funded project to improve organ and tissue donation, included a blended-based "Train the Trainers" program, which was implemented with the support of an international consortium from Croatia, Italy, Slovenia, and Spain. The web-based training included seven modules for which medical aspects, educational tips, and practical activities were scored using a 5-point Likert scale. The overall mean scores of satisfaction were higher than 4 for each module, without significant differences between HCPs and OKPs. In the face-to-face training survey similar scores above 4 were obtained for most items. Knowledge acquisition improved significantly in both HCPs and OKPs, as well as in transplant/donor coordinators, medical doctors, registered nurses, anesthesiologists/intensivists, and intensive care nurses. Improvements in attitudes and perceptions regarding organ donation were also observed, particularly among HCPs. In the accomplishment of the learning process, a successful pass mark of 95% was obtained. The "Train the Trainers" program was associated with an improvement in learning and attitudes of healthcare and non-healthcare professionals for the benefit of organ and tissue donation.

To Effectively Address Complex Healthcare Problems, Continuing Professional Development Must Evolve

ABSTRACT

Continuing professional development aims to provide health professionals with the knowledge, skills, and competencies needed to improve care. Physicians and other clinicians increasingly practice within complex health care delivery organizations aiming to improve the care of populations of patients with multiple problems and differing needs. These organizations are composed of local units in different departments and venues; these teams and the patients they care for change over time. Improving outcomes within constantly changing complex organizations delivering population care takes time and persistence. It takes time to equip critical masses of clinicians and other personnel with knowledge and skills to effect change. Although some changes might be simple, those involving new workflows require implementation support. Not all change will be smooth; individuals need opportunities to learn from and adjust their early intervention efforts, measure effectiveness of change, and sustain successful practices. Longitudinal support is necessary to affect change over complex organizations. This essay proposes that to be more supportive and valuable to health care delivery organizations, continuing professional development needs to intentionally participate in longitudinal, collaborative, context-specific, team-based interventions. An expanded menu of evaluation approaches will better describe the role of continuing professional development in helping health care professionals and organizations address increasingly complex health care delivery problems and improve patient and population outcomes. Selected concepts to achieve these ends are introduced at a high level in this article. Readers are invited to explore concepts that resonate with their current situation in further detail.

The impact of a targeted education package on the knowledge, attitudes, and utilisation of patient reported outcome measures amongst chiropractors in Australia

BACKGROUND

Patient Reported Outcome Measures (PROMs) have been shown to be valid and reliable indicators of health status and treatment outcomes, however, the current knowledge, understanding, and utilisation of PROMs within the Australian Chiropractic profession is limited. This study used the New World Kirkpatrick Model (NWKM) to evaluate whether an online PROM education package could improve knowledge, confidence, attitude, and utilisation of PROMs by chiropractors in Australia.

METHODS

A longitudinal cohort interventional study of chiropractors in Australia. The recruitment phase and data collection period occurred from November 2020 - May 2021. Participants completed three online surveys two weeks before, four weeks after, and 12 weeks after receiving an online education package that included ten evidence-based region-specific modules on PROMs. Survey questions were grouped into five subthemes for analysis according to the NWKM levels: (1) Reaction; (2) Learning - knowledge; (3) Learning - confidence; (4) Behaviour - attitude; (5) and Behaviour - utilisation).

RESULTS

Of the 116 participants that enrolled in the study, 43 completed all three survey and were included in the analysis. There was very positive reaction to the education package with mean response scores (1-5 Likert scale) for the reaction questions ranging from 3.75 to 4.43. There was a small, but significant, increase in knowledge (out of 32) at four weeks (24.3 ± 6.1) and 12 weeks after receiving the education package (27.2 ± 5.5), compared to baseline (27.4 ± 5.1). There was no effect of the intervention on clinician confidence or attitude towards PROMs. Utilisation of function- and pain-related PROMs did not change after the intervention. There was a small and significant (p < 0.05) increase in utilisation of health-related PROMs 12 weeks after the intervention.

CONCLUSION

Despite modest improvements in knowledge, which were retained 12 weeks after the educational package was provided, there was no evidence that participant confidence, attitude, or utilisation of PROMs changed because of the intervention. While the respondents' have positive attitudes and beliefs regarding PROMs use, further education surrounding the clinical translation process into clinical practice is required. Caution is advised when interpreting these findings due to the low participant response and completion rate with the potential for selection bias and the inability to generalise the results.

Massive health education through technological mediation: Analyses and impacts on the syphilis epidemic in Brazil

With syphilis cases on the rise, Brazil declared an epidemic in 2016. To address the consequent public health crisis, the Ministry of Health laid out a rapid response plan, namely, the "Syphilis No!" Project (SNP), a national instrument to fight the disease which encompasses four dimensions: (a) management and governance, (b) surveillance, (c) comprehensive care, and (d) strengthening of educommunication. In the dimension of education, the SNP developed the learning pathway "Syphilis and other Sexually Transmitted Infections (STIs)" to strengthen and promote Health Education. This pathway features 54 Massive Open Online Courses (MOOCs), delivered through the Virtual Learning Environment of the Brazilian Health System (AVASUS). This paper analyzes the impacts of the learning pathway "Syphilis and other STIs" on the response to the epidemic in Brazil, highlighting the educational process of the learning pathway and its social implications from the perspective of the United Nations' 2030 Agenda and its Sustainable Development Goals. Three distinct databases were used to organize the educational data: the learning pathway "Syphilis and other STIs" from AVASUS, the National Registry of HealthCare Facilities from the Brazilian Ministry of Health (MoH), and the Brazilian Occupation Classification, from the Ministry of Labor. The analysis provides a comprehensive description of the 54 courses of the learning pathway, which has 177,732 enrollments and 93,617 participants from all Brazilian regions, especially the Southeast, which accounts for the highest number of enrollees. Additionally, it is worth noting that students living abroad also enrolled in the courses. Data characterization provided a demographic study focused on the course participants' profession and level of care practiced, revealing that the majority (85%) worked in primary and secondary healthcare. These practitioners are the target audience of the learning pathway and, accordingly, are part of the personnel directly engaged in healthcare services that fight the syphilis epidemic in Brazil.

Standards of Care for the Health of Transgender and Gender Diverse People, Version 8

Background: Transgender healthcare is a rapidly evolving interdisciplinary field. In the last decade, there has been an unprecedented increase in the number and visibility of transgender and gender diverse (TGD) people seeking support and gender-affirming medical treatment in parallel with a significant rise in the scientific literature in this area. The World Professional Association for Transgender Health (WPATH) is an international, multidisciplinary, professional association whose mission is to promote evidence-based care, education, research, public policy, and respect in transgender health. One of the main functions of WPATH is to promote the highest standards of health care for TGD people through the Standards of Care (SOC). The SOC was initially developed in 1979 and the last version (SOC-7) was published in 2012. In view of the increasing scientific evidence, WPATH commissioned a new version of the Standards of Care, the SOC-8. Aim: The overall goal of SOC-8 is to provide health care professionals (HCPs) with clinical guidance to assist TGD people in accessing safe and effective pathways to achieving lasting personal comfort with their gendered selves with the aim of optimizing their overall physical health, psychological well-being, and self-fulfillment. Methods: The SOC-8 is based on the best available science and expert professional consensus in transgender health. International professionals and stakeholders were selected to serve on the SOC-8 committee. Recommendation statements were developed based on data derived from independent systematic literature reviews, where available, background reviews and expert opinions. Grading of recommendations was based on the available evidence supporting interventions, a discussion of risks and harms, as well as the feasibility and acceptability within different contexts and country settings. Results: A total of 18 chapters were developed as part of the SOC-8. They contain recommendations for health care professionals who provide care and treatment for TGD people. Each of the recommendations is followed by explanatory text with relevant references. General areas related to transgender health are covered in the chapters Terminology, Global Applicability, Population Estimates, and Education. The chapters developed for the diverse population of TGD people include Assessment of Adults, Adolescents, Children, Nonbinary, Eunuchs, and Intersex Individuals, and people living in Institutional Environments. Finally, the chapters related to gender-affirming treatment are Hormone Therapy, Surgery and Postoperative Care, Voice and Communication, Primary Care, Reproductive Health, Sexual Health, and Mental Health. Conclusions: The SOC-8 guidelines are intended to be flexible to meet the diverse health care needs of TGD people globally. While adaptable, they offer standards for promoting optimal health care and guidance for the treatment of people experiencing gender incongruence. As in all previous versions of the SOC, the criteria set forth in this document for gender-affirming medical interventions are clinical guidelines; individual health care professionals and programs may modify these in consultation with the TGD person.

The relevancy of massive health education in the Brazilian prison system: The course "health care for people deprived of freedom" and its impacts

Introduction

Brazil has one of the largest prison populations globally, with over 682,000 imprisoned people. Prison health is a public health emergency as it presents increasingly aggravating disease rates, mainly sexually transmitted infections (STI). And this problem already affects both developed and developing nations. Therefore, when thinking about intervention strategies to improve this scenario in Brazil, the course "Health Care for People Deprived of Freedom" (ASPPL), aimed at prison health, was developed. This course was implemented in the Virtual Learning Environment of the Brazilian Health System (AVASUS). Given this context, this study analyzed the aspects associated with massive training through technological mediation and its impacts on prison health.

Methods

This cross-sectional study analyzed data from 8,118 ASPPL course participants. The data analyzed were collected from six sources, namely: (i) AVASUS, (ii) National Registry of Health Care Facilities (CNES), (iii) Brazilian Occupational Classification (CBO), (iv) National Prison Department (DEPEN); (v) Brazilian Institute of Geography and Statistics (IBGE); and the (iv) Brazilian Ministry of Health (MoH), through the Outpatient Information System of the Brazilian National Health System (SIA/SUS). A data processing pipeline was conducted using Python 3.8.9.

Results

The ASPPL course had 8,118 participants distributed across the five Brazilian regions. The analysis of course evaluation by participants who completed it shows that 5,190 (63.93%) reported a significant level of satisfaction (arithmetic mean = 4.9, median = 5, and standard deviation = 0.35). The analysis revealed that 3,272 participants (40.31%) are health workers operating in distinct levels of care. The prison system epidemiological data shows an increase in syphilis diagnosis in correctional facilities.

Conclusions

The course enabled the development of a massive training model for various health professionals at all care levels and regions of Brazil. This is particularly important in a country with a continental size and a large health workforce like Brazil. As a result, social and prison health impacts were observed.

Rivalries for attention: insights from a realist evaluation of a postgraduate competency-based medical education implementation in Canada

BACKGROUND

Implementing competency-based medical education (CBME) in post-graduate medical education (PGME) is a complex process that requires multiple systemic changes in a complex system that is simultaneously engaged in multiple initiatives. These initiatives often compete for attention during the implementation of CBME and produce unintended and unanticipated consequences. Understanding the impact of this context is necessary for evaluating the effectiveness of CBME. The purpose of the study was to identify factors, such as contexts and processes, that contribute to the implementation of CBME.

METHODS

We conducted a realist evaluation using data collected from 15 programs through focus groups with residents (2 groups, n = 16) and faculty (one group, n = 8), and semi-structured interviews with program directors (n = 18), and program administrators (n = 12) from 2018 to 2021. Data were analyzed using a template analysis based on a coding framework that was developed from a sample of transcripts, the context-mechanism-outcomes framework for realist evaluations, and the core components of CBME.

RESULTS

The findings demonstrate that simultaneous initiatives in the academic health sciences system creates a key context for CBME implementation - rivalries for attention - and specifically, the introduction of curricular management systems (CMS) concurrent to, but separate from, the implementation of CBME. This context influenced participants' participation, communication, and adaptation during CBME implementation, which led to change fatigue and unmet expectations for the collection and use of assessment data.

CONCLUSIONS

Rival initiatives, such as the concurrent implementation of a new CMS, can have an impact on how programs implement CBME and greatly affect the outcomes of CBME. Mitigating the effects of rivals for attention with flexibility, clear communication, and training can facilitate effective implementation of CBME.

Use of a 2-year continuing professional development programme to change Japanese physicians' attitudes to learning primary care: a qualitative study

OBJECTIVE

To evaluate changes in the learning attitudes of primary care physicians.

DESIGN

Qualitative study through one focus group interview with the programme's participants. Analysis of the focus group content using the Steps for Coding and Theorization method.

SETTING

Japan.

PARTICIPANTS

Eight primary care physicians who completed a 2-year continuing professional development (CPD) programme using a problem-based learning (PBL) approach, focused on acquiring the skills needed to practise as primary care physicians in the community.

RESULTS

Participants described positive changes in their attitudes and behaviours as a result of the training programme. These changes were grouped into three main themes: 'changes in learning methods regarding medical practice', 'encounters with diverse perspectives and values, and confidence gained from those encounters', and 'showing one's attitude towards learning and its influence on others'. The experienced practitioners participating in this study reported that the programme helped them apply their skills more broadly; for example, searching the literature for psychosocial aspects of practice and engaging more comfortably with diverse perspectives. They reported the positive impact of their learning on their coworkers.

CONCLUSION

A 2-year CPD programme using PBL can influence primary care physicians' attitudes and learning-related behaviours. Further research is needed to determine which specific aspects of the programme are the most effective and whether the changes in attitudes and behaviours described affect patient care.

Virtual Learning Environment of the Brazilian Health System (AVASUS): Efficiency of Results, Impacts, and Contributions

The Virtual Learning Environment of the Brazilian Health System (AVASUS) is a free and open distance education platform of the Ministry of Health (MS). AVASUS is a scalable virtual learning environment that has surpassed 800,000 users, 2 million enrollments, and 310 courses in its catalog. The objective of this paper was to assess the impacts of the educational offerings on health services and AVASUS course participants' professional practice. This study analyzed data from AVASUS, the Brazilian National Registry of Health Care Facilities (CNES), the Brazilian Occupational Classification (CBO), and a questionnaire applied to 720-course participants from five regions of Brazil. After acquiring and extracting data, computational methods were used for the evaluation process. Only the responses of 462 participants were considered for data analysis, as they had a formal link to CNES. The results showed that respondents recommended 76.2% of AVASUS courses to peers. Accordingly, the quality of educational offerings motivated 81.3% of such recommendations. In addition, 75.6% of course participants who answered the questionnaire also indicated that AVASUS course contents contribute to enhancing existing health services in the health facilities where they work. Finally, 24.6% of all responses mentioned that courses available in AVASUS were essential in offering new health services in such facilities.

Rapid deployment of a virtual simulation curriculum to prepare for critical care triage during the COVID-19 pandemic

Background

During the COVID-19 pandemic in Ontario, Canada, an EmergencyStandardofCare forMajorSurge was created to establish a uniform process for the “triage” of finite critical care resources. This proposed departure from usual clinical care highlighted the need for an educational tool to prepare physicians for making and communicating difficult triage decisions. We created a just-in-time, virtual, simulation-based curriculum and evaluated its impact for our group of academic Emergency Physicians.

Methods

Our curriculum was developed and evaluated following Stufflebeam’s Context-Input-Process–Product model. Our virtual simulation sessions, delivered online using Microsoft Teams, addressed a range of clinical scenarios involving decisions about critical care prioritization (i.e., Triage). Simulation participants completed a pre-course multiple-choice knowledge test and rating scales pertaining to their attitudes about using the Emergency Standard of Care protocol before and 2–4 weeks after participating. Qualitative feedback about the curriculum was solicited through surveys.

Results

Nine virtual simulation sessions were delivered over 3 weeks, reaching a total of 47 attending emergency physicians (74% of our active department members). Overall, our intervention led to a 36% (95% CI 22.9–48.3%) improvement in participants’ self-rated comfort and attitudes in navigating triage decisions and communicating with patients at the end of life. Scores on the knowledge test improved by 13% (95% CI 0.4–25.6%). 95% of participants provided highly favorable ratings of the course content and similarly indicated that the session was likely or very likely to change their practice. The curriculum has since been adopted at multiple sites around the province.

Conclusion

Our novel virtual simulation curriculum facilitated rapid dissemination of the Emergency Standard of Care for Major Surge to our group of Emergency Physicians despite COVID-19-related constraints on gathering. The active learning afforded by this method improved physician confidence and knowledge with these difficult protocols.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43678-022-00280-6.