Twelve tips for evaluating educational programs

Using Dashboards to Support Continuous Quality Improvement in Undergraduate and Graduate Medical Education

Like other complex systems, medical education programs require a systematic continuous quality improvement (CQI) approach to drive effective improvement. Accreditation bodies in both undergraduate medical education (UME) and graduate medical education (GME) require programs to have effective CQI processes. Dashboards facilitate visualization and tracking of key metrics that impact medical education programming, thus driving excellence. Keys to developing useful dashboards include using existing program evaluation frameworks to identify desired outputs, determine acceptable evidence, and identify key data sources. In developing dashboards, it is important to consider data management including oversight and appropriate sharing of reports. When effectively developed and delivered to key parties, data dashboards serve as valuable tools to drive improvement of medical education programing. The purpose of this paper is to provide guidance for dashboard implementation and use in medical education, with a focus on UME and GME, based on available literature and experiences in our own institutions.

Practical Guide to Ethics in Surgical Education Research

This Guide to Statistics and Methods describes ethical considerations when a study population includes learner participants.

Evolving from Didactic to Dialogic: How to Improve Faculty Development and Support Faculty Developers by Using Action Research

Problem: Since competency-based medical education has gained widespread acceptance to guide curricular reforms, faculty development has been regarded as an indispensable element to make these programs successful. Faculty developers have striven to design and deliver myriad of programs or workshops to better prepare faculty members for fulfilling their teaching roles. However, how faculty developers can improve workshop delivery by researching their teaching practices remains underexplored. Intervention: Action research aims to understand real world practices and advocates for formulation of doable plans through cycles of investigations, and ultimately contributes to claims of knowledge and a progression toward the goal of practice improvement. This methodology aligns with the aim of this study to understand how I could improve a faculty development workshop by researching my teaching practices. Context: In 2016, we conducted four cycles of action research in the context of mini-Clinical Evaluation Exercise (mini-CEX) workshops within a faculty development program aiming for developing teaching and assessment competence in faculty members. We collected multiple sources of qualitative data for thematic analysis, including my reflective journal, field notes taken by a researcher-observer, and post-workshop written reflection and feedback in portfolio from fourteen workshop attendees aiming to develop faculty teaching and assessment competence. Impact: By doing action research, I scrutinized each step as an opportunity for change, enacted adaptive practice and reflection on my teaching practices, and formulated action plans to transform a workshop design through each cycle. In so doing, my workshop evolved from didactic to dialogic with continuous improvement on enhanced engagement, focused discussion and participant empowerment through a collaborative inquiry into feedback practice. Moreover, these processes of action research also supported my growth as a faculty developer. Lessons Learned: The systematic approach of action research serves as a vehicle to enable faculty developers to investigate individual teaching practices as a self-reflective inquiry, to examine, rectify, and transform processes of program delivery, and ultimately introduce themselves as agents for change and improvement.

Twelve tips to develop entrustable professional activities

Entrustable professional activities (EPAs), units of professional practice that require proficient integration of multiple competencies and can be entrusted to a sufficiently competent learner, are increasingly being used to define and inform curricula of health care professionals. The process of developing EPAs can be challenging and requires a deep yet pragmatic understanding of the concepts underlying EPA construction. Based on recent literature and the authors' lessons learned, this article provides the following practical and more or less sequential recommendations for developing EPAs: [1] Assemble a core team; [2] Build up expertise; [3] Establish a shared understanding of the purpose of EPAs; [4] Draft preliminary EPAs; [5] Elaborate EPAs; [6] Adopt a framework of supervision; [7] Perform a structured quality check; [8] Use a Delphi approach for refinement and/or consensus; [9] Pilot test EPAs; [10] Attune EPAs to their feasibility in assessment; [11] Map EPAs to existing curriculum; [12] Build a revision plan.

Teaching Medical Students About Kawasaki Disease: The Development and Evaluation of a Digital Educational Resource

Introduction

The use of digital educational resources has gained widespread popularity across all disciplines in medical education; pediatrics being no exception. In this paper, we report how instructional design and multimedia principles were used to develop and evaluate an e-learning resource on Kawasaki Disease, which was primarily created as a revision aid for undergraduate medical students.

Methods

The design and development of the resource was based upon the Analysis, Design, Development, Implementation and Evaluation (ADDIE) instructional model. A "People, Activities, Contexts and Technologies" (PACT) analysis was initially undertaken to identify learners' needs, and the 12 Principles of Multimedia Design informed the development of the resource. The evaluation strategy was informed by the Usability Evaluation Method for e-Learning Applications and focused on the instructional design parameters of navigation, visual design and intrinsic motivation to learn.

Results and Conclusion

Feedback from the seven medical students who completed and evaluated the resource showed high levels of satisfaction. Students perceived it to be beneficial to their learning and indicated a preference for an interactive digital resource over traditional modes of learning, such as textbooks. However, as this was a small-scale evaluation, suggestions for further evaluation and how this may inform ongoing development of the resource are considered in this paper.

Program evaluation: An educator's portal into academic scholarship

Program evaluation is an "essential responsibility" but is often not seen as a scholarly pursuit. While Boyer expanded what qualifies as educational scholarship, many still need to engage in processes that are rigorous and of a requisite academic standard to be labelled as scholarly. Many medical educators may feel that scholarly program evaluation is a daunting task due to the competing interests of curricular change, remediation, and clinical care. This paper explores how educators can take their questions around outcomes and efficacy of our programs and efficiently engage in education scholarship. The authors outline how educators can examine whether training programs have a desired impact and outcomes, and then how they might leverage this process into education scholarship.

A Thousand Words About Modern Medical Education: A Mini-Review Concerning the Theory of Education

The present article represents a mini-review and a reflective essay concerning modern medical education methods, as well as ways to adapt them to medical education's local conditions (disciplines), including basic medical sciences. We introduced Gagné's theory of learning and other theories – Constructivist, Experiential, and Humanistic – followed by Dennik's "twelve tips" for effective learning and Harden's ten questions for curriculum development. Outcome-based education (OBE) was discussed and related to relevant concepts within Miller's pyramid and Bloom's taxonomy. Harden's SPICES model was emphasized concerning education strategies while discussing the assessment of learning (AoL), assessment as learning (AaL), and assessment for learning (AfL). Finally, the authors advise exploring the adaptation of modern education methods for a specific discipline of basic medical sciences – Human Anatomy – by incorporating the abovementioned concepts and integrating different AfL and AaL assessment tools while conveying a graphical concept map for this scenario.

Teaching Undergraduate Medical Students Non-Technical Skills: An Evaluation Study of a Simulated Ward Experience

PURPOSE

Research suggests that medical students in the UK report a need to be better prepared for the non-technical skills required for the role of a junior doctor. A Simulated Ward Experience was developed in an attempt to address this need. The purpose of this study was (1) to evaluate the effectiveness of the Simulated Ward Experiences by examining students' reactions regarding the quality of teaching and (2) to examine the main drivers of learning and the extent to which students felt it helped them prepare for their future training.

METHODS

A mixed method evaluation study was conducted using a questionnaire and focus groups. Final year students who participated in the Simulated Ward Experience were invited to contribute to the evaluation, out of which 25 completed the questionnaire and 13 took part in focus group interviews. Data analysis were conducted by means of descriptive statistics for questionnaire data and thematic analysis of focus group transcripts.

RESULTS

The median Likert scores for quality of teaching Non-Technical Skills were either very good or excellent, demonstrating that students were highly satisfied with the way in which these were taught. Qualitative data provided further insights into the aspects of the intervention that promoted learning, and these were categorised into four themes, including realism of the simulation; relevance for the role and responsibilities of the Foundation Year 1 Doctor (including Non-Technical Skills); learning from and with others; and supportive learning environment.

CONCLUSION

This evaluation study provides further evidence of the value of learning in a simulated setting, particularly when it is closely aligned to the real clinical context and creates opportunities to practice skills that are perceived to be relevant by the learner. Study limitations are recognised and suggestions for further studies are provided.

Learner Experiences Matter in Interprofessional Palliative Care Education: A Mixed Methods Study

CONTEXT

Interprofessional collaboration is needed in palliative care and many other areas in health care. Pallium Canada's two-day interprofessional Learning Essential Approaches to Palliative care Core courses aim to equip primary care providers from different professions with core palliative care skills.

OBJECTIVES

Explore the learning experience of learners from different professions who participated in Learning Essential Approaches to Palliative care Core courses from April 2015 to March 2017.

METHODS

This mixed methods study was designed as a secondary analysis of existing data. Learners had completed a standardized course evaluation survey online immediately post-course. The survey explored the learning experience across several domains and consisted of seven closed ended (Likert Scales; 1 = "Total Disagree", 5 = "Totally Agree") and three open-ended questions. Quantitative data were analyzed using descriptive statistics and Kruskal-Wallis non-parametric test tests, and qualitative data underwent thematic analysis.

RESULTS

During the study period, 244 courses were delivered; 3045 of 4636 participants responded (response rate 66%); physicians (662), nurses (1973), pharmacists (74), social workers (80), and other professions (256). Overall, a large majority of learners (96%) selected "Totally Agree" or "Agree" for the statement "the course was relevant to my practice". A significant difference was noted across profession groups; X² (4) = 138; p < 0.001. Post-hoc analysis found the differences to exist between physicians and pharmacists (X² = -4.75; p < 0.001), and physicians and social workers (X² = -6.63; p < 0.001). No significant differences were found between physicians and nurses (X² = 1.31; p = 1.00), and pharmacists and social workers (X² = -1.25; p = 1.00). Similar results were noted for five of the other statements.

CONCLUSION

Learners from across profession groups reported this interprofessional course highly across several learning experience parameters, including relevancy for their respective professions. Ongoing curriculum design is needed to fully accommodate the specific learning needs of some of the professions.

Overcoming Gender Bias in Assessment of Surgical Skills

OBJECTIVE

Female and male surgical residents are treated differently based on their gender. The use of assessment tools can help obtain objectivity in surgical skills assessment, avoid gender bias, and promote equal learning opportunities. The objective of the study was to explore whether knowledge of gender causes bias in the rating of surgical skills, and whether the raters' gender, surgical specialty, or experience affect ratings.

DESIGN

The study is designed as a cross-sectional study, where an anonymous video showing surgical performance was rated by surgeons from different surgical specialties and different levels of surgical experience. The same video was presented to two randomized groups as either a male or female surgical trainee performing the procedure.

SETTING

The participants used Objective Structured Assessment of Technical Skills (OSATS) to rate a video-recorded exam from a surgical skills course at Copenhagen Academy for Medical Education and Simulation (CAMES). The video rating was done through an internet-based platform and could thus be performed at home or at another location of choice.

PARTICIPANTS

Novice and experienced surgeons from departments of gastrointestinal surgery, gynaecology, and urology in Denmark.

RESULTS

There was no difference in OSATS rating score of the perceived female and male trainee (female trainee (n = 50):12.26 (SD = 3.08), male trainee (n = 57): 12.00 (SD = 3.28), p-value = 0.67). Rater characteristics: gender, age, surgical experience, and surgical specialty did not affect ratings.

CONCLUSIONS

When OSATS a systematic assessment tool was used in rating surgical trainees we did not find any significant implicit gender bias. The rating was unaffected by the raters' gender, age, surgical specialty, or experience.

Protocol for an integrated evaluation framework to study training, curricular and contextual factors impacting the success of a measurement for improvement training programme for healthcare staff in Ireland

INTRODUCTION

Measurement for improvement is the process of collecting, analysing and presenting data to demonstrate whether a change has resulted in an improvement. It is also important in demonstrating sustainability of improvements through continuous measurement. This makes measurement for improvement a core element in quality improvement (QI) efforts. However, there is little to no research investigating factors that influence measurement for improvement skills in healthcare staff. This protocol paper presents an integrated evaluation framework to understand the training, curricular and contextual factors that influence the success of measurement for improvement training by using the experiences of trainees, trainers, programme and site coordinators.

METHODS AND ANALYSIS

This research will adopt a qualitative retrospective case study design based on constructivist-pragmatic philosophy. The Pressure Ulcers to Zero collaborative and the Clinical Microsystems collaborative from the Irish health system which included a measurement for improvement component have been selected for this study. This paper presents an integrated approach proposing a novel application of two pre-existing frameworks: the Model for Understanding Success in Quality framework and the Kirkpatrick Evaluation Model to evaluate an unexplored QI context and programme. A thematic analysis of the qualitative interview data and the documents collected will be conducted. The thematic analysis is based on a four-step coding framework adapted for this research study. The coding process will be conducted using NVivo V.12 software and Microsoft Excel. A cross-case comparison between the two cases will be performed.

ETHICS AND DISSEMINATION

The study has received an exemption from full ethical review from the Human Research Ethics Committee of University College Dublin, Ireland (LS-E-19-108). Informed consent will be obtained from all participants and the data will be anonymised and stored securely. The results of the study will be disseminated in peer-reviewed journals.

The impact of external academic accreditation of undergraduate medical program on students' satisfaction

BACKGROUND

The external academic accreditation is a quality assurance and auditing process that focuses on the structure, process, and outcome of the education. It is an interrupting and highly demanding process in terms of effort, time, financial, and human resources. However, it is unclear in the literature how much of these external quality assurance practices impeded in the accreditation processes would reflect on the other end of the learning pathway, including student satisfaction.

METHODS

A retrospective quantitative secondary data analysis, with a before-after comparison research design, was performed to evaluate external accreditation's impact on students' mean satisfaction score within two accreditation cycles at King Saud University (KSU)-Bachelor of Medicine, Bachelor of Surgery (MBBS) program.

RESULTS

The overall average students' satisfaction scores pre-and-post the first accreditation cycle were 3.46/5 (±0.35), 3.71 (±0.39), respectively, with a P-value of < 0.001. The effect of post first accreditation cycle was sustainable for a couple of years, then maintained above the baseline of the pre-first accreditation cycle until the pre-second accreditation cycle. Similarly, the overall average students' satisfaction scores pre-and-post the second accreditation cycles were 3.57/5 (±0.30) and 3.70 (±0.34), respectively, with a P-value of 0.04. Compared to the first accreditation cycle, the improvement of the mean score of students' satisfaction rates was not sustained beyond the year corresponding to the post-second accreditation cycle.

CONCLUSION

Both accreditation cycles were associated with an increased score in students' satisfaction. The preparatory phase activities and navigation through the self-study assessment while challenging the program's competencies are essential triggers for quality improvement practices associated with accreditation.

Twelve tips for point-of-care ultrasound teaching in low-resource settings

Point-of-care ultrasound (POCUS) has become the standard of care for many emergency department evaluations. Low-resource settings are ideal for POCUS to help improve access to imaging for a number of indications that are obstetric, disease or trauma related. The following twelve specific tips are aimed toward organizations and individuals who aspire to initiate POCUS training in a low-resource setting.

How to conduct cost and value analyses in health professions education: AMEE Guide No. 139

Growing demand for accountability, transparency, and efficiency in health professions education is expected to drive increased demand for, and use of, cost and value analyses. In this AMEE Guide, we introduce key concepts, methods, and literature that will enable novices in economics to conduct simple cost and value analyses, hold informed discussions with economic specialists, and undertake further learning on more advanced economic topics. The practical structure for conducting analyses provided in this guide will enable researchers to produce robust results that are meaningful and useful for improving educational practice. Key steps include defining the economic research question, identifying an appropriate economic study design, carefully identifying cost ingredients, quantifying, and pricing the ingredients consumed, and conducting sensitivity analyses to explore uncertainties in the results.

How Hands-On Pain Skills Intensive Trainings Complement ECHO Pain and Opioid Management Programs: A Program Evaluation with the Indian Health Service

OBJECTIVE

To evaluate the impact of Pain Skills Intensive trainings (PSIs) as a complement to the Indian Health Service (IHS) and the Chronic Pain and Opioid Management TeleECHO Program (ECHO Pain) collaboration.

DESIGN

On-site PSIs conducted over two to three days were added to complement ECHO Pain at various IHS areas to enhance pain skills proficiency among primary care teams and to expand the reach of ECHO collaboration to ECHO nonparticipants.

SETTING

This evaluation focuses on two PSI trainings offered to IHS clinicians in Albuquerque, New Mexico, and Spokane, Washington, in 2017.

METHODS

The mixed-methods design comprises CME surveys and focus groups at the end of training and 12 to 18 months later. Quality of training and perceived competence were evaluated.

RESULTS

Thirty-eight participants attended the two PSI workshops. All provided CME survey results, and 28 consented to use of their postsession focus group results. Nine clinicians participated in the virtual follow-up focus groups. IHS clinicians rated the PSIs highly, noting their hands-on and interdisciplinary nature. They reported above-average confidence in their skills. Follow-up focus groups indicated they were pursuing expanded options for their patients, consulting other clinicians, serving as pain consultants to their peers, and changing prescribing practices clinic-wide. However, rurality significantly limits access to ancillary and complementary services for many. Clinicians reported the need for additional training in integrating behavioral health into their practice.

CONCLUSIONS

Hands-on pain skills and information on medication-assisted treatment (MAT) are critical to the successful treatment of chronic pain and opioid use disorder. The PSIs provide clinicians with critical competencies in assessment and screening, pain management, and communication skills, complementing required IHS training and telementoring from ECHO Pain.

Exploring the connections between programmatic assessment and program evaluation within competency-based medical education programs

Programmatic assessment and program evaluation are both important within competency-based medical education (CBME) programs. Given this importance, there is value in evaluating programmatic assessment as well as using the information collected in programmatic assessment for the evaluation of CBME programs. In order to help facilitate these two activities, this paper distinguishes between programmatic assessment and program evaluation as well as highlights the connections between them. In doing so, it shows that programmatic assessment and program evaluation, when employed appropriately, can complement each other and contribute to the overall effectiveness of CBME programs.

A Proposed Framework to Develop, Describe and Evaluate Peer-Assisted Learning Programs

PHENOMENON

Peer-assisted learning (PAL) is increasingly used in different fields of education, including medical education, due to its established advantages. However, there are scarce data about the best practice guidelines for PAL program evaluation. The aim of this work was to develop a framework that can describe and develop PAL programs and consequently provide a tool for evaluation and comparison of PAL programs among different institutions.

APPROACH

A thorough literature review was made for assessment of different PAL programs development and implementation strategies, and PAL leaders at the University of Umm Al-Qura in Saudi Arabia were interviewed for development and revision of a framework for PAL program development, description and evaluation.

FINDINGS

A framework of four sections was developed ie context, theory, implementation processes and outcomes, and emergence. The context includes the learning objectives and the logistics. The theory includes the content nature, program design, number of tutees and tutors, participation, program orientation, program duration and timing, tutor recruitment and preparation and faculty involvement. Implementation process and outcome section includes the feedback collection and the pre- and post-intervention students' assessment. The emergence includes tracking the program evolution.

INSIGHTS

Development of a clear well-defined framework for description, implementation and evaluation of a PAL educational program can provide a foundation to unify the terms organizations use to communicate the parameters of PAL programs and overcome the jargon about PAL in the literature. It also can provide comparisons between the programs in an attempt to set best practice guidelines in the future for PAL program developing and implementation.

Social Accountability Frameworks and Their Implications for Medical Education and Program Evaluation: A Narrative Review

PURPOSE

Medical schools face growing pressures to produce stronger evidence of their social accountability, but measuring social accountability remains a global challenge. This narrative review aimed to identify and document common themes and indicators across large-scale social accountability frameworks to facilitate development of initial operational constructs to evaluate social accountability in medical education.

METHOD

The authors searched 5 electronic databases and platforms and the World Wide Web to identify social accountability frameworks applicable to medical education, with a focus on medical schools. English-language, peer-reviewed documents published between 1990 and March 2019 were eligible for inclusion. Primary source social accountability frameworks that represented foundational values, principles, and parameters and were cited in subsequent papers to conceptualize social accountability were included in the analysis. Thematic synthesis was used to describe common elements across included frameworks. Descriptive themes were characterized using the context-input-process-product (CIPP) evaluation model as an organizational framework.

RESULTS

From the initial sample of 33 documents, 4 key social accountability frameworks were selected and analyzed. Six themes (with subthemes) emerged across frameworks, including shared values (core social values of relevance, quality, effectiveness, and equity; professionalism; academic freedom and clinical autonomy) and 5 indicators related to the CIPP model: context (mission statements, community partnerships, active contributions to health care policy); inputs (diversity/equity in recruitment/selection, community population health profiles); processes (curricular activities, community-based clinical training opportunities/learning exposures); products (physician resource planning, quality assurance, program evaluation and accreditation); and impacts (overall improvement in community health outcomes, reduction/prevention of health risks, morbidity/mortality of community diseases).

CONCLUSIONS

As more emphasis is placed on social accountability of medical schools, it is imperative to shift focus from educational inputs and processes to educational products and impacts. A way to begin to establish links between inputs, products, and impacts is by using the CIPP evaluation model.

Curricular needs for training telemedicine physicians: A scoping review

PURPOSE

The use of telemedicine, a part of 'Virtual Care', is rapidly entering mainstream clinical practice. The ideal curriculum for educating physicians to practice in this emerging field has not been established. We examined the literature to evaluate published curricula for quality and comprehensiveness through the lens of Competency-Based Medical Education (CBME).

METHODS

We performed a scoping review using CanMEDS as a framework. Peer-reviewed articles describing telemedicine training curricula were identified. Trainee population, curricular points, stage of implementation, evaluation depth, country, and citations (a marker of quality) were examined.

RESULTS

Forty-three curricula from 11 countries were identified, addressing all training levels and covering multiple specialties. Instructional methods included lectures (60.5%), hands-on experiences (76%), directed reading (24%), online modules (21%), reflection (13%), simulation (34%), and group discussions (16%). Hands-on curricula covered all CanMEDS roles more often. Twenty-nine of the implemented curricula were evaluated; 83% were rated positively.

CONCLUSIONS

Our scoping review helps inform more comprehensive and efficacious curricula for teaching telemedicine. We suggest centering curricula on a competency-based, outcomes-oriented framework such as CanMEDS with multiple teaching modalities complementing hands-on experiences. This will facilitate rigorous telemedicine training to deliver on the promise of high-quality patient care.

Fellowship Training for the Future Clinical Microbiology Laboratory Director

Formal medical and public health microbiology (MPHM) fellowship programs play a key role in preparing future clinical microbiology laboratory directors for their leadership and management responsibilities. Given the continually evolving MPHM field, fellowships must remain adaptable to changes in the field, providing trainees with the opportunity to engage with newly emerging diagnostic modalities, while continuing to emphasize the "bread and butter" techniques of clinical microbiology. This article discusses the key components of a fellowship program and provides recommendations for incorporating educational best practices.

Twelve tips for conducting educational design research in medical education

Despite a steady growth in educational innovations and studies investigating the acceptance and effectiveness of these innovations, medical education has not realized sufficient improvement in practice and outcomes from these investments. In light of this lack of impact, there has been a growing call for studies that more effectively bridge the gap between research and practice. This paper introduces Educational Design Research (EDR) as a promising approach to address this challenge. Twelve tips are provided to inspire and guide medical educators to conduct EDR to achieve the dual goals of tackling a significant educational problem in a specific context while at the same time advancing the theoretical knowledge that may be used to improve practice elsewhere.

Next Steps in the Implementation of Learning Analytics in Medical Education: Consensus From an International Cohort of Medical Educators

BACKGROUND

With the implementation of competency-based assessment systems, education programs are collecting increasing amounts of data about medical learners. However, learning analytics are rarely employed to use this data to improve medical education.

OBJECTIVE

We identified outstanding issues that are limiting the effective adoption of learning analytics in medical education.

METHODS

Participants at an international summit on learning analytics in medical education generated key questions that need to be addressed to move the field forward. Small groups formulated questions related to data stewardship, learner perspectives, and program perspectives. Three investigators conducted an inductive qualitative content analysis on the participant questions, coding the data by consensus and organizing it into themes. One investigator used the themes to formulate representative questions that were refined by the other investigators.

RESULTS

Sixty-seven participants from 6 countries submitted 195 questions. From them, we identified 3 major themes: implementation challenges (related to changing current practices to collect data and utilize learning analytics); data (related to data collection, security, governance, access, and analysis); and outcomes (related to the use of learning analytics for assessing learners and faculty as well as evaluating programs and systems). We present the representative questions and their implications.

CONCLUSIONS

Our analysis highlights themes regarding implementation, data management, and outcomes related to the use of learning analytics in medical education. These results can be used as a framework to guide stakeholder education, research, and policy development that delineates the benefits and challenges of using learning analytics in medical education.

Developing occupational therapists' capabilities for decision-making capacity assessments: how does a support role facilitate workplace learning?

INTRODUCTION

Healthcare practitioners are required to develop capabilities in an effective and efficient manner. Yet, developing capabilities in healthcare settings can be challenging due to the unpredictable nature of practice and increasing workloads. Unsurprisingly, healthcare practitioner development is often situated outside of practice, for example in formal teaching sessions. Supporting practitioners to develop capabilities through engagement with day-to-day practice, whilst advantageous in terms of authenticity and being highly valued, remains a key challenge for healthcare educators. This qualitative interview study aimed to explain, from the learner's perspective, how a dedicated support role develops occupational therapists' capability to contribute to decision-making capacity assessments.

METHODS

Individual semi-structured interviews were conducted with a purposive sample of 12 occupational therapists. Informed by workplace learning theory, interview transcripts were analyzed using thematic analysis process.

RESULTS

Participants provided rich descriptions of how they developed in their capability to engage in decision-making capacity assessments. Participants reported that their learning was facilitated by the dedicated support role in three key ways: 1) structuring a journey of learning, 2) providing tailored guidance, and 3) fostering a supportive learning environment.

DISCUSSION

Participants valued the authentic workplace learning opportunities afforded by the dedicated support role. Findings suggest that capabilities, such as decision-making capacity assessment, can be developed through practice when enriched by a dedicated support role. However, further research examining the sustainability and transferability of this model and its application to other capabilities are warranted.

Can combined online and face-to-face continuing medical education improve the clinical knowledge and skills of family doctors in Vietnam? A cluster randomised controlled trial

OBJECTIVES

To assess the effectiveness of a combined online and face-to-face continuing medical education (CME) programme, for improvement in clinical knowledge and skills of family doctors, in comparison with a control group; and to explore the self-reported satisfaction, competencies and confidence of those in the intervention group.

METHODS

We used a cluster randomised controlled trial, with pre- and post-testing, and a feedback survey at the end of the 18-month CME programme. The measurements consisted of a multiple-choice test, an objective structured clinical examination test and an anonymously self-administered questionnaire.

RESULTS

There were 58 family doctors from four provinces in the intervention group and 32 doctors from three provinces in control group, both in the Mekong Delta region in Vietnam. The mean age of participants was 47.8 years, and the female/male ratio was 1/2.9. After training, the intervention group had significantly higher scores on overall knowledge (mean difference = 1.4, 95% CI 1.0-1.86, P < 0.001; Cohen's d 1.36, Pearson's r 0.53), in four of the five education modules: peptic disorders, diabetes, hypertension and bone-muscle-joint diseases (Pearson's r 0.56, 0.56, 0.34 and 0.4, respectively), and in problem-solving skills (Pearson's r 0.27). Self-reports showed a positive learning attitude, strong interest and improved confidence and competency among doctors in the intervention group.

CONCLUSIONS

A combined online and face-to-face CME programme proved applicable and effective for improving the clinical knowledge and problem-solving skills of family doctors in Vietnam.

A novel approach to the program evaluation committee

BACKGROUND

The Accreditation Council for Graduate Medical Education requires each residency program to have a Program Evaluation Committee (PEC) but does not specify how the PEC should be designed. We sought to develop a PEC that promotes resident leadership and provides actionable feedback.

METHODS

Participants were residents and faculty in the Traditional Internal Medicine residency program at Yale School of Medicine (YSM). One resident and one faculty member facilitated a 1-h structured group discussion to obtain resident feedback on each rotation. PEC co-facilitators summarized the feedback in written form, then met with faculty Firm Chiefs overseeing each rotation and with residency program leadership to discuss feedback and generate action plans. This PEC process was implemented in all inpatient and outpatient rotations over a 4-year period. Upon conclusion of the second and fourth years of the PEC initiative, surveys were sent to faculty Firm Chiefs to assess their perceptions regarding the utility of the PEC format in comparison to other, more traditional forms of programmatic feedback. PEC residents and faculty were also surveyed about their experiences as PEC participants.

RESULTS

The PEC process identified many common themes across inpatient and ambulatory rotations. Positives included a high caliber of teaching by faculty, highly diverse and educational patient care experiences, and a strong emphasis on interdisciplinary care. Areas for improvement included educational curricula on various rotations, interactions between medical and non-medical services, technological issues, and workflow problems. In survey assessments, PEC members viewed the PEC process as a rewarding mentorship experience that provided residents with an opportunity to engage in quality improvement and improve facilitation skills. Firm chiefs were more likely to review and make rotation changes in response to PEC feedback than to traditional written resident evaluations but preferred to receive both forms of feedback rather than either alone CONCLUSIONS: The PEC process at YSM has transformed our program's approach to feedback delivery by engaging residents in the feedback process and providing them with mentored quality improvement and leadership experiences while generating actionable feedback for program-wide change. This has led to PEC groups evaluating additional aspects of residency education.

The impact of a national cardiotocography education program on neonatal and maternal outcomes: A historical cohort study

INTRODUCTION

Studies indicate an association between errors in cardiotocography (CTG) management and hypoxic brain injuries among newborns. Continuing professional education is recommended. We aimed to examine whether the implementation of a national interprofessional CTG education program in Denmark was associated with a decrease in risk of fetal hypoxia measured by umbilical cord pH < 7.00, 5-minute Apgar score <7 or neonatal therapeutic hypothermia. As a secondary aim, we assessed whether the educational intervention was associated with an increase in operative deliveries.

MATERIAL AND METHODS

We conducted a historical cohort study from 2009 to 2015 including all intended vaginal deliveries with liveborn singletons in cephalic presentation and gestational age ≥37 weeks. Data were retrieved from the Medical Birth Register and the National Patient Register. The study period was divided in three: pre-implementation (2009-2012), implementation (2013) and post-implementation (2014-2015). Using logistic regression we estimated odds ratios (OR) of fetal hypoxia outcomes using the pre-implementation period as reference. Analyses were adjusted for potential maternal, neonatal and delivery-associated confounders. Missing data were accounted for by multiple imputation.

RESULTS

In all, 331 282 deliveries were included. Overall risks of pH < 7.00, Apgar score <7 and therapeutic hypothermia were respectively 0.45%, 0.58% and 0.06%. Adjusted OR in the post-implementation period were 1.12 (95% confidence interval [CI] 1.00-1.26), 0.99 (95% CI 0.90-1.10) and 1.34 (95% CI 0.99-1.82) for the three outcomes, respectively. The pH missingness equaled 12.4%. Odds of emergency cesarean section was unaltered, whereas the odds of assisted vaginal delivery decreased by 14% (0.86, 95% CI 0.84-0.89).

CONCLUSIONS

Healthcare professionals are considered the weakest link of CTG technology. We did not find that increasing healthcare professionals' CTG interpretation skills affected the risk of fetal hypoxia. Missing data for pH values were substantial and represent a limitation of the study. We cannot with certainty rule out that missingness masked a true effect of the intervention. Our study indicates that assisted vaginal deliveries can be decreased without an increased risk of fetal hypoxia. Dilution of effect in a complex clinical setting, rare outcomes, insufficient intervention and a possible overestimation of the impact of errors in CTG management might explain the lack of effect.

Faculty development program evaluation: a need to embrace complexity

Faculty development is essential for renewing and assisting faculty to maintain teaching effectiveness and adapt to innovations in Health Professions educational institutions. The evaluation of faculty development programs appears to be a significant step in maintaining its relevance and efficiency. Yet, little has been published on the specific case of faculty development program evaluation in spite of the availability of general program evaluation models. These models do not measure or capture the information educators want to know about outcomes and impacts of faculty development. We posit that two reasons account for this. The first is the evolving nature of faculty development programs as they adapt to current reforms and innovations. The second involves the limitations imposed by program evaluation models that fail to take into account the multiple and unpredictable outcomes and impacts of faculty development. It is generally accepted that the outcomes and impacts are situated at various levels, ranging from the individual to the institutional and cultural levels. This calls for evaluation models that better capture the complexity of the impacts of faculty development, in particular the reciprocal relationships between program components and outcomes. We suggest conceptual avenues, based on Structuration Theory, that could lead to identifying the multilevel impacts of faculty development.

Evaluation of critical thinking course for premedical students using literature and film

PURPOSE

Premedical education is one stage in which various educational approaches are used to promote critical thinking. Given that critical thinking ability could be regarded as one of the intended outcomes of social science and humanities education, this study explored the effectiveness of a course to promote critical thinking in a premedical curriculum using both literature and film.

METHODS

Fifty-one 2nd year premedical students enrolled in a 'Critical Thinking for Premeds' course. Students were required to read or watch a selected material, submit group discussion agendas, attend five group discussion sessions, and write critical essays. Five tutors facilitated the group discussions, observed and assessed the students' performance and critical essay. Students' critical thinking disposition and opinion on assigned reading materials were examined before and after the course. A program evaluation survey was conducted to investigate the students' reaction after the course.

RESULTS

On average, students appreciated 78.6% of the total assigned materials. The students indicated that group discussions and the narrative comments of facilitators contributed the most to develop critical thinking. After the course, the students' tendency preferring cheaper price books, as well as medicine-related books decreased significantly. Students who had critical essay scores greater than or equal to the median demonstrated a significant improvement in critical thinking disposition scores.

CONCLUSION

The course was well-accepted by premedical students and had several positive outcomes. A more effective use of the course could be anticipated with a clearer explanation of the purpose, the consideration of previous reading experience, and use of complementary learning activities.

Effect of In Situ High-Fidelity Simulation Training on the Emergency management of Pneumonia (INSTEP): a mixed-methods study

Background

The patient safety agenda has propelled the rise of simulation education, but relatively few evaluations of simulation-based educational interventions have focused on patient outcomes.

Objective

To evaluate the impact of an in situ, high-fidelity simulation teaching intervention on the management of community-acquired pneumonia in the ambulatory care unit of a district general hospital.

Methods

This study used a mixed-methods approach to evaluate the impact of a programme of 10 in situ high-fidelity simulation education sessions delivered to a total of 10 junior doctors, nine nurses and seven healthcare assistants. Participants were tasked with managing a manikin simulating a patient with pneumonia in real time in a working clinical area. Subsequent structured debrief emphasised key themes from the national guidelines on pneumonia management. The intervention was evaluated through an immediate feedback form, follow-up semistructured interviews by independent qualitative researchers that underwent content analysis and triangulation with audit data on compliance with national pneumonia guidelines before and after the simulation intervention.

Results

The in situ simulation intervention was valued by participants both in immediate written feedback and in follow-up semistructured interviews. In these interviews, 17 of 18 participants were able to identify a self-reported change in practice following the simulation intervention. Furthermore, most participants reported observing a change in the clinical practice of their colleagues following the training. Collected audit data did not show a statistically significant change in compliance with the guidelines for the management of pneumonia.

Conclusion

This study found evidence of a change in both self-reported and observed clinical practice following a simulation intervention, supporting expert opinion that simulation education can impact clinician behaviours and patient outcomes in complex clinical scenarios. Furthermore, this feasibility study provides a transferrable method to evaluate the real-world impact of simulation education that merits further investigation through an appropriately powered study.

Using Appreciative Inquiry to Inform Program Evaluation in Graduate Medical Education

BACKGROUND

Graduate medical education programs are expected to conduct an annual program evaluation. While general guidelines exist, innovative and feasible approaches to program evaluations may help efforts at program improvement. Appreciative Inquiry is an approach that focuses on successful moments, effective processes, and programs' strengths.

OBJECTIVE

We implemented a novel application of Appreciative Inquiry and its 4 phases (Inquire, Imagine, Innovate, and Implement) and demonstrate how it led to meaningful improvements in a pediatric pulmonology fellowship program.

METHODS

As part of the Inquire and Imagine phases, the authors developed an interview guide that aligned with Appreciative Inquiry concepts. Two faculty members conducted semistructured interviews with a convenience sample of 11 of 14 fellowship alumni. Interviews were audiotaped, transcribed, and reviewed. A summary of the findings was presented to the Program Evaluation Committee, which then directed the Innovate and Implement phases.

RESULTS

Appreciative Inquiry was acceptable to the alumni and feasible with the authors' self-directed learning approach and minimal administrative and financial support. In the Inquire phase, alumni identified program strengths and successes. In the Imagine phase, alumni identified program changes that could aid transition to independent practice for future fellows (an identified program goal). Based on the results of the Appreciative Inquiry, program leadership and the Program Evaluation Committee selected improvements for implementation.

CONCLUSIONS

For small programs, Appreciative Inquiry is an innovative and feasible approach to program evaluation that facilitates actionable program improvement recommendations.

Educational sessions may not be enough to improve knowledge about hand hygiene: Assessing the knowledge about hand hygiene of health workers before and after an educational workshop in Sudan

Background: In an attempt to defeat the high prevalence of health care associated infections, by raising the awareness about hand hygiene, we implemented a quasi-experimental study as part of a quality improvement project to evaluate the efficacy of focused educational workshop on knowledge about hand hygiene among health care workers in Sudan. Methods: Seventy-three participants were recruited from two public hospitals and one private hospital in Khartoum, Sudan in September 2017. The knowledge before and after the educational workshop was assessed for each participant using the World Health Organization hygiene associated infection questionnaire. We analyzed data using SPSS v22 and used McNemar’s test to compare knowledge before and after the workshop. Results: Around 40% of participants worked in general ward and 85% of them were women. Almost 70% were midwives or nurses. The mean age of participants was 28.4 years. We compared the knowledge of hand hygiene between nurses and doctors before the workshop, and the results showed that nurses had better knowledge in almost all aspects of hand hygiene. When we compared the knowledge before and immediately after the workshop for all participants, we found that there was no significant improvement in almost all aspects of knowledge about hand hygiene (P>0.05). Conclusion: Comparing the knowledge before and after the workshop, we found no significant improvement in almost all aspects of hand hygiene. Of note, nurses’ knowledge about hand hygiene was consistently higher than doctors.  Additional studies are needed to identify the optimal design of educational sessions.

The development and evaluation of mini-GEMs - short, focused, online e-learning videos in geriatric medicine

Mini Geriatric E-Learning Modules (Mini-GEMs) are short, focused, e-learning videos on geriatric medicine topics, hosted on YouTube, which are targeted at junior doctors working with older people. This study aimed to explore how these resources are accessed and used. The authors analyzed the viewing data from 22 videos published over the first 18 months of the Mini-GEM project. We conducted a focus group of U.K. junior doctors considering their experiences with Mini-GEMS. The Mini-GEMs were viewed 10,291 times over 18 months, equating to 38,435 minutes of total viewing time. The average viewing time for each video was 3.85 minutes. Learners valued the brevity and focused nature of the Mini-GEMs and reported that they watched them in a variety of settings to supplement clinical experiences and consolidate learning. Watching the videos led to an increase in self-reported confidence in managing older patients. Mini-GEMs can effectively disseminate clinical teaching material to a wide audience. The videos are valued by junior doctors due to their accessibility and ease of use.

Twelve tips to avoid ethical pitfalls when recruiting students as subjects in medical education research

Medical education research has unique characteristics that raise their own set of ethical issues, which differ significantly from those commonly found in clinical research. In contexts where researchers have a dual role as teachers, free consent to participate in research may be undermined and students' data must be kept confidential from faculty who play any role in their academic or professional path. Faculty members who recruit students as research subjects within their institution for education research should pay particular attention to ensure students' consent to participate is indeed free and continuous and that their privacy is adequately protected. A good understanding of ethical standards and of the appropriate strategies to fulfill them is essential to conduct ethical medical education research and to ensure ethics approval is obtained. These twelve tips draw from the Declaration of Helsinki, from the ICMJE recommendations and from the example of their application to medical education research in a Canadian and North American context. They aim to act as a reminder and as a guide to address the main ethical issues which should be given proper consideration when designing a study involving students as subjects for medical education research.

iCBLS: An interactive case-based learning system for medical education

Medical students should be able to actively apply clinical reasoning skills to further their interpretative, diagnostic, and treatment skills in a non-obtrusive and scalable way. Case-Based Learning (CBL) approach has been receiving attention in medical education as it is a student-centered teaching methodology that exposes students to real-world scenarios that need to be solved using their reasoning skills and existing theoretical knowledge. In this paper, we propose an interactive CBL System, called iCBLS, which supports the development of collaborative clinical reasoning skills for medical students in an online environment. The iCBLS consists of three modules: (i) system administration (SA), (ii) clinical case creation (CCC) with an innovative semi-automatic approach, and (iii) case formulation (CF) through intervention of medical students' and teachers' knowledge. Two evaluations under the umbrella of the context/input/process/product (CIPP) model have been performed with a Glycemia study. The first focused on the system satisfaction, evaluated by 54 students. The latter aimed to evaluate the system effectiveness, simulated by 155 students. The results show a high success rate of 70% for students' interaction, 76.4% for group learning, 72.8% for solo learning, and 74.6% for improved clinical skills.

Enhanced hospital-based learning at a medical school through application of management principles - a case study

BACKGROUND

A hospital with all its brimming activity constitutes a unique learning environment for medical students. However, to organise high-quality education within this context is a task of great complexity. This paper describes a teaching hospital case, where management principles were applied to enhance the learning quality of medical education.

METHODS

Traditional attempts from the faculty had been unsuccessful in improving learning among medical students at a teaching hospital. We therefore applied management principles to be able to improve the learning quality. An evaluation was performed from the perspectives of management (course directors/ heads of health care departments), medical students, and physician supervisors. Presages were defined, including educational resources and management; processes were adjusted, including learning activities and staff schedules; and products were assessed.

RESULTS

Charting and benchmarking the use of local educational resources identified unused funding. Structured recurrent collaboration within resource utilization was established between course directors and heads of all concerned health care departments. By formulating a joint agreement, the identified assets were used to reorganise the course, to create constructive alignment, and to increase assigned supervisor time. This resulted in a sustainable improvement of learning quality and culture.

CONCLUSION

By using management principles in combination with a scholarship of teaching and learning, it was possible to locate and redistribute educational resources in an effective way. This improved student learning and the learning culture of the health care departments. We propose that such an initiative could also be transferable to other contexts. Faculty leaders facing similar problems should consider the advantages of a structured collaboration with health care department heads.

Twelve tips for medical teaching in low-resource settings

Teaching in a low-resource setting comes with its own challenges and pitfalls. Many concepts, ideas and strategies can be transferred but need to be adapted to the different environment. This article highlights some of the challenges and obstacles that healthcare professionals working in this setting might encounter when setting up an educational intervention. The following twelve specific tips are aimed toward individual healthcare workers, independent charities, and smaller non-governmental organizations (NGOs) who wish to initiate small-scale teaching projects or larger educational ventures. The article covers general matters to consider prior to embarking on this journey and includes cultural, educational, linguistic, and social aspects.

Ten years of simulation-based training in pediatric anesthesia: The inception, evolution, and dissemination of the Managing Emergencies in Pediatric Anesthesia (MEPA) course

2016 marked the 10-year anniversary of the inception of the Managing Emergencies in Paediatric Anaesthesia (MEPA) course. This simulation-based program was originally created to allow trainees in pediatric anesthesia to experience operating room emergencies which although infrequent, would be considered key competencies for any practicing anesthetist with responsibility for providing care to children. Since its original manifestation, the course has evolved in content, scope, and worldwide availability, such that it is now available at over 60 locations on five continents. The content has been modified for different learner groups and translated into several languages. This article describes the history, evolution, and dissemination of the MEPA course to share lessons learnt with educators considering the launch of similar initiatives in their field.

Curated Collections for Educators: Five Key Papers about Program Evaluation

The evaluation of educational programs has become an expected part of medical education. At some point, all medical educators will need to critically evaluate the programs that they deliver. However, the evaluation of educational programs requires a very different skillset than teaching. In this article, we aim to identify and summarize key papers that would be helpful for faculty members interested in exploring program evaluation.

In November of 2016, the 2015-2016 Academic life in emergency medicine (ALiEM) Faculty Incubator program highlighted key papers in a discussion of program evaluation. This list of papers was augmented with suggestions by guest experts and by an open call on Twitter. This resulted in a list of 30 papers on program evaluation. Our authorship group then engaged in a process akin to a Delphi study to build consensus on the most important papers about program evaluation for medical education faculty.

We present our group’s top five most highly rated papers on program evaluation. We also summarize these papers with respect to their relevance to junior medical education faculty members and faculty developers.

Program evaluation is challenging. The described papers will be informative for junior faculty members as they aim to design literature-informed evaluations for their educational programs.

Twelve tips for planning and conducting a participatory evaluation

Participatory evaluation involves a partnership between program evaluators and stakeholders. This paper provides tips for planning and conducting a participatory evaluation of a medical education program. The tips highlight the need to recognize the importance of judgment in participatory evaluation, assess the appropriateness of participatory evaluation for the setting, determine a predominant stream of participatory evaluation, and select stakeholders for participation carefully. The tips also suggest that one should initiate participation at the program planning stage, engage a participatory evaluator, develop an evaluation framework, associate participatory evaluation with more than just qualitative methods, and use technology to facilitate participation. Furthermore, the tips illuminate that while individuals can use participatory evaluation to build evaluation capacity, it is important that they use three dimensions (i.e. control of decision-making, stakeholder selection, depth of participation) for determining the level of "participatory-ness," as well as publish and reflect on their use of participatory evaluation.

Roadmap for creating an accelerated three-year medical education program

Medical education is undergoing significant transformation. Many medical schools are moving away from the concept of seat time to competency-based education and introducing flexibility in the curriculum that allows individualization. In response to rising student debt and the anticipated physician shortage, 35% of US medical schools are considering the development of accelerated pathways. The roadmap described in this paper is grounded in the experiences of the Consortium of Accelerated Medical Pathway Programs (CAMPP) members in the development, implementation, and evaluation of one type of accelerated pathway: the three-year MD program. Strategies include developing a mission that guides curricular development - meeting regulatory requirements, attaining institutional buy-in and resources necessary to support the programs, including student assessment and mentoring - and program evaluation. Accelerated programs offer opportunities to innovate and integrate a mission benefitting students and the public.

ABBREVIATIONS

CAMPP: Consortium of accelerated medical pathway programs; GME: Graduate medical education; LCME: Liaison committee on medical education; NRMP: National residency matching program; UME: Undergraduate medical education.

Twelve tips for assessing surgical performance and use of technical assessment scales

Using validated assessment scales for technical competence can help structure and standardize assessment and feedback for both the trainee and the supervisor and thereby avoid bias and drive learning. Correct assessment of operative skills can establish learning curves and allow adequate monitoring. However, the assessment of surgical performance is not an easy task, since it includes many proxy parameters, which are hard to measure. Although numerous technical assessment scales exist, both within laparoscopic and open surgery, the validity evidence is often sparse, and this can raise doubts about reliability and educational outcome. Furthermore, the implementation of technical assessment scales varies due to several obstacles and doubts about accurate use. In this 12-tips article, we aim to give the readers a critical and useful appraisal of some of the common questions and misunderstandings regarding the use of surgical assessment scales and provide tips to ease and overcome potential pitfalls.

Lost in Transition: Examining Transitions in Psychotherapy Training

OBJECTIVE

Disruptions are inevitable during psychiatry residency training and can affect resident learning and patient care. This exploratory study examined the nature and impact of transitions in psychotherapy training.

METHODS

PGY2-5 residents (45/150; 30% response rate) and psychotherapy supervisors (46/247; 18.6% response rate) were surveyed about transitional events during residency training in psychotherapy.

RESULTS

Supervisors and residents ranked the frequency of occurrence of transitional events and their impact very similarly, as well as the "feed forward" items when transitioning to a new supervisor. Residents feeling confused or overwhelmed with the balancing of learning differing models with differing levels of comfort or knowledge was ranked as the issue that occurred most frequently by both supervisors and residents.

CONCLUSIONS

This study highlights issues that arise at transitions during psychotherapy training in psychiatry residency. Strategies for managing these periods are discussed, with a focus on resident learning and improved continuity of patient care.

Education scholarship in emergency medicine part 3: a “how-to” guide

Successful emergency medicine (EM) education scholarship requires a systematic approach that includes searching the (grey) literature, mobilizing resources, adopting frameworks to focus the innovation, integrating a component of program evaluation, and disseminating the innovation via traditional and emerging avenues. This paper provides direction for EM teachers and educators looking to transform their education innovation into scholarship. Recommendations on producing EM education scholarship from the 2013 consensus conference of the Academic Section of the Canadian Association of Emergency Physicians are presented.