RCT = results confounded and trivial: the perils of grand educational experiments

Is There a Problem With Evidence in Health Professions Education?

ABSTRACT

What constitutes evidence, what value evidence has, and how the needs of knowledge producers and those who consume this knowledge might be better aligned are questions that continue to challenge the health sciences. In health professions education (HPE), debates on these questions have ebbed and flowed with little sense of resolution or progress. In this article, the authors explore whether there is a problem with evidence in HPE using thought experiments anchored in Argyris' learning loops framework.From a single-loop perspective ("How are we doing?"), there may be many problems with evidence in HPE, but little is known about how research evidence is being used in practice and policy. A double-loop perspective ("Could we do better?") suggests expectations of knowledge producers and knowledge consumers might be too high, which suggests more system-wide approaches to evidence-informed practice in HPE are needed. A triple-loop perspective ("Are we asking the right questions?") highlights misalignments between the dynamics of research and decision-making, such that scholarly inquiry may be better approached as a way of advancing broader conversations, rather than contributing to specific decision-making processes.The authors ask knowledge producers and consumers to be more attentive to the translation from knowledge to evidence. They also argue for more systematic tracking and audit of how research knowledge is used as evidence. Given that research does not always have to serve practical purposes or address the problems of a particular program or institution, the relationship between knowledge and evidence should be understood in terms of changing conversations and influencing decisions.

Improving Health Professional Recognition and Response to Child Maltreatment and Intimate Partner Violence: Protocol for Two Mixed Methods Pilot Randomized Controlled Trials

BACKGROUND

The optimal educational approach for preparing health professionals with the knowledge and skills to effectively recognize and respond to family violence, including child maltreatment and intimate partner violence, remains unclear. The Violence, Evidence, Guidance, and Action (VEGA) Family Violence Education Resources is a novel intervention that can be completed via self-directed learning or in a workshop format; both approaches focus on improving health professional preparedness to address family violence.

OBJECTIVE

Our studies aim to determine the acceptability and feasibility of conducting a randomized controlled trial to evaluate the effectiveness of the self-directed (experimental intervention) and workshop (active control) modalities of VEGA, as an adjunct to standard education, to improve learner (Researching the Impact of Service provider Education [RISE] with Residents) and independent practice (RISE with Veterans) health professional preparedness, knowledge, and skills related to recognizing family violence in their health care encounters.

METHODS

The RISE with Residents and RISE with Veterans research studies use embedded experimental mixed methods research designs. The quantitative strand for each study follows the principles of a pilot randomized controlled trial. For RISE with Residents, we aimed to recruit 80 postgraduate medical trainees; for RISE with Veterans, we intended to recruit 80 health professionals who work or have worked with Veterans (or their family members) of the Canadian military or the Royal Canadian Mounted Police in a direct service capacity. Participants complete quantitative assessments at baseline, after intervention, and at 3-month follow-up. A subset of participants from each arm also undergoes a qualitative semistructured interview with the aim of describing participants' perceptions of the value and impact of each VEGA modality, as well as research burden. Scores on potential outcome measures will be mapped to excerpts of qualitative data via a mixed methods joint display to aid in the interpretation of findings.

RESULTS

We consented 71 individuals to participate in the RISE with Residents study. Data collection was completed on August 31, 2023, and data are currently being cleaned and prepared for analysis. As of January 15, 2024, we consented 34 individuals in the RISE with Veterans study; data collection will be completed in March 2024. For both studies, no data analysis had taken place at the time of manuscript submission. Results will be disseminated through peer-reviewed publications; academic conferences; and posting and sharing of study summaries and infographics on social media, the project website, and via professional network listserves.

CONCLUSIONS

Reducing the impacts of family violence remains a pressing public health challenge. Both research studies will provide a valuable methodological contribution about the feasibility of trial methods in health professions education focused on family violence. They will also contribute to education science about the differences in the effectiveness of self-directed versus facilitator-led learning strategies.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05490121, https://clinicaltrials.gov/study/NCT05490121; ClinicalTrials.gov NCT05490004, https://clinicaltrials.gov/study/NCT05490004.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/50864.

Comparison of multimodal active learning and single-modality procedural simulation for central venous catheter insertion for incoming residents in anesthesiology: a prospective and randomized study

BACKGROUND

Active learning methods, including low-fidelity simulation, are useful but the incremental learning effect of each method is however limited. We designed this study to assess if combining flipped classroom and the modified Peyton's « 4-steps» method during procedural simulation (intervention group [IG]) would provide better learning results than simulation alone (control group [CG]) in the context of central venous catheter insertion training.

METHODS

This prospective, single-center, and randomized study took place in 2017 in a single simulation center. All first year Anesthesiology residents of Ile de France area at the start of their residency were randomly included either in the IG or CG during a seminar aimed at providing initial procedural skills with low-fidelity simulation. A composite learning score which included knowledge MCQ and a questionnaire assessing satisfaction and value of the training session was recorded after training (primary outcome, /100). A randomized sub-group of learners of each group were video-recorded during central venous catheter insertion at the end of training and their skills were evaluated with validated tools, including a specific checklist and a global rating scale (GRS).

RESULTS

Among 89 anesthesiology residents, 48 and 41 learners were randomized in the intervention and control groups respectively. Of the IG residents, 40/48 (83%) had read the learning material prior to the session. There was no significant difference regarding the composite outcome ([IG]= 81.1 vs [CG] = 80.5 /100 (p = 0.68)). Results of the post-session MCQ knowledge questionnaire were also non-significantly different. Residents were similarly satisfied and described a significant improvement of their knowledge and skills after training. Learners highly valued the training session as a mean to improve their future practice. No significant differences regarding checklist and GRS scores were observed.

CONCLUSIONS

A multimodal active learning strategy of procedural learning did not provide better learning outcomes when compared to a traditional simulation method. In both groups, satisfaction was high and perception of the acquired theoretical and practical knowledge was improved after training.

Can a Brief Online Interdisciplinary Program Enhance the Understanding of Musculoskeletal Pain amongst Primary Care Staff? A Prospective Study

Background

Primary care staff do not provide consistent education on musculoskeletal pain management to patients in accordance with the recommendations of clinical practice guidelines. We have developed a concise online learning program to bridge this gap.

Objectives

To investigate (1) the effectiveness of 1-hour musculoskeletal pain neurophysiology education program conducted by a physiotherapist on primary care staff; (2) the correlation between demographic factors of staff and the interdisciplinary learning performances.

Methods

We piloted a multicenter single-blind prospective study on sixty-four staff from the pharmacy department in eight public primary care clinics. Participants in the intervention group attended the 1-hour online program compared to a control group. The Neurophysiology of Pain Questionnaire (NPQ) to assess learning and the 10-point Likert scale program evaluation form to assess learning reaction of participants after the program were used.

Results

The participants reacted positively to the online learning program. The intervention group significantly improved in their musculoskeletal pain knowledge by a greater mean NPQ score difference 2.39 ( p < 0.001) compared with the control group. There were poor correlations between the demographic factors and their learning.

Conclusion

The pilot study shows that primary care staff may still utilize a biomedical approach in managing musculoskeletal pain. The improvement in knowledge demonstrates that short online programs could be a valuable part of interdisciplinary education in primary care because it is easily accessible by healthcare professionals and can benefit other healthcare staff regardless of their background.

Costs and Economic Impacts of Physician Continuous Professional Development: A Systematic Scoping Review

PURPOSE

Nearly all health care professionals engage in continuous professional development (CPD), yet little is known about the cost and cost-effectiveness of physician CPD. Clarification of key concepts, comprehensive identification of published work, and determination of research gaps would facilitate application of existing evidence and planning for future investigations. The authors sought to systematically map study themes, methods, and outcomes in peer-reviewed literature on the cost and value of physician CPD.

METHOD

The authors conducted a scoping review, systematically searching MEDLINE, Embase, PsycInfo, and Cochrane Library databases for comparative economic evaluations of CPD for practicing physicians through April 2020. Two reviewers, working independently, screened all articles for inclusion. Three reviewers iteratively reviewed all included articles to inductively identify key features including participants, educational interventions, study designs, cost ingredients, and cost analyses. Two reviewers then independently reexamined all included articles to code these features.

RESULTS

Of 3,338 potentially eligible studies, 111 were included. Physician specialties included internal, family, or general medicine (80 studies [72%]), surgery (14 studies [13%]), and medicine subspecialties (7 studies [6%]). Topics most often addressed general medicine (45 studies [41%]) or appropriate drug use (37 studies [33%]). Eighty-seven studies (78%) compared CPD with no intervention. Sixty-three studies (57%) reported the cost of training, and 79 (71%) evaluated the economic impact (money saved/lost following CPD). Training cost ingredients (median 3 itemized per study) and economic impact ingredients (median 1 per study) were infrequently and incompletely identified, quantified, or priced. Twenty-seven studies (24%) reported cost-impact expressions such as cost-effectiveness ratio or net value. Nineteen studies (17%) reported sensitivity analyses.

CONCLUSIONS

Studies evaluating the costs and economic impact of physician CPD are few. Gaps exist in identification, quantification, pricing, and analysis of cost outcomes. The authors propose a comprehensive framework for appraising ingredients and a preliminary reference case for economic evaluations.

Establishing the effectiveness of technology-enabled dementia education for health and social care practitioners: a systematic review

BACKGROUND

Dementia prevalence is increasing globally and yet evidence suggest that gaps exist in dementia-specific knowledge among health and social care practitioners. Technological modes of educational delivery may be as effective as traditional education and can provide practitioners with increased accessibility to dementia training. Benefits of digitally based dementia education have been established including pedagogical strategies that influence dementia knowledge and care attitudes. This review aimed to appraise and synthesise contemporary experimental evidence that evaluated technology-enabled dementia education for health and social care practitioners. Outcomes based on Kirkpatrick's Model were learner satisfaction; knowledge, skills, and attitudes; behaviours; and results.

METHODS

MEDLINE, CINAHL, and Web of Science were among 8 bibliographic databases searched from January 2005 until February 2020. Keywords included dementia and education (and terms for technological modes of education, learning, or training). We included experimental and quasi-experimental studies. Medical Education Research Study Quality Instrument established the overall quality of included studies and pragmatic application of Mixed Methods Appraisal Tool established individual study quality and highlighted methodological features of educational research. Narrative synthesis was conducted as heterogeneous outcome data precluded meta-analysis.

RESULTS

We identified 21 relevant studies: 16 evaluated online dementia education and 5 evaluated computer-based approaches. Most studies used before-after designs and study quality was moderate overall. Most studies reported knowledge-based outcomes with statistically significant findings favouring the training interventions. Positive effects were also observed in studies measuring skills and attitudinal change. Fewer studies reported significant findings for behavioural change and results due to training. Case-based instruction was a frequently described instructional strategy in online dementia education and videos were common information delivery modes. CD-ROM training and simulation activities were described in computer-based dementia education.

DISCUSSION

Future emphasis must be placed on teaching and learning methods within technology-enabled dementia education which should be role relevant and incorporate active and interactive learning strategies. Future evaluations will require contextually relevant research methodologies with capacity to address challenges presented by these complex educational programmes and multi-component characteristics.

SYSTEMATIC REVIEW REGISTRATION

This systematic review is based on a protocol registered with PROSPERO ( CRD42018115378 ).

The Development of a Video Intervention to Improve Senior Medical Students' Performance on Outpatient Telephone Encounters: a Delphi Analysis and Randomized Controlled Trial

Introduction

Postgraduate trainees address outpatient telephone calls (OTCs) with little prior training. This study determines the skills necessary for OTCs and examines whether a video intervention improves medical students' performance on simulated OTCs.

Materials and Methods

We utilized a Delphi technique to determine skills needed for OTCs and created a 9-min video teaching these skills. Senior medical students were randomized to Intervention (viewed video) and Control (did not view video) groups. Students were assessed pre-/post-intervention on simulated OTCs. The primary outcome was the between-group difference in improvement.

Results

The Delphi yielded 34 important skills with the highest focus on communication (n = 13) and triage (n = 6). Seventy-two students completed assessments (Control, n = 41; Intervention, n = 31). The score (mean ± SD) improved 4.3% in the Control group (62.3 ± 14.3% to 66.6 ± 25.0%) and 12.2% in the Intervention group (60.7 ± 15.2% to 72.9 ± 20.4%, p = 0.15). The effect size measured by Cohen's d was 0.55, considered effective (> 0.33) for an educational intervention.

Conclusions

This project fills a gap in OTC training. The use of the Delphi technique, intervention development based on the results, and evaluation of efficacy is a process that could be reproduced for other educational gaps.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-021-01331-w.

Health professions education scholarship: The emergence, current status, and future of a discipline in its own right

Medical education, as a domain of scholarly pursuit, has enjoyed a remarkably rapid development in the past 70 years and is now more commonly known as health professions education (HPE) scholarship. Evidenced by a solid increase of publications, numbers of specialized journals, professional associations, national and international conferences, academies for medical educators, masters and doctoral courses, and the establishment of many units of HPE scholarship, the domain of HPE education scholarship has matured into a scholarly discipline in its own right. In this contribution, the author reviews the developments of the field from Boyer's four criteria that determine scholarship: discovery, integration, application, and teaching. Born mid-20th century, and in the first decades developed in the predominant area of physician education, HPE scholarship has matured, with increasing breadth, depth, and volume of scholars, publications, conferences, and dedicated centers for research and development. The author concludes that, given the infrastructure that has emerged, HPE can arguably be considered a discipline in its own right. This academic question may not matter hugely for practices of scholarly work in this domain, and any stance in this academic debate inevitably reflects a personal view, but the author would support the view of health professions scholarship as being a unique niche, with inherent dependence on both medical and other health professional sciences, on the one hand, and social sciences, including educational sciences, on the other hand.

Internal coherence matters: Lessons for nutrition and dietetics research

Aim

Internal coherence in research refers to the alignment between ontology (nature of reality), epistemology (nature of knowledge), axiology (values), methodology and methods and is an important but often overlooked element of research quality. We therefore aimed to illustrate the concept of internal coherence in nutrition and dietetics research, and its importance beyond individual elements of study quality.

Method

A targeted literature search in Nutrition and Dietetics was used to identify research illustrating one example of three main approaches to research (scientific, interpretive and critical inquiry) published between November 2017 and November 2020. Studies were included if they related to education research based on the expertise of the authors, and illustrated diverse points about internal coherence. The authors independently critiqued included studies for internal coherence and synthesised their findings.

Results

From 76 manuscripts, 14 were identified as describing education research. Of the three selected studies that were critiqued, all had elements of internal coherence, in particular alignment between epistemology and methodology. However, each had elements of misalignment too, specifically between epistemology, axiology and method. The results point to the profession's historical groundings privileging the scientific approach, showing how this can yield misalignments, particularly when describing the limitations of interpretive and critical inquiry approaches.

Conclusion

This review demonstrates the importance of internal coherence as a marker of quality, over and above existing quality assessment checklists for qualitative and quantitative methodologies. As such, it can help authors, reviewers and editors to improve the quality of nutrition and dietetics research and its reporting.

Synchronous distance education vs traditional education for health science students: A systematic review and meta-analysis

CONTEXT

Synchronous distance education (SDE) has been widely used for health science students in recent years. This study examined the effectiveness and acceptance of SDE compared with traditional education for health science students and explored the potential moderators that could impact the pooled results.

METHODS

A systematic review and meta-analysis was conducted of randomised controlled trials (RCTs) from January 2000 to March 2020 searched on nine electronic databases, including Web of Science, PubMed, Cochrane Library, Scopus, EMBASE, CINAHL, ERIC, PsycINFO, and ProQuest Dissertations and Theses. The outcomes measured were knowledge, skills with objective assessments and overall satisfaction with subjective evaluations. The pooled results were calculated using random-model effects, and moderators were explored through meta-regression.

RESULTS

A total of seven RCTs with 594 participants were included. At the post-test level, the pooled effect size of knowledge acquisitions (SMD 0.12, 95% CI -0.07-0.32) showed insignificant difference between the SDE and traditional education groups (P = .207), with low heterogeneity (I²  = 17.6%). Subgroup analyses observed no factors that significantly impacted the pooled results of knowledge acquisition at the post-test levels (P for interaction > 0.05). Knowledge gains from pretest to post-test in SDE groups also did not differ significantly between groups (SMD 0.15, 95% CI -0.22-0.53; P = .428). The pooled effect size of skills (SMD 0.02, 95% CI -0.24-0.28; P = .735) was similarly insignificant. The pooled effect size of overall satisfaction (SMD 0.60, 95% CI 0.38-0.83; P < .001) significantly favoured SDE over traditional education. Incorporating two-group studies without randomisations did not significantly change the overall results of knowledge acquisition at the post-test level (SMD -0.002, 95% CI -0.11-0.10; P = .994), with moderate heterogeneity (I²  = 61.9%).

CONCLUSIONS

Synchronous distance education was not significantly different from traditional education in effectiveness and had higher satisfaction ratings. Our findings might provide indications for adoptions of online remote education in health science education centres.

The myth of ivory tower versus practice-oriented research: A systematic review of randomised studies in medical education

CONTEXT

A long-standing myth in medical education research is a divide between two different poles: research aiming to advance theory with little focus on practical applications ('ivory tower' research) and practically oriented research aiming to serve educators and decision-makers with little focus on advancing theory ('in-the-trenches' practice). We explored this myth in a sample of randomised medical education studies using Stokes' four-quadrant framework for the classification of research perspective.

METHODS

We searched MEDLINE, Embase, CINAHL, PsychINFO, ERIC, Web of Science and Scopus for studies in medical education using a randomised design that were published between 1 January 2018 and 31 December 2018. We used Stokes' four-quadrant framework to categorise the studies according to their use of theory, concepts and their justification for practical use. We compared medical education research published in medical education journals and clinical journals.

RESULTS

A total of 150 randomised studies were included in the analysis. The largest segment of studies (46.7%) was categorised as use-inspired basic research (Pasteur's Quadrant), closely followed by pure applied research (40.7%, Edison's Quadrant). Only a few studies were categorised as aiming to advance knowledge with no thought for practical educational application (2.0%, Bohr's Quadrant). The proportion of studies that included educational concepts and theory differed according to publication in clinical journals or medical education journals: 40.5% vs 71.8%, respectively, P < .001. There were no differences between journals with regard to the proportion of studies that included a practical educational or clinical rationale (P = .99).

CONCLUSION

In a large sample of studies using randomised designs, we found no evidence to support the myth that medical education research divides between two singular poles represented by 'ivory tower research' and 'in-the-trenches practice'. We did confirm prevailing assumptions regarding an emphasis on non-theoretical medical education research in clinical journals.

Effect of mental rehearsal on team performance and non-technical skills in surgical teams: systematic review

BACKGROUND

Simulation-based training in medical education has become a common method to develop both technical and non-technical skills in teams. Mental rehearsal (MR) is the cognitive act of simulating a task in our heads to pre-experience tasks imaginatively. It has been used widely to improve individual and collective performance in fields outside healthcare, and offers potential for more efficient training in time-pressured surgical and medical team contexts. This study aimed to review the available literature to determine the impact of MR on team performance and non-technical skills in healthcare.

METHODS

MEDLINE, Embase, British Educational Index, CINAHL, Web of Science, PsycInfo and Cochrane databases were searched for the period 1994-2018. The primary outcome measure was improvement in team performance and non-technical skills. Study quality of RCTs was assessed using the Medical Education Research Quality Instrument. The reported impacts of MR in all included studies were mapped on to the Kirkpatrick framework for evaluation of educational interventions.

RESULTS

Eight studies with 268 participants were identified that met the inclusion criteria, of which there were six randomized trials, one prospective pragmatic trial and one qualitative study. Three studies found MR to be effective in improving team non-technical skills. MR practices were varied and often poorly defined. MR benefited team non-technical skills when it was specifically designed to do so, but was not an automatic consequence of technical MR alone. The majority of studies demonstrated benefits of MR for technical performance, but only three showed positive impacts on teamwork. Overall the studies were of low quality and lacked sufficient discriminatory focus to examine impacts on teamwork dynamics.

CONCLUSION

MR can improve technical performance, but the benefits on non-technical skills are less clear. Future research should look at longitudinal mixed-method evaluation designs and focus on real clinical teams.

Interpreting the effectiveness of a summer reading program: The eye of the beholder

In applying a methods-oriented approach to evaluation, this study interpreted the effectiveness of a summer reading program from three different stakeholder perspectives: practitioners from the school district, the funding agency supporting the program, and the policymakers considering mandating summer school. Archival data were obtained on 2330 students reading below benchmark in Grades 2-5. After propensity score matching participants to peers who did not attend the summer program, the final sample consisted of 630 students. Pre-to-posttest growth models revealed positive effects in Grades 2-4 (standardized slopes of .40-.54), but fifth graders demonstrated negligible improvement (standardized slope of .15). The standardized mean differences of propensity score matched treatment and control group students indicated null effects in all grade levels (d = -.13 to .05). Achieving proficient reading performance also was not attributable to summer school participation. Findings underscore the importance of operationalizing effectiveness in summative evaluation.

Beyond the Medical Model: Thinking Differently about Medical Education and Medical Education Research

Issue: In medical education, teaching is currently viewed as an intervention that causes learning. The task of medical education research is seen as establishing which educational interventions produce the desired learning outcomes. This 'medical model' of education does not do justice to the dynamics of education as an open, semiotic, recursive system rather than a closed, causal system. Evidence: Empirical 'evidence' of 'what works' - that is, what is supposed to affect 'learning' - has become the norm for medical educational improvements, where generalized summary outcomes of research are often presented as must-follow guidelines for myriad future educational situations. Such investigations of educational processes tend to lack an explicit engagement with the purposes of medical education, which we suggest to understand in terms of qualification (the acquisition of knowledge, skills, and understanding), socialization (becoming a member of the professional group) and subjectification (becoming a thoughtful, independent, responsible professional). In addition, investigations of educational processes tend to rely on causal assumptions that are inadequate for capturing the dynamics of educational communication and interaction. Although we see an increasing acknowledgement of the context-dependency of teaching practices toward educational aims, the currently prevailing view in medical education and educational research limits understanding of what is actually going on when educators teach and students participate in medical education - a situation which seriously hinders advancements in the field. Implications: In this paper, we hope to inform discussion about the practice of medical education by proposing to view medical education in terms of three domains of purpose (professional qualification, professional socialization, and professional subjectification) and with full acknowledgement of the dynamics of educational interaction and communication. Such a view implies that curriculum design, pedagogy, assessment, and evaluation should be reoriented to include and integrate all three purposes in educational practice. It also means that medical education research findings cannot be applied in just any teaching context without carefully considering the value of the suggested courses of actions toward the particular educational aims and teaching setting. In addition, medical educational research would need to investigate all three purposes and recognize the openness, semiotic nature, and recursivity of education in offering implications for teaching practice.

Critical analysis of evidence about the impacts on surgical teams of 'mental practice' in systematic reviews: a systematic rapid evidence assessment (SREA)

BACKGROUND

Mental Rehearsal (MR) the cognitive act of simulating a task in our heads to pre-experience events imaginatively. It has been used widely to improve individual and collective performance in fields outside healthcare and offers potential for more efficient training in time pressured surgical and medical team contexts. The study aims to review the current systematic review literature to determine the impact of MP on surgical performance and learning.

METHODS

Medline, Embase, British Educational Index, CINAHL, Web of Science PsycINFO, Cochrane databased were searched in the period 1994-2018. The primary outcomes measure were performance improvements in surgical technical skills, stress reduction, confidence and team performance. Study quality of the Systematic Reviews was assessed using AMSTAR 2, a critical appraisal tool for systematic reviews. The reported impacts of MP in all included studies were mapped onto Kirkpatrick's framework for the evaluation of educational interventions.

RESULTS

Six Systematic reviews were identified which met the inclusion criteria, of which all reported positive and varying benefits of MP on surgical performance, confidence, and coping strategies. However, reported impacts on a modified Kirkpatrick's framework did not exceed level 3. Mental practice was described in terms of mental imagery and mental rehearsal with most authors using each of the terms in their search strategies. The impacts on transfer to practice and the long- term acquisition of skills, but also personal uptake of mental practice routines were not reported.

CONCLUSION

The majority of studies demonstrate benefits of MP for technical performance. Overall the systematic reviews were of medium to high quality. However, studies lacked a sufficiently articulated evaluation methodology to examine impacts beyond the immediate experimentations. This is also due to the limitations found in the primary studies. Future research should look at longitudinal mixed method evaluation designs and focus on real clinical teams.

Fundamentals of randomized designs: AMEE Guide No. 128

This AMEE Guide summarizes fundamentals of a major experimental design option for medical education researchers: the randomised study. Medical education researchers face an overwhelming taxonomy of study design options; given the breadth of information on experimental design, the purpose of this Guide is to offer a resource for medical education researchers wishing to equip themselves with helpful information for when to match a study's objective and the use of randomised designs. Once a research question has been formulated study design is the cornerstone of the intricate, nested activities of any research project. Researchers negotiate many decisions in the pursuit of choosing an appropriate design approach; failure to do so can undermine a project's capacity to, for example, sufficiently test a hypothesis or theory. Written as an introduction, this Guide is intended for medical education researchers seeking to build on and synthesise the existing corpus of literature on experimental and quasi-experimental design approaches. While not comprehensive, presented are key concepts alongside relevant examples from the field of health professions education.

Blended learning of radiology improves medical students' performance, satisfaction, and engagement

Purpose

To evaluate the impact of blended learning using a combination of educational resources (flipped classroom and short videos) on medical students’ (MSs) for radiology learning.

Material and methods

A cohort of 353 MSs from 2015 to 2018 was prospectively evaluated. MSs were assigned to four groups (high, high-intermediate, low-intermediate, and low achievers) based on their results to a 20-MCQs performance evaluation referred to as the pretest. MSs had then free access to a self-paced course totalizing 61 videos based on abdominal imaging over a period of 3 months. Performance was evaluated using the change between posttest (the same 20 MCQs as pretest) and pretest results. Satisfaction was measured using a satisfaction survey with directed and spontaneous feedbacks. Engagement was graded according to audience retention and attendance on a web content management system.

Results

Performance change between pre and posttest was significantly different between the four categories (ANOVA, P = 10⁻⁹): low pretest achievers demonstrated the highest improvement (mean ± SD, + 11.3 ± 22.8 points) while high pretest achievers showed a decrease in their posttest score (mean ± SD, − 3.6 ± 19 points). Directed feedback collected from 73.3% of participants showed a 99% of overall satisfaction. Spontaneous feedback showed that the concept of “pleasure in learning” was the most cited advantage, followed by “flexibility.” Engagement increased over years and the number of views increased of 2.47-fold in 2 years.

Conclusion

Learning formats including new pedagogical concepts as blended learning, and current technologies allow improvement in medical student’s performance, satisfaction, and engagement.

A pragmatic randomised controlled trial of SAFMEDS to produce fluency in interpretation of electrocardiograms

BACKGROUND

SAFMEDS (Say-All-Fast-Minute-Every-Day-Shuffled) is a flashcard-type behavioural instructional methodology, involving one-minute learning trials that function both as practice and assessment, used to facilitate the development of fluency in a behaviour. The primary research question was whether SAFMEDS engenders improvement in performance beyond that conferred by usual teaching. A secondary research question was whether SAFMEDS is an effective method of producing fluency in Electrocardiogram (ECG) interpretation.

METHODS

A pilot study was conducted to determine sample size required to power the pragmatic randomised controlled trial (RCT). For the subsequent RCT, participants were randomly assigned to a "usual teaching" control group (n = 14) or the SAFMEDS intervention group (n = 13), with the recognition of 15 cardiac conditions on ECGs (e.g., atrial fibrillation, complete heart block) targeted. Intervention group participants' performance was tracked over eight weeks as they worked towards achieving the fluency criterion. Percentage accuracy in ECG interpretation was assessed at baseline and post-test for both groups. An ANCOVA was conducted to assess for differences in the performance of the intervention and control group at post-test while controlling for the baseline performance of participants. At post-test, the numbers of participants achieving fluency within the intervention group was examined.

RESULTS

A large effect size of SAFMEDS (partial η2 = .67) was identified when controlling for the effects of baseline performance. At post-test, the intervention group significantly outperformed (M = 61.5%; SD = 12.1%) the control group (M = 31.6%; SD = 12.5%, p < .001). In total, 7 of 13 intervention group participants achieved fluency. Participants required an average of 51.9 one-minute trials (SD = 18.8) to achieve fluency.

CONCLUSIONS

SAFMEDS offers a useful adjunct to usual teaching within medical education. Further research could assess whether learning retains, is stable, and transfers to clinical practice.

Improving Outcomes

OBJECTIVES

To determine the strength of study design and outcomes in literature describing pathology education for medical students.

METHODS

A search was conducted for articles related to pathology education published over 45 years describing an educational intervention. The primary data collected included phase of education, domain of objectives, number of learners and institutions, type of intervention, use of a comparison/control group, randomization, and strength of statistical analysis.

RESULTS

Of 655 articles, 54 (8%) met inclusion criteria. The majority (65%) reported experiences of 100 learners or fewer, and only one was multi-institutional. Only 46% used a comparison/control group. Statistical significance of results was not reported in 39%. None examined outcomes at a point distant from the educational intervention.

CONCLUSIONS

Most studies describing pathology educational interventions are not of strong experimental design. Consumers of educational research should be cognizant of these potential weaknesses in educational studies.

Maximizing the acquisition of core communication skills at the start of medical training

BACKGROUND

Clinical communication teaching for medical undergraduates may involve real patient contact alongside simulated patient (SP) contact. However, there is still comparatively little known about the experience of learning with real patients and how that may impact on the SP encounter.

AIM

To explore the impact of real patient contact on the experience of communication skills training and SP contact for first-year medical undergraduate students.

METHODS

As part of the 6-year MBBS undergraduate medical degree at Imperial College London, students are obliged to undertake communication skills training, which involves teaching with simulated and real patients. In 2017 (toward the end of formal teaching), a small sample of Year 1 medical students, who had taken part in extra-curricular teaching with real patients were recruited for the study to compare their performance with a control group in a SP encounter. The performance of both groups was analyzed alongside follow-up focus group data from a sample of the study group.

RESULTS

Quantitative analysis revealed there was no significant difference in communication skills during a scored SP interview between students with real patient contact and those without. Focus group data, however, revealed valuable insights into the experience of learning with real patients. Students reported a marked increase in their confidence and ability to naturalize their communication skills as a result of real patient contact. Students also reported that skills gained through real patient contact may not always transfer easily to the SP setting.

CONCLUSION

Real patient contact is an invaluable component of communication training for undergraduate medical students. For successful implementation, there needs to be a clear curricular purpose at pedagogical, practical and organizational levels. Students' experience of real patient contact can provide an informed foundation upon which to implement other modes of teaching.

Evaluation of a Preclerkship Learning Community Model for Delivering Clinical Skills Curriculum

BACKGROUND

Medical schools are increasingly using learning communities (LCs) for clinical skills curriculum delivery despite little research on LCs employed for this purpose. We evaluated an LC model compared with a non-LC model for preclerkship clinical skills curriculum using Kirkpatrick's hierarchy as an evaluation framework.

METHODS

The first LC cohort's (N = 101; matriculating Fall 2013) reaction to the LC model was assessed with self-reported surveys. Change in skills and learning transfer to clerkships was measured with objective structured clinical examinations (OSCEs) at the end of years 2 and 3 and first and last clerkship preceptor evaluations; the LC cohort and the prior cohort (N = 86; matriculating Fall 2012) that received clinical skills instruction in a non-LC format were compared with Mann-Whitney U tests.

RESULTS

The LC model for preclerkship clinical skills curriculum was rated as excellent or good by 96% of respondents in Semesters 1 to 3 (N = 95). Across multiple performance domains, 96% to 99% of students were satisfied to very satisfied with their LC faculty preceptors (N varied by item). For the end of preclerkship OSCE, the LC cohort scored higher than the non-LC cohort in history gathering (P = .003, d = 0.50), physical examination (P = .019, d = 0.32), and encounter documentation (P ⩽ .001, d = 0.47); the non-LC cohort scored higher than the LC cohort in communication (P = .001, d = 0.43). For the end of year 3 OSCE, the LC cohort scored higher than the non-LC cohort in history gathering (P = .006, d = 0.50) and encounter note documentation (P = .027, d = 0.24); there was no difference in physical examination or communication scores between cohorts. There was no detectable difference between LC and non-LC student performance on the preceptor evaluation forms at either the beginning or end of the clerkship curriculum.

CONCLUSIONS

We observed limited performance improvements for LC compared with non-LC students on the end of the preclerkship OSCE but not on the clerkship preceptor evaluations. Additional studies of LC models for clinical skills curriculum delivery are needed to further elucidate their impact on the professional development of medical students.

The Quality Of Evidence In Preclinical Medical Education Literature: A Systematic Review

INTRODUCTION

To practice effective evidence-based teaching, the need for well-designed studies that describe outcomes related to educational interventions is critical. The quality of the literate in basic science disciplines is unknown. The study objective was to conduct a systematic review of the literature to assess study design in articles describing innovations in preclinical medical education.

METHOD

The authors searched PubMed for all articles published in English between 2000 and 2017 describing interventions in preclinical medical education related to anatomy, physiology, and biochemistry. Articles were scored using a modification of the Medical Education Research Study Quality Instrument.

RESULTS

Of the 817 articles identified, 177 met final inclusion criteria (75 anatomy, 86 physiology, and 16 biochemistry). Laboratory, student-driven, and online activities were the most frequently reported. The average score for all papers was 15.7 (27 points possible). More than 80% reported experiences with one cohort of students and >97% involved only one institution. Only 25-49% of reports utilized a comparison (control) group. Proper statistical models for analysis of results were used in only 44-62% of papers.

CONCLUSION

Manuscripts had a strong tendency toward single institutional studies that involved one cohort of students. The use of a control/comparison group when assessing effectiveness was seen in <50% and nearly all reported outcomes solely in the form of student satisfaction or factual recall/skill performance.

Clinical reasoning skills in final-year dental students: A qualitative cross-curricula comparison

AIMS

The aim of this research was to explore the perceptions of undergraduate dental students regarding clinical reasoning skills and also discover the influences of different curriculum designs on the acquisition of these skills by students.

METHODS

Eighteen final-year students from three different dental schools with varied curricula and cultures participated in the current research. The research used qualitative methodology. The study took place in 2013-2014. Interviews captured the participants' own understanding of clinical reasoning and its acquisition plus they "talked through" a clinical problem using a "think-aloud" technique. Thematic analysis was used to analyse the transcripts of the recorded interviews. Results obtained were related to curriculum structure.

RESULTS

Unfamiliarity with the term clinical reasoning was common in students. Students from different schools used different strategies to reason when discussing clinical vignettes. Clinical reasoning process was dominated by pattern recognition. Students' behaviours seemed to be influenced by cultural factors.

CONCLUSIONS

This research contributes to a greater understanding of how students learn, understand and apply dental clinical reasoning which will improve educational practices in the future.

What We Do and Do Not Know about Teaching Medical Image Interpretation

Educators in medical image interpretation have difficulty finding scientific evidence as to how they should design their instruction. We review and comment on 81 papers that investigated instructional design in medical image interpretation. We distinguish between studies that evaluated complete offline courses and curricula, studies that evaluated e-learning modules, and studies that evaluated specific educational interventions. Twenty-three percent of all studies evaluated the implementation of complete courses or curricula, and 44% of the studies evaluated the implementation of e-learning modules. We argue that these studies have encouraging results but provide little information for educators: too many differences exist between conditions to unambiguously attribute the learning effects to specific instructional techniques. Moreover, concepts are not uniformly defined and methodological weaknesses further limit the usefulness of evidence provided by these studies. Thirty-two percent of the studies evaluated a specific interventional technique. We discuss three theoretical frameworks that informed these studies: diagnostic reasoning, cognitive schemas and study strategies. Research on diagnostic reasoning suggests teaching students to start with non-analytic reasoning and subsequently applying analytic reasoning, but little is known on how to train non-analytic reasoning. Research on cognitive schemas investigated activities that help the development of appropriate cognitive schemas. Finally, research on study strategies supports the effectiveness of practice testing, but more study strategies could be applicable to learning medical image interpretation. Our commentary highlights the value of evaluating specific instructional techniques, but further evidence is required to optimally inform educators in medical image interpretation.

Scoping systematic review on the extent, nature and quality of evidence underlying ophthalmic and paraophthalmic education

BACKGROUND

Effective education of relevant professionals underpins provision of quality eye healthcare.

OBJECTIVES

This scoping systematic review had 2 aims: first to investigate the extent and nature of scholarly output published on ophthalmic and paraophthalmic education, and second to focus on the quality of reporting of randomised controlled trials (RCTs) identified.

STUDY SELECTION

A search strategy was created and applied to PubMed. Any scholarly publications on any aspect of education of those involved in the care of patients with visual problems as the main theme or context was selected. Predefined data were extracted.

FINDINGS

Of 255 studies included, the most common type of scholarly publications were descriptions of an educational innovation, opinion pieces and descriptive studies. RCTs made up 5.5% of the sample. Most of the 14 RCTs failed to report most of the items recommended in the CONSORT guidelines.

CONCLUSIONS

This review highlights the need for investigators, ethical committees and journals to insist on a better quality of RCT conduct than is presently apparent, but also that clinicians should not be blind to the strengths of non-RCT-based studies in the field of education.

Design, analysis, and conclusions: Telling a consistent causal story

Pharmacy educators are dedicated to providing the best education to student pharmacists and post-graduate trainees. This involves developing and evaluating new educational approaches or activities, as well as working to understand better other factors that may affect student and program outcomes. Although we may not realize it, the idea of causality, or the presence of a cause-and-effect relationship, is embedded in much of what we do when engaging in pharmacy educational scholarship. Saying that an educational intervention is effective at improving test scores implies that the intervention caused the increase. Perhaps more subtly, a cause-and-effect relationship is implied when identifying poor interviewing skills as a factor that reduces the likelihood that a student will obtain a pharmacy practice residency. The extent to which causal claims can be made depends on appropriate study design and analysis. Similarly, the language used to present these claims is related to study design and analysis. Unfortunately, study designs used when evaluating educational approaches do not always align with the language used to report the results. This review provides a brief overview of current thinking related to causal inference. The role of study design and analysis in causal inference is also discussed along with recommendations for study design and analysis to facilitate making appropriate causal claims. After reading this article, readers should be able to select a study design that best aligns with their particular research question and report their findings in ways that are faithful to the strengths of the study design.

Scholarly Conversations in Medical Education

This supplement includes the eight research papers accepted by the 2016 Research in Medical Education Program Planning Committee. In this Commentary, the authors use "conversations in medical education" as a guiding metaphor to explore what these papers contribute to the current scholarly discourse in medical education. They organize their discussion around two domains: the topic of study and the methodological approach. The authors map the eight research papers to six "hot topics" in medical education: (1) curriculum reform, (2) duty hours restriction, (3) learner well-being, (4) innovations in teaching and assessment, (5) self-regulated learning, and (6) learning environment, and to three purposes commonly served by medical education research: (1) description, (2) justification, and (3) clarification. They discuss the range of methods employed in the papers. The authors end by encouraging educators to engage in these ongoing scholarly conversations.

Bonfire red titles

None

Use of 3D printed models in medical education: A randomized control trial comparing 3D prints versus cadaveric materials for learning external cardiac anatomy

Three-dimensional (3D) printing is an emerging technology capable of readily producing accurate anatomical models, however, evidence for the use of 3D prints in medical education remains limited. A study was performed to assess their effectiveness against cadaveric materials for learning external cardiac anatomy. A double blind randomized controlled trial was undertaken on undergraduate medical students without prior formal cardiac anatomy teaching. Following a pre-test examining baseline external cardiac anatomy knowledge, participants were randomly assigned to three groups who underwent self-directed learning sessions using either cadaveric materials, 3D prints, or a combination of cadaveric materials/3D prints (combined materials). Participants were then subjected to a post-test written by a third party. Fifty-two participants completed the trial; 18 using cadaveric materials, 16 using 3D models, and 18 using combined materials. Age and time since completion of high school were equally distributed between groups. Pre-test scores were not significantly different (P = 0.231), however, post-test scores were significantly higher for 3D prints group compared to the cadaveric materials or combined materials groups (mean of 60.83% vs. 44.81% and 44.62%, P = 0.010, adjusted P = 0.012). A significant improvement in test scores was detected for the 3D prints group (P = 0.003) but not for the other two groups. The finding of this pilot study suggests that use of 3D prints do not disadvantage students relative to cadaveric materials; maximally, results suggest that 3D may confer certain benefits to anatomy learning and supports their use and ongoing evaluation as supplements to cadaver-based curriculums. Anat Sci Educ 9: 213-221. © 2015 American Association of Anatomists.

Standing on the Precipice: Evaluating Final-Year Physiotherapy Students' Perspectives of Their Curriculum as Preparation for Primary Health Care Practice

Purpose: To explore final-year physiotherapy students' perceptions of primary health care practice to determine (1) aspects of their curriculum that support their learning, (2) deficiencies in their curriculum, and (3) areas that they believe should be changed to adequately equip them to make the transition from student to primary health care professional. Methods: Framework analysis methodology was used to analyze group opinion obtained using structured group feedback sessions. Sixty-eight final-year physiotherapy students from the four higher education institutions in Ireland participated. Results: The students identified several key areas that (1) supported their learning (exposure to evidence-based practice, opportunities to practise with problem-based learning, and interdisciplinary learning experiences); (2) were deficient (primary health care placements, additional active learning sessions, and further education and practice opportunities for communication and health promotion), and (3) required change (practice placements in primary health care, better curriculum organization to accommodate primary health care throughout the programme with the suggestion of a specific primary health care module). Conclusion: This study provides important insights into physiotherapy students' perceptions of primary health care. It also provides important indicators of the curriculum changes needed to increase graduates' confidence in their ability to take up employment in primary health care.

Vertically integrated medical education and the readiness for practice of graduates

BACKGROUND

Medical curricula become more and more vertically integrated (VI) to prepare graduates better for clinical practice. VI curricula show early clinical education, integration of biomedical sciences and focus on increasing clinical responsibility levels for trainees. Results of earlier questionnaire-based studies indicate that the type of the curriculum can affect the perceived preparedness for work as perceived by students or supervisors. The aim of the present study is to determine difference in actual performance of graduates from VI and non-VI curricula.

METHODS

We developed and implemented an authentic performance assessment based on different facets of competence for medical near-graduates in the role of beginning residents on a very busy day. Fifty nine candidates participated: 30 VI (Utrecht, The Netherlands) and 29 non-VI (Hamburg, Germany). Two physicians, one nurse and five standardized patients independently assessed each candidate on different facets of competence. Afterwards, the physicians indicated how much supervision they estimated each candidate would require on nine so called "Entrustable Professional Activities (EPAs)" unrelated to the observed scenarios.

RESULTS

Graduates from a VI curriculum received significantly higher scores by the physicians for the facet of competence "active professional development", with features like 'reflection' and 'asking for feedback'. In addition, VI graduates scored better on the EPA "solving a management problem", while the non-VI graduates got higher scores for the EPA "breaking bad news".

CONCLUSIONS

This study gives an impression of the actual performance of medical graduates from VI and non-VI curricula. Even though not many differences were found, VI graduates got higher scores for features of professional development, which is important for postgraduate training and continuing education.

Simulation using novel equipment designed to explain spirometric abnormalities in respiratory disease enhances learning in higher cognitive domains

Simulation of disorders of respiratory mechanics shown by spirometry provides insight into the pathophysiology of disease but some clinically important disorders have not been simulated and none have been formally evaluated for education. We have designed simple mechanical devices which, along with existing simulators, enable all the main dysfunctions which have diagnostic value in spirometry to be simulated and clearly explained with visual and haptic feedback. We modelled the airways as Starling resistors by a clearly visible mechanical action to simulate intra- and extra-thoracic obstruction. A narrow tube was used to simulate fixed large airway obstruction and inelastic bands to simulate restriction. We hypothesized that using simulators whose action explains disease promotes learning especially in higher domain educational objectives. The main features of obstruction and restriction were correctly simulated. Simulation of variable extra-thoracic obstruction caused blunting and plateauing of inspiratory flow, and simulation of intra-thoracic obstruction caused limitation of expiratory flow with marked dynamic compression. Multiple choice tests were created with questions allocated to lower (remember and understand) or higher cognitive domains (apply, analyse and evaluate). In a cross-over design, overall mean scores increased after 1½ h simulation spirometry (43-68 %, effect size 1.06, P < 0.0001). In higher cognitive domains the mean score was lower before and increased further than lower domains (Δ 30 vs 20 %, higher vs lower effect size 0.22, P < 0.05). In conclusion, the devices successfully simulate various patterns of obstruction and restriction. Using these devices medical students achieved marked enhancement of learning especially in higher cognitive domains.

Systematic reviews in medical education: a practical approach: AMEE guide 94

The twentieth century saw a paradigm shift in medical education, with acceptance that 'knowledge' and 'truth' are contextual, in flux and always evolving. The twenty-first century has seen a greater explosion in computer technology leading to a massive increase in information and an ease of availability, both offering great potential to future research. However, for many decades, there have been voices within the health care system raising an alarm at the lack of evidence to support widespread clinical practice; from these voices, the concept of and need for evidence-based health-care has grown. Parallel to this development has been the emergence of evidence-based medical education; if healthcare is evidence-based, then the training of practitioners who provide this healthcare must equally be evidence-based. Evidence-based medical education involves the systematic collection, synthesis and application of all available evidence, when available, and not just the opinion of experts. This represented a seismic shift from a position of expert based consensus guidance to evidence led guidance for evolving clinical knowledge. The aim of this guide is to provide a practical approach to the development and application of a systematic review in medical education; a valid method used in this guide to seek and substantiate the effects of interventions in medical education.

Context matters: emergent variability in an effectiveness trial of online teaching modules

CONTEXT

Conducting research in real life settings (effectiveness studies) can introduce many confounding factors. Efficacy studies seek to control for researcher bias and data quality rather than considering how the efficacy of an intervention is changed by the contexts in which it is used. Relatively little is known about the impact of context on educational interventions, in particular on multimedia learning.

METHODS

An effectiveness study to understand implementation variance of online educational modules in surgery clerkships was conducted in six US medical schools participating in an efficacy trial of different multimedia designs. Student and teacher experiences were captured through focus groups and one-to-one interviews with trial participants and their teachers. Audio-recordings of these sessions were transcribed and analysed using grounded theory techniques.

RESULTS

Differences were identified in student and teacher perceptions of how the educational intervention had been implemented and how its uptake had been influenced by context-dependent factors: (i) the intervention was implemented in different ways to suit different educational contexts and this influenced how students and teachers responded to it; (ii) the ways students and teachers interacted with, and behaved around, the intervention influenced its uptake; (iii) the way the intervention was perceived by students and teachers influenced its uptake; and (iv) the medium and design of the intervention had a directing influence on its uptake.

CONCLUSIONS

It was observed that each institutional context formed a complex educational ecology. The intervention became interwoven with different educational ecologies so that it could no longer be considered a stable variable across the study. We suggest that researchers should conduct implementation-profiling studies in advance of any intervention-based research to account for the constructing nature of educational ecologies on their interventions and in doing so to more clearly differentiate between efficacy and effectiveness studies.

Cost analyses approaches in medical education: there are no simple solutions

CONTEXT

Medical education is expensive. Although we have made progress in working out 'what works' in medical education, there are few data on whether medical education offers value relative to cost. Research into cost and value in medical education is beset by problems. One of the major problems is the lack of clear definitions for many of the terms commonly used. Phrases such as cost-effectiveness analysis, cost-benefit analysis, cost-utility analysis and cost-feasibility analysis are used without authors explaining to readers what they mean (and sometimes without authors themselves understanding what they mean). Sometimes such terms are used interchangeably and sometimes they are used as rhetorical devices without any real evidence that backs up such rhetoric as to the cost-effectiveness or otherwise of educational interventions. The frequent misuse of these terms is surprising considering the importance of the topics under consideration and the need for precision in many aspects of medical education.

METHODS

Here we define commonly used terms in cost analyses and give examples of their usage in the context of medical education.

CONCLUSIONS

Cost-effectiveness analysis refers to the evaluation of two or more alternative educational approaches or interventions according to their costs and their effects in producing a certain outcome. Cost-benefit analysis refers to 'the evaluation of alternatives according to their costs and benefits when each is measured in monetary terms'. Cost-utility analysis is the examination of two or more alternatives according to their cost and their utility. In this context, utility means the satisfaction among individuals as a result of one or more outcome or the perceived value of the expected outcomes to a particular constituency. Cost-feasibility analysis involves simply measuring the cost of a proposed intervention in order to decide whether it is feasible.

Influence of PBL with open-book tests on knowledge retention measured with progress tests

The influence of problem-based learning (PBL) and open-book tests on long-term knowledge retention is unclear and subject of discussion. Hypotheses were that PBL as well as open-book tests positively affect long-term knowledge retention. Four progress test results of fifth and sixth-year medical students (n = 1,648) of three medical schools were analyzed. Two schools had PBL driven curricula, and the third one had a traditional curriculum (TC). One of the PBL schools (PBLob) used a combination of open-book (assessing backup knowledge) and closed-book tests (assessing core knowledge); the other two schools (TC and PBLcb) only used closed-book tests. The items of the progress tests were divided into core and backup knowledge. T tests (with Bonferroni correction) were used to analyze differences between curricula. PBL students performed significantly better than TC students on core knowledge (average effect size (av ES) = 0.37-0.74) and PBL students tested with open-book tests scored somewhat higher than PBL students tested without such tests (av ES = 0.23-0.30). Concerning backup knowledge, no differences were found between the scores of the three curricula. Students of the two PBL curricula showed a substantially better long-term knowledge retention than TC students. PBLob students performed somewhat better on core knowledge than PBLcb students. These outcomes suggest that a problem-based instructional approach in particular can stimulate long-term knowledge retention. Distinguishing knowledge into core and backup knowledge and using open-book tests alongside closed-book tests could enhance long-term core knowledge retention.

Rethinking programme evaluation in health professions education: beyond 'did it work?'

CONTEXT

For nearly 40 years, outcome-based models have dominated programme evaluation in health professions education. However, there is increasing recognition that these models cannot address the complexities of the health professions context and studies employing alternative evaluation approaches that are appearing in the literature. A similar paradigm shift occurred over 50 years ago in the broader discipline of programme evaluation. Understanding the development of contemporary paradigms within this field provides important insights to support the evolution of programme evaluation in the health professions.

METHODS

In this discussion paper, we review the historical roots of programme evaluation as a discipline, demonstrating parallels with the dominant approach to evaluation in the health professions. In tracing the evolution of contemporary paradigms within this field, we demonstrate how their aim is not only to judge a programme's merit or worth, but also to generate information for curriculum designers seeking to adapt programmes to evolving contexts, and researchers seeking to generate knowledge to inform the work of others.

DISCUSSION

From this evolution, we distil seven essential elements of educational programmes that should be evaluated to achieve the stated goals. Our formulation is not a prescriptive method for conducting programme evaluation; rather, we use these elements as a guide for the development of a holistic 'programme of evaluation' that involves multiple stakeholders, uses a combination of available models and methods, and occurs throughout the life of a programme. Thus, these elements provide a roadmap for the programme evaluation process, which allows evaluators to move beyond asking whether a programme worked, to establishing how it worked, why it worked and what else happened. By engaging in this process, evaluators will generate a sound understanding of the relationships among programmes, the contexts in which they operate, and the outcomes that result from them.

The role of radiology in preclinical anatomy: a critical review of the past, present, and future

RATIONALE AND OBJECTIVES

Radiology has been an increasingly important component of preclinical anatomy instruction since the 1960s. The global status of medical imaging pedagogies and radiologists' roles in medical anatomy education is not well established but is important in determining the specialty's contribution to undergraduate medical education.

MATERIALS AND METHODS

PubMed was searched with various combinations of MeSH terms including "radiology," "undergraduate medical education," and "anatomy." Articles were reviewed for relevance, and referenced articles of possible relevance were hand-traced to ensure a wide capture of articles.

RESULTS

Although more medical schools around the world are using medical imaging to teach anatomy, some regions, such as the United States, show a decline in the proportion of imaging taught by radiologists. Lectures, small group discussions, and self-instruction remain the mainstay of current pedagogies and have witnessed dramatic changes over the past few decades with respect to the types of imaging used. Newer pedagogies use contextual and hands-on experiences to improve spatial and application principles. Qualitative and quantitative studies report somewhat mixed results of pedagogical efficacies but demonstrate generally high acceptance by students and instructors and often significant exam score improvement. Radiology as a specialty must overcome several challenges for it to become more involved in anatomy education, including teaching incentives and protected academic time.

CONCLUSIONS

As anatomy instruction and clinical medicine grow increasingly digital, it is ever more important that radiologists continue to develop new anatomy pedagogies and contribute to anatomy education in greater roles.