Method and reporting quality in health professions education research: a systematic review

Assessing methodological quality in dental education research using MERSQI: Analysis of publications from two journals

PURPOSE

The Medical Education Research Study Quality Instrument (MERSQI) has been used frequently to assess the methodological quality of medical education but not for dental education. The present study aimed to assess the methodological quality using MERSQI scores of articles published in the Journal of Dental Education (JDE) and the European Journal of Dental Education (EJDE).

METHODS

A cross-sectional assessment of the quality of manuscripts published in 2012, 2017, and 2022 JDE and EJDE was conducted. MERSQI data, numbers of authors, first and corresponding author degrees, geographic origins, and funding information were also extracted for each included study. Descriptive and analytical statistics were conducted, and significance level was set at α < 0.05.

RESULTS

Four hundred ninety-five articles met the inclusion criteria. The most common study design was a single-group cross-sectional or single-group posttest and conducted in one institution for all studied years. In all journals and years, studies were assessed mainly by participants. The study outcome was mostly satisfaction, attitudes, perceptions, opinions, and general facts. The total mean MERSQI score for each journal and year varied. Year and geographic origin significantly affected the total MERSQI score. Papers originating from Asia had the highest score, followed by South America, Europe, North America, Oceania, and Africa.

CONCLUSION

MERSQI score is applicable to the assessment of the methodological quality of dental educational research. The MERSQI score for most of the domains was similar for both journals. The MERSQI score was affected by publication years and geographic origins.

Improving Graduate Medical Education by Aggregating Data Across the Medical Education Continuum

ABSTRACT

Meaningful improvements to graduate medical education (GME) have been achieved in recent decades, yet many GME improvement pilots have been small trials without rigorous outcome measures and with limited generalizability. Thus, lack of access to large-scale data is a key barrier to generating empiric evidence to improve GME. In this article, the authors examine the potential of a national GME data infrastructure to improve GME, review the output of 2 national workshops on this topic, and propose a path toward achieving this goal.The authors envision a future where medical education is shaped by evidence from rigorous research powered by comprehensive, multi-institutional data. To achieve this goal, premedical education, undergraduate medical education, GME, and practicing physician data must be collected using a common data dictionary and standards and longitudinally linked using unique individual identifiers. The envisioned data infrastructure could provide a foundation for evidence-based decisions across all aspects of GME and help optimize the education of individual residents.Two workshops hosted by the National Academies of Sciences, Engineering, and Medicine Board on Health Care Services explored the prospect of better using GME data to improve education and its outcomes. There was broad consensus about the potential value of a longitudinal data infrastructure to improve GME. Significant obstacles were also noted.Suggested next steps outlined by the authors include producing a more complete inventory of data already being collected and managed by key medical education leadership organizations, pursuing a grass-roots data sharing pilot among GME-sponsoring institutions, and formulating the technical and governance frameworks needed to aggregate data across organizations.The power and potential of big data is evident across many disciplines, and the authors believe that harnessing the power of big data in GME is the best next step toward advancing evidence-based physician education.

Measuring wellbeing: A scoping review of metrics and studies measuring medical student wellbeing across multiple timepoints

PURPOSE

Studies have demonstrated poor mental health in medical students. However, there is wide variation in study design and metric use, impairing comparability. The authors aimed to examine the metrics and methods used to measure medical student wellbeing across multiple timepoints and identify where guidance is necessary.

METHODS

Five databases were searched between May and June 2021 for studies using survey-based metrics among medical students at multiple timepoints. Screening and data extraction were done independently by two reviewers. Data regarding the manuscript, methodology, and metrics were analyzed.

RESULTS

221 studies were included, with 109 observational and 112 interventional studies. There were limited studies (15.4%) focused on clinical students. Stress management interventions were the most common (40.2%). Few (3.57%) interventional studies followed participants longer than 12 months, and 38.4% had no control group. There were 140 unique metrics measuring 13 constructs. 52.1% of metrics were used only once.

CONCLUSIONS

Unique guidance is needed to address gaps in study design as well as unique challenges surrounding medical student wellbeing surveys. Metric use is highly variable and future research is necessary to identify metrics specifically validated in medical student samples that reflect the diversity of today's students.

Transfer of skills between laparoscopic and robot-assisted surgery: a systematic review

BACKGROUND

Robot-assisted surgery is today well-implemented in many surgical specialties, but requires another skill set than laparoscopy. Most often, robot-assisted surgery is considered add-on to laparoscopic skills but very little is known about the transfer of skills. The aim of the study was to examine to what extent surgical skills are transferable between laparoscopic and robot-assisted surgery.

METHODS

A systematic search was conducted in three databases (Ovid Medline, Embase, and Web of Science). Studies investigating transfer of skills between laparoscopy and robot-assisted surgery in either a phantom-based, simulation-based, animal model, or clinical setting were eligible for inclusion. Quality assessment was done using the Medical education research study quality instrument and educational New Ottawa Scale.

RESULTS

Of 15,610 studies identified, 89 studies continued to full-text reading, and 37 studies were included. Four studies were found non-comparable and were left out of the results for the primary outcome. All 33 studies explored transfer from laparoscopy to robot-assisted surgery and 17 found a positive transfer whereas 15 did not. Only 11 studies explored transfer from robot-assisted surgery to laparoscopy, of which only three found a positive transfer.

CONCLUSION

An almost equal number of publications found a positive transfer and no transfer from laparoscopic to robot-assisted surgery. Fewer studies explored the transfer from robot-assisted surgery to laparoscopy. Very little evidence supports that surgeons trained solely in robot-assisted surgery can perform laparoscopy. This must be considered in future training programs as robot-assisted surgery is expected to become the first-in-line modality for many future surgeons.

Research Methodologies in Health Professions Education Publications: Breadth and Rigor

PURPOSE

Research methodologies represent assumptions about knowledge and ways of knowing. Diverse research methodologies and methodological standards for rigor are essential in shaping the collective set of knowledge in health professions education (HPE). Given this relationship between methodologies and knowledge, it is important to understand the breadth of research methodologies and their rigor in HPE research publications. However, there are limited studies examining these questions. This study synthesized current trends in methodologies and rigor in HPE papers to inform how evidence is gathered and collectively shapes knowledge in HPE.

METHOD

This descriptive quantitative study used stepwise stratified cluster random sampling to analyze 90 papers from 15 HPE journals published in 2018 and 2019. Using a research design codebook, the authors conducted group coding processes for fidelity, response process validity, and rater agreement; an index quantifying methodological rigor was developed and applied for each paper.

RESULTS

Over half of research methodologies were quantitative (51%), followed by qualitative (28%), and mixed methods (20%). No quantitative and mixed methods papers reported an epistemological approach. All qualitative papers that reported an epistemological approach (48%) used social constructivism. Most papers included participants from North America (49%) and Europe (20%). The majority of papers did not specify participant sampling strategies (56%) or a rationale for sample size (80%). Among those reported, most studies (81%) collected data within 1 year.The average rigor score of the papers was 56% (SD = 17). Rigor scores varied by journal categories and research methodologies. Rigor scores differed between general HPE journals and discipline-specific journals. Qualitative papers had significantly higher rigor scores than quantitative and mixed methods papers.

CONCLUSIONS

This review of methodological breadth and rigor in HPE papers raises awareness in addressing methodological gaps and calls for future research on how the authors shape the nature of knowledge in HPE.

Surgical endoscopy education research: how are we doing?

BACKGROUND

Surgical endoscopy (SE), the official journal of the Society of American Gastrointestinal and Endoscopic Surgeons and the European Association for Endoscopic Surgery, is an important source of new evidence pertaining to surgical education in the field. However, qualitative deficiencies in medical education research have prompted medical education leaders to advocate for increased methodological rigor. The purpose of this study is to review the quality of education-focused research published through SE.

METHODS

A PubMed search examining all SE articles categorized as education-related research from 2010 to 2019 was conducted; studies not meeting inclusion criteria were excluded. Remaining publications were independently reviewed, classified, and scored by 7 raters using the medical education research study quality instrument (MERSQI). Intraclass correlation was calculated and data were examined with descriptive statistics.

RESULTS

A total of 227 studies met inclusion criteria. There was no significant difference in number of publications by year (average 25.88 [SD 5.6]); 60% were conducted outside of the United States, and 47% (n = 106) were funded. The average MERSQI was 12.5 (SD 2). Most studies used two-group non-random (42%, n = 96) or post/cross-sectional designs (29%, n = 65). Thirty-six (16%) were randomized controlled trials. Multi-institutional studies comprised 24% (n = 54). Of the manuscripts, 96% (n = 217) reported at least one measure of validity evidence and 28% (n = 67) described three levels of validity evidence. Studies primarily reported changes in skills or knowledge (45%, n = 103) or satisfaction or general facts (44%, n = 99), while patient-related outcomes encompassed 3% (n = 6) of studies. ICC between raters was 0.93 (CI 0.90-0.93, p < 0.001).

CONCLUSIONS

Based on publications to date, this journal's peer review process appears to facilitate the dissemination of education-related studies of moderate to good quality. However, there were uncovered deficits, ranging from validity evidence to study designs and level of outcomes. This journal's breadth of viewership offers a potential venue to advance education-related research.

The Status of SBIRT Training in Health Professions Education: A Cross-Discipline Review and Evaluation of SBIRT Curricula and Educational Research

PURPOSE

To assess the quality of curricular research on the Screening Brief Intervention and Referral to Treatment (SBIRT) approach and determine the presence of useful training modalities, particularly motivational interviewing (MI) training, across health care training curricula.

METHOD

The authors conducted a systematic review of published, peer-reviewed studies in PubMed, ERIC, CINAHL, Ovid HealthSTAR, and PsycINFO databases through March 2021 for English-language studies describing SBIRT, a curriculum for health care trainees, and curricular intervention outcomes. After the records were independently assessed, data were extracted and 20% of the studies were double-coded for interrater reliability.

RESULTS

Of 1,856 studies, 95 were included in the review; 22 had overlapping samples and were consolidated into 10 nested studies, leaving 83 total. Interrater reliability ranged from moderate (κ = .74, P < .001) to strong (κ = .91, P < .001) agreement. SBIRT training was delivered to trainees across many professions, including nursing (n = 34, 41%), medical residency (n = 28, 34%), and social work (n = 24, 29%). Nearly every study described SBIRT training methods (n = 80, 96%), and most reported training in MI (n = 54, 65%). On average, studies reported 4.06 (SD = 1.64) different SBIRT training methods and 3.31 (SD = 1.59) MI training methods. Their mean design score was 1.92 (SD = 0.84) and mean measurement score was 1.89 (SD = 1.05). A minority of studies measured SBIRT/MI skill (n = 23, 28%), and 4 studies (5%) set a priori benchmarks for their curricula.

CONCLUSIONS

SBIRT training has been delivered to a wide range of health care trainees and often includes MI. Rigor scores for the studies were generally low due to limited research designs and infrequent use of objective skill measurement. Future work should include predefined training benchmarks and validated skills measurement.

Quality of cost evaluations of physician continuous professional development: Systematic review of reporting and methods

INTRODUCTION

We sought to evaluate the reporting and methodological quality of cost evaluations of physician continuing professional development (CPD).

METHODS

We conducted a systematic review, searching MEDLINE, Embase, PsycInfo, and the Cochrane Database for studies comparing the cost of physician CPD (last update 23 April 2020). Two reviewers, working independently, screened all articles for inclusion. Two reviewers extracted information on reporting quality using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS), and on methodological quality using the Medical Education Research Study Quality Instrument (MERSQI) and a published reference case.

RESULTS

Of 3338 potentially eligible studies, 62 were included. Operational definitions of methodological and reporting quality elements were iteratively revised. Articles reported mean (SD) 43% (20%) of CHEERS elements for the Title/Abstract, 56% (34%) for Introduction, 66% (19%) for Methods, 61% (17%) for Results, and 66% (30%) for Discussion, with overall reporting index 292 (83) (maximum 500). Valuation methods were reported infrequently (resource selection 10 of 62 [16%], resource quantitation 10 [16%], pricing 26 [42%]), as were descriptions/discussion of the physicians trained (42 [68%]), training setting (42 [68%]), training intervention (40 [65%]), sensitivity analyses of uncertainty (9 [15%]), and generalizability (30 [48%]). MERSQI scores ranged from 6.0 to 16.0 (mean 11.2 [2.4]). Changes over time in reporting index (initial 241 [105], final 321 [52]) and MERSQI scores (initial 9.8 [2.7], final 11.9 [1.9]) were not statistically significant (p ≥ 0.08).

DISCUSSION

Methods and reporting of HPE cost evaluations fall short of current standards. Gaps exist in the valuation, analysis, and contextualization of cost outcomes.

Datasets for Automated Affect and Emotion Recognition from Cardiovascular Signals Using Artificial Intelligence- A Systematic Review

Simple Summary

We reviewed the literature on the publicly available datasets used to automatically recognise emotion and affect using artificial intelligence (AI) techniques. We were particularly interested in databases with cardiovascular (CV) data. Additionally, we assessed the quality of the included papers. We searched the sources until 31 August 2020. Each step of identification was carried out independently by two reviewers to maintain the credibility of our review. In case of disagreement, we discussed them. Each action was first planned and described in a protocol that we posted on the Open Science Framework (OSF) platform. We selected 18 works focused on providing datasets of CV signals for automated affect and emotion recognition. In total, data for 812 participants aged 17 to 47 were analysed. The most frequently recorded signal was electrocardiography. The authors most often used video stimulation. Noticeably, we did not find much necessary information in many of the works, resulting in mainly low quality among included papers. Researchers in this field should focus more on how they carry out experiments.

Abstract

Our review aimed to assess the current state and quality of publicly available datasets used for automated affect and emotion recognition (AAER) with artificial intelligence (AI), and emphasising cardiovascular (CV) signals. The quality of such datasets is essential to create replicable systems for future work to grow. We investigated nine sources up to 31 August 2020, using a developed search strategy, including studies considering the use of AI in AAER based on CV signals. Two independent reviewers performed the screening of identified records, full-text assessment, data extraction, and credibility. All discrepancies were resolved by discussion. We descriptively synthesised the results and assessed their credibility. The protocol was registered on the Open Science Framework (OSF) platform. Eighteen records out of 195 were selected from 4649 records, focusing on datasets containing CV signals for AAER. Included papers analysed and shared data of 812 participants aged 17 to 47. Electrocardiography was the most explored signal (83.33% of datasets). Authors utilised video stimulation most frequently (52.38% of experiments). Despite these results, much information was not reported by researchers. The quality of the analysed papers was mainly low. Researchers in the field should concentrate more on methodology.

Online learning developments in undergraduate medical education in response to the COVID-19 pandemic: A BEME systematic review: BEME Guide No. 69

BACKGROUND

The COVID-19 pandemic spurred an abrupt transition away from in-person educational activities. This systematic review investigated the pivot to online learning for nonclinical undergraduate medical education (UGME) activities and explored descriptions of educational offerings deployed, their impact, and lessons learned.

METHODS

The authors systematically searched four online databases and conducted a manual electronic search of MedEdPublish up to December 21, 2020. Two authors independently screened titles, abstracts and full texts, performed data extraction and assessed risk of bias. A third author resolved discrepancies. Findings were reported in accordance with the STORIES (STructured apprOach to the Reporting in healthcare education of Evidence Synthesis) statement and BEME guidance.

RESULTS

Fifty-six articles were included. The majority (n = 41) described the rapid transition of existing offerings to online formats, whereas fewer (n = 15) described novel activities. The majority (n = 27) included a combination of synchronous and asynchronous components. Didactics (n = 40) and small groups (n = 26) were the most common instructional methods. Teachers largely integrated technology to replace and amplify rather than transform learning, though learner engagement was often interactive. Thematic analysis revealed unique challenges of online learning, as well as exemplary practices. The quality of study designs and reporting was modest, with underpinning theory at highest risk of bias. Virtually all studies (n = 54) assessed reaction/satisfaction, fewer than half (n = 23) assessed changes in attitudes, knowledge or skills, and none assessed behavioral, organizational or patient outcomes.

CONCLUSIONS

UGME educators successfully transitioned face-to-face instructional methods online and implemented novel solutions during the COVID-19 pandemic. Although technology's potential to transform teaching is not yet fully realized, the use of synchronous and asynchronous formats encouraged virtual engagement, while offering flexible, self-directed learning. As we transition from emergency remote learning to a post-pandemic world, educators must underpin new developments with theory, report additional outcomes and provide details that support replication.

Barriers and facilitators to program directors' use of the medical education literature: a qualitative study

BACKGROUND

It is unclear how often frontline clinical teachers are using this literature and its evidence base in teaching and assessment. Our study purpose was to examine postgraduate program director perspectives on the utilization and integration of evidence-based medical education literature in their teaching and assessment practices.

METHODS

The authors conducted semi-structured telephone interviews with a convenience sample of current and former program directors from across Canada. Interviews were transcribed and analyzed inductively to distil pertinent themes.

RESULTS

In 2017, 11 former and current program directors participated in interviews. Major themes uncovered included the desire for time-efficient and easily adaptable teaching and assessment tools. Participants reported insufficient time to examine the medical education literature, and preferred that it be 'synthesized for them'. (i.e., Best evidence guidelines). Participants recognised continuing professional development and peer to peer sharing as useful means of education about evidence-based tools. Barriers to the integration of the literature in practice included inadequate time, lack of financial compensation for teaching and assessment, and the perception that teaching and assessment of trainees was not valued in academic promotion.

DISCUSSION

Faculty development offices should consider the time constraints of clinical teachers when planning programming on teaching and assessment. To enhance uptake, medical education publications need to consider approaches that best meet the needs of a targeted audiences, including frontline clinical teachers. This may involve novel methods and formats that render evidence and findings from their studies more easily 'digestible' by clinical teachers to narrow the knowledge to practice gap.

Effective online learning strategies for leadership and policy undergraduate courses for nursing students: a rapid review

OBJECTIVES

Due to the importance of developing leadership competencies during nursing education, it is critical to make evidence-based decisions regarding the transformation from face-to-face to online delivery of leadership and policy courses for nursing students in the wake of the COVID-19 pandemic. This rapid review aims to identify effective learning strategies for creating online leadership and policy courses for undergraduate nursing students.

DATA SOURCES AND REVIEW METHODS

A rapid review methodology was used. Searches in CINAHL and ERIC yielded 4112 records. After screening, seven articles were included. The Criteria for Describing and Evaluating Training Interventions in Healthcare Professions (CRe-DEPTH) tool was used for quality appraisal and data extraction. A narrative synthesis approach was used to summarize the data.

RESULTS

The learning activities were heterogeneous in terms of content and format. Articles described the use of discussion forums, case studies, virtual clinical learning experiences, microblogging, and video clips. The methods of evaluation for these learning activities also varied greatly.

CONCLUSION

The findings will act as a steppingstone to help develop an online undergraduate leadership and policy nursing course. This review also demonstrated the need for rigorous evaluation of learning activities. The use of a tool such as the CRe-DEPTH can help instructors plan and report on their learning interventions or courses.

Online Educational Tools for Caregivers of People with Dementia: A Scoping Literature Review

BACKGROUND

Informal caregivers of people with dementia provide the majority of health-based care to people with dementia. Providing this care requires knowledge and access to resources, which caregivers often do not receive. We set out to evaluate the effect of online educational tools on informal caregiver self-efficacy, quality of life, burden/stress, depression, and anxiety, and to identify effective processes for online educational tool development.

METHODS

We conducted a scoping review of articles on online educational interventions for informal caregivers of people with dementia searching CINAHL, MEDLINE, EMBASE, and PubMed from 1990 to March 2018, with an updated search conducted in 2020. The identified articles were screened and the data were charted.

RESULTS

33 articles that reported on 24 interventions were included. There is some evidence that online interventions improve caregiver-related outcomes such as self-efficacy, depression, dementia knowledge, and quality of life; and decrease caregiver burden. Common findings across the studies included the need for tailored, stage-specific information applicable to the caregiver's situation and the use of psychosocial techniques to develop the knowledge components of the interventions.

CONCLUSION

We demonstrate the importance of having caregivers and health-care professionals involved at all stages of tool conceptualization and development. Online tools should be evaluated with robust trials that focus on how increased knowledge and development approaches affect caregiver-related outcomes.

Collaborative healthcare education programmes for continuing professional education in low and middle-income countries: A Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 65

BACKGROUND

Large discrepancies exist between standards of healthcare provision in high-income (HICs) and low and middle-income countries (LMICs). The root cause is often financial, resulting in poor infrastructure and under-resourced education and healthcare systems. Continuing professional education (CPE) programmes improve staff knowledge, skills, retention, and practice, but remain costly and rare in low-resource settings. One potential solution involves healthcare education collaborations between institutions in HICs and LMICs to provide culturally appropriate CPE in LMICs. To be effective, educational partnerships must address the challenges arising from differences in cultural norms, language, available technology and organisational structures within collaborating countries.

METHODS

Seven databases and other sources were systematically searched on 7 July 2020 for relevant studies. Citations, abstracts, and studies were screened and consensus was reached on which to include within the review. 54 studies were assessed regarding the type of educational programme involved, the nature of HIC/LMIC collaboration and quality of the study design.

RESULTS

Studies varied greatly regarding the types and numbers of healthcare professionals involved, pedagogical and delivery methods, and the ways in which collaboration was undertaken. Barriers and enablers of collaboration were identified and discussed. The key findings were: 1. The methodological quality of reporting in the studies was generally poor. 2. The way in which HIC/LMIC healthcare education collaboration is undertaken varies according to many factors, including what is to be delivered, the learner group, the context, and the resources available. 3. Western bias was a major barrier. 4. The key to developing successful collaborations was the quality, nature, and duration of the relationships between those involved.

CONCLUSION

This review provides insights into factors that underpin successful HIC/LMIC healthcare CPE collaborations and outlines inequities and quality issues in reporting.

Social media in undergraduate medical education: A systematic review

INTRODUCTION

There are over 3.81 billion worldwide active social media (SoMe) users. SoMe are ubiquitous in medical education, with roles across undergraduate programmes, including professionalism, blended learning, well being and mentoring. Previous systematic reviews took place before recent explosions in SoMe popularity and revealed a paucity of high-quality empirical studies assessing its effectiveness in medical education. This review aimed to synthesise evidence regarding SoMe interventions in undergraduate medical education, to identify features associated with positive and negative outcomes.

METHODS

Authors searched 31 key terms through seven databases, in addition to references, citation and hand searching, between 16 June and 16 July 2020. Studies describing SoMe interventions and research on exposure to existing SoMe were included. Title, abstract and full paper screening were undertaken independently by two reviewers. Included papers were assessed for methodological quality using the Medical Education Research Study Quality Instrument (MERSQI) and/or the Standards for Reporting Qualitative Research (SRQR) instrument. Extracted data were synthesised using narrative synthesis.

RESULTS

112 studies from 26 countries met inclusion criteria. Methodological quality of included studies had not significantly improved since 2013. Engagement and satisfaction with SoMe platforms in medical education are described. Students felt SoMe flattened hierarchies and improved communication with educators. SoMe use was associated with improvement in objective knowledge assessment scores and self-reported clinical and professional performance, however evidence for long term knowledge retention was limited. SoMe use was occasionally linked to adverse impacts upon mental and physical health. Professionalism was heavily investigated and considered important, though generally negative correlations between SoMe use and medical professionalism may exist.

CONCLUSIONS

Social media is enjoyable for students who may improve short term knowledge retention and can aid communication between learners and educators. However, higher-quality study is required to identify longer-term impact upon knowledge and skills, provide clarification on professionalism standards and protect against harms.

Transesophageal Echocardiography Simulator Training: A Systematic Review and Meta-analysis of Randomized Controlled Trials

SUMMARY STATEMENT

We aimed to assess the learning effects of novice transesophageal echocardiography (TEE) simulator training and to identify gaps in existing studies. We performed a systematic review and meta-analysis of randomized controlled trials (RCTs) comparing the learning effects of novice TEE training with versus without simulators, searching published articles and proceedings in 6 major databases in June 2019. We included 9 RCTs (268 participants). Compared with nonsimulator training, TEE simulator training resulted in higher skill and knowledge posttraining test scores with large effect sizes (standardized mean difference = 0.81 for skill, 1.61 for knowledge; low-certainty evidence) and higher training satisfaction with a small effect size (standardized mean difference = 0.36; very low-certainty evidence). No RCTs reported training budget or patient outcomes. Additional well-designed studies with low risk of bias and large sample sizes are needed to provide reliable and robust findings and develop more effective TEE simulation-based training curricula.

Is There a Replication Crisis in Medical Education Research?

Scholars are increasingly aware that studies-across many disciplines-cannot be replicated by independent researchers. Here, the authors describe how medical education research may be vulnerable to this "replication crisis," explain how researchers can act together to reduce risks, and discuss the positive steps that can increase confidence in research findings. Medical education research contributes to policy and influences practitioner behavior. Findings that cannot be replicated suggest that the original research was not credible. This risk raises the possibility that unhelpful or even harmful changes to medical education have been implemented as a result of research that appeared defensible but was not. By considering these risk factors, researchers can increase the likelihood that studies are generating credible results. The authors discuss and provide examples of 6 factors that may endanger the replicability of medical education research: (1) small sample sizes, (2) small effect sizes, (3) exploratory designs, (4) flexibility in design choices, analysis strategy, and outcome measures, (5) conflicts of interest, and (6) very active fields with many competing research teams. Importantly, medical education researchers can adopt techniques used successfully elsewhere to improve the rigor of their investigations. Researchers can improve their work through better planning in the development stage, carefully considering design choices, and using sensible data analysis. The wider medical education community can help by encouraging higher levels of collaboration among medical educators, by routinely evaluating existing educational innovations, and by raising the prestige of replication and collaborative medical education research. Medical education journals should adopt new approaches to publishing. As medical education research improves, so too will the quality of medical education and patient care.

The Impact of a Medical Education Research Faculty Development Program on Career Development, Through the Lens of Social Cognitive Career Theory

OBJECTIVES

The Medical Education Research Certificate at the Council of Residency Directors in Emergency Medicine (MERC at CORD), a specialized adaptation of the Association of American Medical Colleges MERC program, provides faculty development in education research in emergency medicine. However, its long-term influence on career development remains unknown. Our study explored the impact of MERC at CORD on career development through the lens of social cognitive career (SCC) theory.

METHODS

This was a prospective qualitative study using a constructivist/interpretivist paradigm to assess long-term career development outcomes. A purposeful randomized stratified sampling strategy of MERC at CORD graduates (2011-2014) ensured diversity of representation (sex, region, number of research publications, and project group leadership). Subjects were invited by e-mail to participate in semistructured phone interviews. Thematic analysis by two independent reviewers followed an iterative process until saturation was reached.

RESULTS

Twelve graduates were interviewed. All engaged with MERC at CORD early in their careers with minimal previous education research experience. Currently, all hold medical education leadership positions. Graduates had a mean of 19.3 publications (range = 9-43). Themes explaining reasons for participating in MERC at CORD include: desire for education research skills, recommendation of mentors/colleagues, and accessibility. Themes citing the program's value to career development include networking/collaboration, mentorship, informational framework to build upon, and the application of theoretical knowledge through experiential learning. MERC at CORD impacted career development aligning with the core domains of SCC theory including self-efficacy, outcome expectations, and goals.

CONCLUSION

MERC at CORD enhanced the long-term career development of participants by providing a core knowledge framework in a mentored, experiential learning environment. Participants identified themes aligned with SCC theory as influential in their long-term career advancement in medical education including the development of education research skills, successful completion of education research, career acceleration, promotion, niche development, and formulation of professional goals.

Effectiveness of blended learning in pharmacy education: A systematic review and meta-analysis

Background & objective

Though blended learning (BL), is widely adopted in higher education, evaluating effectiveness of BL is difficult because the components of BL can be extremely heterogeneous. Purpose of this study was to evaluate the effectiveness of BL in improving knowledge and skill in pharmacy education.

Methods

PubMed/MEDLINE, Scopus and the Cochrane Library were searched to identify published literature. The retrieved studies from databases were screened for its title and abstracts followed by the full-text in accordance with the pre-defined inclusion and exclusion criteria. Methodological quality was appraised by modified Ottawa scale. Random effect model used for statistical modelling.

Key findings

A total of 26 studies were included for systematic review. Out of which 20 studies with 4525 participants for meta-analysis which employed traditional teaching in control group. Results showed a statistically significant positive effect size on knowledge (standardized mean difference [SMD]: 1.35, 95% confidence interval [CI]: 0.91 to 1.78, p<0.00001) and skill (SMD: 0.68; 95% CI: 0.19 to 1.16; p = 0.006) using a random effect model. Subgroup analysis of cohort studies showed, studies from developed countries had a larger effect size (SMD: 1.54, 95% CI: 1.01 to 2.06), than studies from developing countries(SMD: 0.44, 95% CI: 0.23 to 0.65, studies with MCQ pattern as outcome assessment had larger effect size (SMD: 2.81, 95% CI: 1.76 to 3.85) than non-MCQs (SMD 0.53, 95% CI 0.33 to 0.74), and BL with case studies (SMD 2.72, 95% CI 1.86–3.59) showed better effect size than non-case-based studies (SMD: 0.22, CI: 0.02 to 0.41).

Conclusion

BL is associated with better academic performance and achievement than didactic teaching in pharmacy education.

Quality Appraisal and Assurance Techniques for Free Open Access Medical Education (FOAM) Resources: A Rapid Review

Free open access medical education (FOAM) has disrupted traditional modes of knowledge translation and dissemination. These are popular resources with a wide educational reach. Nephrology has been a leader in FOAM, but many skeptics still question the accuracy and reliability of this content. Recently, quality-assurance techniques have been developed to address these concerns. These techniques may be helpful for readers to appraise the online literature and for institutions to reward the production of high-quality open educational resources. We performed a rapid review of the literature. A medical librarian conducted a systematic search of the Medline and Cumulative Index of Nursing and Allied Health Literature databases. Two independent assessors screened and selected articles, performed a hand-search of reference lists, and scored articles on their quality using the Medical Education Research Study Quality Instrument. Thirteen reports were included for the final descriptive analysis. We identified 10 quality-assessment techniques, and 4 of them having been validated. The quality of the reports was fairly high, with an average Medical Education Research Study Quality Instrument score of 11.5 of 18 (SD, 2.3; range, 7.25-14.25). The calculated Cronbach α was 0.85. There is burgeoning literature on the topic of critical appraisal of open educational resources, and, more specifically, FOAM resources. Many of the techniques used are of varying quality and developed with different intended uses and audiences. By continuing to refine these tools, we can continue not only to support and legitimize the FOAM movement, but also foster individual critical appraisal skills that increasingly are necessary in this age of information.

Improving hospital falls screening and mitigation using a health professional education framework

OBJECTIVE

Although health professional education has the potential to mitigate hospital falls risk, the best methods to develop, deliver and evaluate health professional education remain unclear. This study applied evidence-based approaches to education design to improve falls risk mitigation.

DESIGN

Mixed methods using questionnaires to evaluate health professionals knowledge of evidence-based falls risk assessment and mitigation, followed by semi-structured interviews with individual health professionals.

SETTING

Five large Australian hospitals.

PARTICIPANTS

For each hospital, 10 clinical leaders from nursing and allied health professions were invited to participate in falls workshops.

METHODS

46 participants received a three-hour education program on the latest evidence in hospital falls risk assessment and how to implement evidence-based falls screening and management. This was based on the "4P" education model (Presage, Planning, Process and Product). They were taught practical skills to enable them to educate other health professionals.

RESULTS

The education workshop significantly changed participants' views about best practice guidelines for falls screening and prevention. Participants felt more confident in assessing falls risk and judging and implementing the best mitigation strategies. They were prepared and motivated to educate others about falls prevention and satisfied with the skills gained.

CONCLUSIONS

A high-quality education program grounded in a rigorous quality framework improved health professionals knowledge regarding evidence-based falls prevention. Use of evidence-based rationales for behaviour change promotes effective learning.

Unpacking student learning from an early experience with the Pharmacists' Patient Care Process

BACKGROUND

To illuminate learning, a case study approach was used to examine early, authentic experiences within Pharmacists' Patient Care Process (PPCP)-focused practices.

EDUCATIONAL ACTIVITY

Six students were matched with five practitioners and spent five half-days in a primary care clinic in a PPCP-committed health system. Students practiced interviewing, determining the patient's medication experience, and formulating the beginnings of the assessment, as well as observing and debriefing on the completion of the process by a practitioner mentor. The Five R Model was used to prompt student learning reflection. In addition, instructors examined students' work for evidence of transformative learning and observations were captured using forms of reflective practice and collaborative debriefing.

CRITICAL ANALYSIS OF THE EDUCATIONAL ACTIVITY

Reflection performance ratings varied; however, there was strong evidence of transformative learning for all students. Specifically, most student reflections demonstrated a focus on elaborating on existing frames of reference. The most prevalent indicator of transformative learning was exploration of options for new roles, relationships, and actions. The codes from instructors' observations revealed five categories of learning evidence, with the strongest in the patient centeredness category. The process of reviewing student work products, documenting instructor observations, and collaborative debriefing resulted in insights for curricular improvement and explanations for learning difficulties. Further work is needed in understanding student experiential learning intentions and their influence on learning and reflection. Additionally, further research should explore the value of longitudinal assessment of reflection and the value of assessing student work products using criteria beyond traditional reflection criteria.

Education interventions for health professionals on falls prevention in health care settings: a 10-year scoping review

Background

Falls in hospitals are a major risk to patient safety. Health professional education has the potential to be an important aspect of falls prevention interventions. This scoping review was designed to investigate the extent of falls prevention education interventions available for health professionals, and to determine the quality of reporting.

Method

A five stage scoping review process was followed based on Arksey and O’Malley’s framework and refined by the Joanna Briggs Institute Methodology for JBI Scoping Reviews. Five online databases identified papers published from January 2008 until May 2019. Papers were independently screened by two reviewers, and data extracted and analysed using a quality reporting framework.

Results

Thirty-nine publications were included. Interventions included formal methods of educational delivery (for example, didactic lectures, video presentations), interactive learning activities, experiential learning, supported learning such as coaching, and written learning material. Few studies employed comprehensive education design principles. None used a reporting framework to plan, evaluate, and document the outcomes of educational interventions.

Conclusions

Although health professional education is recognised as important for falls prevention, no uniform education design principles have been utilised in research published to date, despite commonly reported program objectives. Standardised reporting of education programs has the potential to improve the quality of clinical practice and allow studies to be compared and evaluated for effectiveness across healthcare settings.

The assessment of procedural skills in physiotherapy education: a measurement study using the Rasch model

Background

Procedural skills are a key element in the training of future physiotherapists. Procedural skills relate to the acquisition of appropriate motor skills, which allow the safe application of clinical procedures to patients. In order to evaluate procedural skills in physiotherapy education validated assessment instruments are required. Recently the assessment of procedural skills in physiotherapy education (APSPT) tool was developed. The overall aim of this study was to establish the structural validity of the APSPT. In order to do this the following objectives were examined: i) the fit of the items of APSPT to the Rasch-model, ii) the fit of the overall score to the Rasch model, iii) the difficulty of each test item and iv) whether the difficulty levels of the individual test items cover the whole capacity spectrum of students in pre-registration physiotherapy education.

Methods

For this observational cross-sectional measurement properties study a convenience sample of 69 undergraduate pre-registration physiotherapy students of the HES-SO Valais-Wallis was recruited. Participants were instructed to perform a task procedure on a simulated patient. The performance was evaluated with the APSPT. A conditional maximum likelihood approach was used to estimate the parameters of a partial credit model for polytomous item responses. Item fit, ordering of thresholds, targeting and goodness of fit to the Rasch model was assessed.

Results

Item fit statistics showed that 25 items of the APSPT showed adequate fit to the Rasch model. Disordering of item thresholds did not occur and the targeting of the APSPT was adequate to measure the abilities of the included participants. Undimensionality and subgroup homogeneity were confirmed.

Conclusion

This study presented evidence for the structural validity of the APSPT. Undimensionality of the APSPT was confirmed and therefore presents evidence that the latent dimension of procedural skills in physiotherapy education consists of several subcategories. However, the results should be interpreted with caution given the small sample size.

Leadership training programs in graduate medical education: a systematic review

BACKGROUND

With the increasing recognition that leadership skills can be acquired, there is a heightened focus on incorporating leadership training as a part of graduate medical education. However, there is considerable lack of agreement regarding how to facilitate acquisition of these skills to resident, chief resident, and fellow physicians.

METHODS

Articles were identified through a search of Ovid MEDLINE, EMBASE, CINAHL, ERIC, PsycNet, Cochrane Systemic Reviews, and Cochrane Central Register of Controlled Trials from 1948 to 2019. Additional sources were identified through contacting authors and scanning references. We included articles that described and evaluated leadership training programs in the United States and Canada. Methodological quality was assessed via the MERSQI (Medical Education Research Study Quality Instrument).

RESULTS

Fifteen studies, which collectively included 639 residents, chief residents, and fellows, met the eligibility criteria. The format, content, and duration of these programs varied considerably. The majority focused on conflict management, interpersonal skills, and stress management. Twelve were prospective case series and three were retrospective. Seven used pre- and post-test surveys, while seven used course evaluations. Only three had follow-up evaluations after 6 months to 1 year. MERSQI scores ranged from 6 to 9.

CONCLUSIONS

Despite interest in incorporating structured leadership training into graduate medical education curricula, there is a lack of methodologically rigorous studies evaluating its effectiveness. High-quality well-designed studies, focusing particularly on the validity of content, internal structure, and relationship to other variables, are required in order to determine if these programs have a lasting effect on the acquisition of leadership skills.

The Effectiveness of Teaching Clinical Empathy to Medical Students: A Systematic Review and Meta-Analysis of Randomized Controlled Trials

PURPOSE

Clinical empathy is a necessary trait to provide effective patient care, despite differences in how it is defined and constructed. The aim of this study was to examine whether empathy interventions in medical students are effective and how confounding factors potentially moderate this effect.

METHOD

The authors performed a systematic review and meta-analysis. They searched the literature published between 1948 and 2018 for randomized controlled trials that examined empathy interventions in medical students. The search (database searching, citation tracking, hand-searching relevant journals) yielded 380 studies, which they culled to 16 that met the inclusion criteria. For the meta-analysis, they used a random effects model to produce a pooled estimate of the standardized mean difference (SMD), then completed subgroup analyses.

RESULTS

The authors found evidence of the possibility of response and reporting bias. The pooled SMD was 0.68 (95% confidence interval 0.43, 0.93), indicating a moderately positive effect of students developing empathy after an intervention compared with those in the control groups. There was no evidence of publication bias, but heterogeneity was significantly high (I = 88.5%, P < .01). Subgroup analyses indicated that significant moderating factors for developing empathy were age, country, scope of empathy measurement, type of empathy intervention, and presence of rehearsal. Moderating factors with limited evidence were sex, study quality, journal impact factor, and intervention characteristics.

CONCLUSIONS

Despite heterogeneity and biases, empathy interventions in medical students are effective. These findings reinforce arguments in the literature and add considerable rigor from the meta-analysis. The authors propose a conceptual model for educators to follow when designing empathy interventions in medical students.

How do novices learn physical examination skills? A systematic review of the literature

BACKGROUND

Physical Examination (PE) skills are vital for patient care, and many medical students receive their first introduction to them in their pre-clinical years. A substantial amount of curriculum time is devoted to teaching these skills in most schools. Little is known about the best way to introduce PE skills to novice learners.

OBJECTIVE

Our objective was to conduct a systematic review of how medical students are first taught PE skills and the evidence supporting these strategies.

DESIGN

We searched ERIC, SCOPUS, MEDLINE, PubMed and EMBASE for descriptions of complete PE curricula for novice learners. Inclusion criteria were: (1) English language; (2) subjects were enrolled in medical school and were in the preclinical portion of their training; (3) description of a method to teach physical examination skills for the first time; (4) description of the study population; (5) Description of a complete PE curriculum. We used the Medical Education Research Study Quality Instrument (MERSQI) score to evaluate the quality of evidence provided.

RESULTS

Our search returned 5,418 articles; 32 articles met our inclusion criteria. Two main types of curricula were reported: comprehensive 'head-to-toe' PE curricula (18%) and organ system-based curricula (41%). No studies compared these directly, and only two evaluated trainees' clinical performance. The rest of the articles described interventions used across curricula (41%). Median MERSQI score was 10.1 Interquartile range 8.1-12.4. We found evidence for the use of non-faculty teaching associates, technology-enhanced PE education, and the addition of clinical exposure to formal PE teaching.

CONCLUSIONS

The current literature on teaching PE is focused on describing innovations to head-to-toe and organ system-based curricula rather than their relative effectiveness, and is further limited by its reliance on short-term outcomes. The optimal strategy for novice PE instruction remains unknown.

Cost evaluations in health professions education: a systematic review of methods and reporting quality

CONTEXT

High-quality research into education costs can inform better decision making. Improvements to cost research can be guided by information about the research questions, methods and reporting of studies evaluating costs in health professions education (HPE). Our objective was to appraise the overall state of the field and evaluate temporal trends in the methods and reporting quality of cost evaluations in HPE research.

METHODS

We searched the MEDLINE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), EMBASE, Business Source Complete and ERIC (Education Resources Information Centre) databases on 31 July 2017. To evaluate trends over time, we sampled research reports at 5-year intervals (2001, 2006, 2011 and 2016). All original research studies in HPE that reported a cost outcome were included. The Medical Education Research Study Quality Instrument (MERSQI) and the BMJ economic checklist were used to appraise methodological and reporting quality, respectively. Trends in quality over time were analysed.

RESULTS

A total of 78 studies were included, of which 16 were published in 2001, 15 in 2006, 20 in 2011 and 27 in 2016. The region most commonly represented was the USA (n = 43). The profession most commonly referred to was that of the physician (n = 46). The mean ± standard deviation (SD) MERSQI score was 10.9 ± 2.6 out of 18, with no significant change over time (p = 0.55). The mean ± SD BMJ score was 13.5 ± 7.1 out of 35, with no significant change over time (p = 0.39). A total of 49 (63%) studies stated a cost-related research question, 23 (29%) stated the type of cost evaluation used, and 31 (40%) described the method of estimating resource quantities and unit costs. A total of 16 studies compared two or more interventions and reported both cost and learning outcomes.

CONCLUSIONS

The absolute number of cost evaluations in HPE is increasing. However, there are shortcomings in the quality of methodology and reporting, and these are not improving over time.

Use of the "Step-back" Method for Education Research Consultation at the National Level: A Pilot Study

BACKGROUND

There are a limited number of emergency medicine (EM) physicians with expertise in education research. The Harvard Macy "step-back" method is an emerging model utilized to gather group feedback. Despite its use in multiple educational settings, there are little published data demonstrating effectiveness.

OBJECTIVES

Our objective was to create and evaluate a national faculty development session providing consultation in education research utilizing the step-back method.

METHODS

This was a pilot study. EM experts in education research from across the country served as facilitators for a faculty development session held at the 2018 Council of Emergency Medicine Residency Directors Academic Assembly. Small groups consisting of two or three facilitators and one or two participants were formed and each participant underwent a step-back consultation for their education research study. Participants wrote their study question before and after the session. After the session, facilitators and participants completed an evaluative survey consisting of multiple-choice, Likert-type, and free-response items. Descriptive statistics were reported. Qualitative analysis using a thematic approach was performed on free-response data. Participant study questions were assessed by the PICO (population, intervention, comparison, outcome) and FINER (feasible, interesting, novel, ethical, relevant) criteria. Both scales were evaluated using a two-way random-consistency intraclass correlation. Before and after scores were evaluated with a paired t-test.

RESULTS

Twenty-four facilitators and 13 participants completed the step-back session. Evaluations from 20 facilitators and nine participants were submitted and analyzed. Sixteen of 20 facilitators felt that the step-back method "greatly facilitated" their ability to share their education research expertise. All facilitators and participants recommended that the session be provided at a future academic assembly. Regarding suggestions for improvement, qualitative analysis revealed three major themes: praise for the session, desire for additional time, and a room set up more conducive to small group work. Seven of nine responding participants felt that the session was "very valuable" for improving the strength of their study methods. Qualitative analysis regarding change in study as a result of the step-back session yielded four major themes: refinement of study question, more specific outcomes and measurements, improvement in study design, and greater understanding of study limitations. Both FINER and PICO scale comparisons showed improvement pre- and postintervention (PICO 60% relative increase; FINER 16% relative increase). Neither achieved statistical significance (PICO t(5) = -1.835, p = 0.126; and FINER t(5) = -1.305, p = 0.249).

CONCLUSION

A national-level education research consultation utilizing the step-back method was feasible to implement and highly valued by facilitators and participants. Potential positive outcomes include refinement of study question, more specific outcomes and measurements, improvement in study design, and greater understanding of limitations. These results may inform others who want to utilize this method.

Quantitative Research Methods in Medical Education

There has been a dramatic growth of scholarly articles in medical education in recent years. Evaluating medical education research requires specific orientation to issues related to format and content. Our goal is to review the quantitative aspects of research in medical education so that clinicians may understand these articles with respect to framing the study, recognizing methodologic issues, and utilizing instruments for evaluating the quality of medical education research. This review can be used both as a tool when appraising medical education research articles and as a primer for clinicians interested in pursuing scholarship in medical education.

Critical Appraisal of Emergency Medicine Educational Research: The Best Publications of 2016

OBJECTIVES

The objectives were to critically appraise the emergency medicine (EM) medical education literature published in 2016 and review the highest-quality quantitative and qualitative studies.

METHODS

A search of the English language literature in 2016 querying MEDLINE, Scopus, Education Resources Information Center (ERIC), and PsychInfo identified 510 papers related to medical education in EM. Two reviewers independently screened all of the publications using previously established exclusion criteria. The 25 top-scoring quantitative studies based on methodology and all six qualitative studies were scored by all reviewers using selected scoring criteria that have been adapted from previous installments. The top-scoring articles were highlighted and trends in medical education research were described.

RESULTS

Seventy-five manuscripts met inclusion criteria and were scored. Eleven quantitative and one qualitative papers were the highest scoring and are summarized in this article.

CONCLUSION

This annual critical appraisal series highlights the best EM education research articles published in 2016.

Research Pioneers in Emergency Medicine-Reflections on Their Paths to Success and Advice to Aspiring Researchers: A Qualitative Study

STUDY OBJECTIVE

Research in basic, translational, and clinical emergency medicine has made great strides since the formalization of emergency medicine as a specialty. Our objective is to identify and analyze strategies used by emergency medicine research pioneers to inform further advancement of research in emergency medicine, particularly for aspiring researchers and those in emerging areas, using emergency medicine medical education as one example.

METHODS

This was a prospective, grounded-theory, qualitative study, using a constructivist/interpretivist paradigm. Leading basic science, translational, and clinical emergency medicine researchers who completed residency before 1995 were eligible for structured interviews. Thematic coding followed an iterative process until saturation was reached. A theoretic model was developed and analyzed.

RESULTS

Research pioneers valued advanced methodological training and mentorship. Barriers to funding were lack of recognition of emergency medicine as a specialty, absence of a research history, and lack of training and funding resources. Deliberate interventions to improve emergency medicine research included educational sessions at national meetings, external (to emergency medicine) mentor pairings, targeted funding by emergency medicine organizations, and involvement with funding agencies. Pioneers facilitate research excellence by serving as mentors and allocating funds or protected time to develop researchers. To advance emerging subfields of research in emergency medicine, pioneers recommend advanced methodological training that is specific to the area, deliberate mentorship, and the formation of research consortia to conduct generalizable outcomes-based studies.

CONCLUSION

Research pioneers in emergency medicine cite mentorship, advanced skills obtained through fellowship or graduate degrees, deliberate collaboration with experienced researchers, support from emergency medicine organizations, and forming networks as the cornerstones of success.

Does the medical literature remain inadequately described despite having reporting guidelines for 21 years? - A systematic review of reviews: an update

PURPOSE

Reporting guidelines (eg, Consolidated Standards of Reporting Trials [CONSORT] statement) are intended to improve reporting standards and enhance the transparency and reproducibility of research findings. Despite accessibility of such guidelines, researchers are not required to adhere to them. Our goal was to determine the current status of reporting quality in the medical literature and examine whether adherence of reporting guidelines has improved since the inception of reporting guidelines.

MATERIALS AND METHODS

Eight reporting guidelines, such as CONSORT, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), STrengthening the Reporting of OBservational studies in Epidemiology (STROBE), Quality of Reporting of Meta-analysis (QUOROM), STAndards for Reporting of Diagnostic accuracy (STARD), Animal Research: Reporting In Vivo Experiments (ARRIVE), Consolidated Health Economic Evaluation Reporting Standards (CHEERS), and Meta-analysis of Observational Studies in Epidemiology (MOOSE) were examined. Our inclusion criteria included reviews published between January 1996 to September 2016 which investigated the adherence to reporting guidelines in the literature that addressed clinical trials, systematic reviews, observational studies, meta-analysis, diagnostic accuracy, economic evaluations, and preclinical animal studies that were in English. All reviews were found on Web of Science, Excerpta Medical Database (EMBASE), MEDLINE, and Cumulative Index to Nursing and Allied Health Literature (CINAHL).

RESULTS

Among the general searching of 26,819 studies by using the designed searching method, 124 studies were included post screening. We found that 87.9% of the included studies reported suboptimal adherence to reporting guidelines. Factors associated with poor adherence included non-pharmacological interventions, year of publication, and trials concluding with significant results. Improved adherence was associated with better study designs such as allocation concealment, random sequence, large sample sizes, adequately powered studies, multiple authorships, and being published in journals endorsing guidelines.

CONCLUSION

We conclude that the level of adherence to reporting guidelines remains suboptimal. Endorsement of reporting guidelines by journals is important and recommended.

CAEP 2016 Academic Symposium: A Writer's Guide to Key Steps in Producing Quality Medical Education Scholarship

A key skill for successful clinician educators is the effective dissemination of scholarly innovations and research. Although there are many ways to disseminate scholarship, the most accepted and rewarded form of educational scholarship is publication in peer-reviewed journals. This paper provides direction for emergency medicine (EM) educators interested in publishing their scholarship via traditional peer-reviewed avenues. It builds upon four literature reviews that aggregated recommendations for writing and publishing high-quality quantitative and qualitative research, innovations, and reviews. Based on the findings from these literature reviews, the recommendations were prioritized for importance and relevance to novice clinician educators by a broad community of medical educators. The top items from the expert vetting process were presented to the 2016 Canadian Association of Emergency Physicians (CAEP) Academic Symposium Consensus Conference on Education Scholarship. This community of EM educators identified the highest yield recommendations for junior medical education scholars. This manuscript elaborates upon the top recommendations identified through this consensus-building process.

Randomized controlled trials of simulation-based interventions in Emergency Medicine: a methodological review

The number of trials assessing Simulation-Based Medical Education (SBME) interventions has rapidly expanded. Many studies show that potential flaws in design, conduct and reporting of randomized controlled trials (RCTs) can bias their results. We conducted a methodological review of RCTs assessing a SBME in Emergency Medicine (EM) and examined their methodological characteristics. We searched MEDLINE via PubMed for RCT that assessed a simulation intervention in EM, published in 6 general and internal medicine and in the top 10 EM journals. The Cochrane Collaboration risk of Bias tool was used to assess risk of bias, intervention reporting was evaluated based on the "template for intervention description and replication" checklist, and methodological quality was evaluated by the Medical Education Research Study Quality Instrument. Reports selection and data extraction was done by 2 independents researchers. From 1394 RCTs screened, 68 trials assessed a SBME intervention. They represent one quarter of our sample. Cardiopulmonary resuscitation (CPR) is the most frequent topic (81%). Random sequence generation and allocation concealment were performed correctly in 66 and 49% of trials. Blinding of participants and assessors was performed correctly in 19 and 68%. Risk of attrition bias was low in three-quarters of the studies (n = 51). Risk of selective reporting bias was unclear in nearly all studies. The mean MERQSI score was of 13.4/18.4% of the reports provided a description allowing the intervention replication. Trials assessing simulation represent one quarter of RCTs in EM. Their quality remains unclear, and reproducing the interventions appears challenging due to reporting issues.

The Uphill Battle of Performing Education Scholarship: Barriers Educators and Education Researchers Face

Introduction

Educators and education researchers report that their scholarship is limited by lack of time, funding, mentorship, expertise, and reward. This study aims to evaluate these groups’ perceptions regarding barriers to scholarship and potential strategies for success.

Methods

Core emergency medicine (EM) educators and education researchers completed an online survey consisting of multiple-choice, 10-point Likert scale, and free-response items in 2015. Descriptive statistics were reported. We used qualitative analysis applying a thematic approach to free-response items.

Results

A total of 204 educators and 42 education researchers participated. Education researchers were highly productive: 19/42 reported more than 20 peer-reviewed education scholarship publications on their curricula vitae. In contrast, 68/197 educators reported no education publications within five years. Only a minority, 61/197 had formal research training compared to 25/42 education researchers. Barriers to performing research for both groups were lack of time, competing demands, lack of support, lack of funding, and challenges achieving scientifically rigorous methods and publication. The most common motivators identified were dissemination of knowledge, support of evidence-based practices, and promotion. Respondents advised those who seek greater education research involvement to pursue mentorship, formal research training, collaboration, and rigorous methodological standards.

Conclusion

The most commonly cited barriers were lack of time and competing demands. Stakeholders were motivated by the desire to disseminate knowledge, support evidence-based practices, and achieve promotion. Suggested strategies for success included formal training, mentorship, and collaboration. This information may inform interventions to support educators in their scholarly pursuits and improve the overall quality of education research in EM.

Comparing Quick Sequential Organ Failure Assessment Scores to End-tidal Carbon Dioxide as Mortality Predictors in Prehospital Patients with Suspected Sepsis

Introduction

Early identification of sepsis significantly improves outcomes, suggesting a role for prehospital screening. An end-tidal carbon dioxide (ETCO₂) value ≤ 25 mmHg predicts mortality and severe sepsis when used as part of a prehospital screening tool. Recently, the Quick Sequential Organ Failure Assessment (qSOFA) score was also derived as a tool for predicting poor outcomes in potentially septic patients.

Methods

We conducted a retrospective cohort study among patients transported by emergency medical services to compare the use of ETCO₂ ≤ 25 mmHg with qSOFA score of ≥ 2 as a predictor of mortality or diagnosis of severe sepsis in prehospital patients with suspected sepsis.

Results

By comparison of receiver operator characteristic curves, ETCO₂ had a higher discriminatory power to predict mortality, sepsis, and severe sepsis than qSOFA.

Conclusion

Both non-invasive measures were easily obtainable by prehospital personnel, with ETCO₂ performing slightly better as an outcome predictor.

Completeness of the reporting of evidence-based practice educational interventions: a review

CONTEXT

Complete reporting of intervention details in trials of evidence-based practice (EBP) educational interventions is essential to enable clinical educators to translate research evidence about interventions that have been shown to be effective into practice. In turn, this will improve the quality of EBP education.

OBJECTIVES

This study was designed to examine the completeness of reporting of EBP educational interventions in published studies and to assess whether missing details of educational interventions could be retrieved by searching additional sources and contacting study authors.

METHODS

A systematic review of controlled trials that had evaluated EBP educational interventions was conducted using a citation analysis technique. Forward and backward citations of the index articles were tracked until March 2016. The TIDieR (template for intervention description and replication) checklist was used to assess the completeness of intervention reporting. Missing details were sought from: (i) the original publication; (ii) additional publicly available sources, and (iii) the study authors.

RESULTS

Eighty-three articles were included; 45 (54%) were randomised controlled trials (RCTs) and 38 (46%) were non-RCTs. The majority of trials (n = 62, 75%) involved medical professionals. None of the studies completely reported all of the main items of the educational intervention within the original publication or in additional sources. However, details became complete for 17 (20%) interventions after contact with the respective authors. The item most frequently missing was 'intervention materials', which was missing in 80 (96%) of the original publications, in additional sources for 77 (93%) interventions, and in 59 (71%) studies after contact with the authors. Authors of 69 studies were contacted; 33 provided the details requested.

CONCLUSIONS

The reporting of EBP educational interventions is incomplete and remained so for the majority of studies, even after study authors had been contacted for missing information. Collaborative efforts involving authors and editors are required to improve the completeness of reporting of EBP educational interventions.

The View From the Top: Academic Emergency Department Chairs' Perspectives on Education Scholarship

Education scholarship continues to grow within emergency medicine (EM) and in academic medicine in general. Despite a growing interest, would-be education scholars often struggle to find adequate mentorship, research training, funding, and protected time to produce rigorous scholarship. The ways in which individual academic EM departments can support this mission remains an area in need of description.

OBJECTIVES

We sought to describe academic EM department chairs' perceptions of education scholarship and facilitators and barriers to producing high-quality education scholarship.

METHODS

We conducted a qualitative study using a grounded theory-derived approach. Participants were solicited directly, and semistructured interviews were conducted via telephone. Interviews were transcribed verbatim and were analyzed by three study investigators using a coding matrix. Discrepancies in coding were resolved via in depth discussion.

RESULTS

We interviewed seven EM chairs from academic departments throughout North America (six in geographically diverse regions of the United States and one in western Canada). Chairs described education scholarship as lacking clearly defined and measurable outcomes, as well as methodologic rigor. They identified that education faculty within their departments need training and incentives to pursue scholarly work in a system that primarily expects teaching from educators. Chairs acknowledged a lack of access to education research expertise and mentorship within their own departments, but identified potential resources within their local medical schools and universities. They also voiced willingness to support career development opportunities and scholarly work among faculty seeking to perform education research.

CONCLUSIONS

Academic EM chairs endorse a need for methodologic training, mentorship, and access to expertise specific to education scholarship. While such resources are often rare within academic EM departments, they may exist within local universities and schools of medicine. Academic EM chairs described themselves as willing and able to support faculty who wish to pursue this type of work.

A systematic review and meta-analysis of online versus alternative methods for training licensed health care professionals to deliver clinical interventions

BACKGROUND

Online training is growing in popularity and yet its effectiveness for training licensed health professionals (HCPs) in clinical interventions is not clear. We aimed to systematically review the literature on the effectiveness of online versus alternative training methods in clinical interventions for licensed Health Care Professionals (HCPs) on outcomes of knowledge acquisition, practical skills, clinical behaviour, self-efficacy and satisfaction.

METHODS

Seven databases were searched for randomised controlled trials (RCTs) from January 2000 to June 2015. Two independent reviewers rated trial quality and extracted trial data. Comparative effects were summarised as standardised mean differences (SMD) and 95% confidence intervals. Pooled effect sizes were calculated using a random-effects model for three contrasts of online versus (i) interactive workshops (ii) taught lectures and (iii) written/electronic manuals.

RESULTS

We included 14 studies with a total of 1089 participants. Most trials studied medical professionals, used a workshop or lecture comparison, were of high risk of bias and had small sample sizes (range 21-183). Using the GRADE approach, we found low quality evidence that there was no difference between online training and an interactive workshop for clinical behaviour SMD 0.12 (95% CI -0.13 to 0.37). We found very low quality evidence of no difference between online methods and both a workshop and lecture for knowledge (workshop: SMD 0.04 (95% CI -0.28 to 0.36); lecture: SMD 0.22 (95% CI: -0.08, 0.51)). Lastly, compared to a manual (n = 3/14), we found very low quality evidence that online methods were superior for knowledge SMD 0.99 (95% CI 0.02 to 1.96). There were too few studies to draw any conclusions on the effects of online training for practical skills, self-efficacy, and satisfaction across all contrasts.

CONCLUSIONS

It is likely that online methods may be as effective as alternative methods for training HCPs in clinical interventions for the outcomes of knowledge and clinical behaviour. However, the low quality of the evidence precludes drawing firm conclusions on the relative effectiveness of these training methods. Moreover, the confidence intervals around our effect sizes were large and could encompass important differences in effectiveness. More robust, adequately powered RCTs are needed.

Conference presentation to publication: a retrospective study evaluating quality of abstracts and journal articles in medical education research

BACKGROUND

There is little evidence regarding the comparative quality of abstracts and articles in medical education research. The Medical Education Research Study Quality Instrument (MERSQI), which was developed to evaluate the quality of reporting in medical education, has strong validity evidence for content, internal structure, and relationships to other variables. We used the MERSQI to compare the quality of reporting for conference abstracts, journal abstracts, and published articles.

METHODS

This is a retrospective study of all 46 medical education research abstracts submitted to the Society of General Internal Medicine 2009 Annual Meeting that were subsequently published in a peer-reviewed journal. We compared MERSQI scores of the abstracts with scores for their corresponding published journal abstracts and articles. Comparisons were performed using the signed rank test.

RESULTS

Overall MERSQI scores increased significantly for published articles compared with conference abstracts (11.33 vs 9.67; P < .001) and journal abstracts (11.33 vs 9.96; P < .001). Regarding MERSQI subscales, published articles had higher MERSQI scores than conference abstracts in the domains of sampling (1.59 vs 1.34; P = .006), data analysis (3.00 vs 2.43; P < .001), and validity of evaluation instrument (1.04 vs 0.28; P < .001). Published articles also had higher MERSQI scores than journal abstracts in the domains of data analysis (3.00 vs 2.70; P = .004) and validity of evaluation instrument (1.04 vs 0.26; P < .001).

CONCLUSIONS

To our knowledge, this is the first study to compare the quality of medical education abstracts and journal articles using the MERSQI. Overall, the quality of articles was greater than that of abstracts. However, there were no significant differences between abstracts and articles for the domains of study design and outcomes, which indicates that these MERSQI elements may be applicable to abstracts. Findings also suggest that abstract quality is generally preserved from original presentation to publication.

Difficult Interpersonal Encounters with Medical Students and Residents: Two Objective Standardized Teaching Encounters

INTRODUCTION

Objective standardized teaching exercises (OSTEs) are widely used to develop professional competencies, especially in the health care professions. An OSTE involves exposing different providers to the same, time-limited scenario that is concurrently observed and/or recorded for either formative or summative evaluation. As there are limited resources available for creating a resident-specific OSTE, especially those applicable to family and community medicine residents, we created and evaluated a resident OSTE (R-OSTE) for second- and third-year family and community medicine residents.

METHODS

This R-OSTE involved two cases. The first featured Taylor, a third-year medical student resistant to feedback. The second featured Kris, a first-year resident nervous about approaching the attending on duty. Our R-OSTE had residents teaching interpersonal skills to trained actors in a standardized learner role.

RESULTS

Residents in the teaching role were formatively evaluated by peer observers (fellow residents) and standardized learners on interpersonal domains such as communication and professionalism. Learners gave residents an average performance rating of 4.9 on a 1 to 6 scale with 1 = Very Poor and 6 = Excellent. Residents also evaluated the OSTE itself, rating their experience on multiple teaching-related statements. Eighty-six percent of residents agreed this exercise was an appropriate development activity for family medicine residents. Overall, our R-OSTE was rated highly for relevance to teaching by the residents.

DISCUSSION

The residents were rated highly by both peer observers and standardized learners. However, there was little variability in peer observer scores, indicating the need for an alternative method of measurement.

Creating a Cadre of Fellowship-Trained Medical Educators, Part II: A Formal Needs Assessment to Structure Postgraduate Fellowships in Medical Education Scholarship and Leadership

PURPOSE

Education leaders at the 2012 Academic Emergency Medicine Consensus Conference on education research proposed that dedicated postgraduate education scholarship fellowships (ESFs) might provide an effective model for developing future faculty as scholars. A formal needs assessment was performed to understand the training gap and inform the development of ESFs.

METHOD

A mixed-methods needs assessment was conducted of four emergency medicine national stakeholder groups in 2013: department chairs; faculty education/research leaders; existing education fellowship directors; and current education fellows/graduates. Descriptive statistics were reported for quantitative data. Qualitative data from semistructured interviews and free-text responses were analyzed using a thematic approach.

RESULTS

Participants were 11/15 (73%) education fellowship directors, 13/20 (65%) fellows/graduates, 106/239 (44%) faculty education/research leaders, and a convenience sample of 26 department chairs. Department chairs expected new education faculty to design didactics (85%) and teach clinically (96%). Faculty education/research leaders thought new faculty were inadequately prepared for job tasks (83.7%) and that ESFs would improve the overall quality of education research (91.1%). Fellowship directors noted that ESFs provide skills, mentorship, and protected time for graduates to become productive academicians. Current fellows/graduates reported pursing an ESF to develop skills in teaching and research methodology.

CONCLUSIONS

Stakeholder groups uniformly perceived a need for training in education theory, clinical teaching, and education research. These findings support dedicated, deliberate training in these areas. Establishment of a structure for scholarly pursuits prior to assuming a full-time position will effectively prepare new faculty. These findings may inform the development, implementation, and curricula of ESFs.

Maintenance of Certification and the Challenge of Professionalism

Board certification has been part of the social contract in which physicians commit to maintaining up-to-date scientific knowledge and improving the quality of patient care. However, the maintenance of certification program has been controversial. This review summarizes the philosophical underpinnings, published literature, recent improvements, and future directions of the American Board of Pediatrics maintenance of certification program.

Reporting Guidelines for Health Care Simulation Research: Extensions to the CONSORT and STROBE Statements

INTRODUCTION

Simulation-based research (SBR) is rapidly expanding but the quality of reporting needs improvement. For a reader to critically assess a study, the elements of the study need to be clearly reported. Our objective was to develop reporting guidelines for SBR by creating extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statements.

METHODS

An iterative multistep consensus-building process was used on the basis of the recommended steps for developing reporting guidelines. The consensus process involved the following: (1) developing a steering committee, (2) defining the scope of the reporting guidelines, (3) identifying a consensus panel, (4) generating a list of items for discussion via online premeeting survey, (5) conducting a consensus meeting, and (6) drafting reporting guidelines with an explanation and elaboration document.

RESULTS

The following 11 extensions were recommended for

CONSORT

item 1 (title/abstract), item 2 (background), item 5 (interventions), item 6 (outcomes), item 11 (blinding), item 12 (statistical methods), item 15 (baseline data), item 17 (outcomes/estimation), item 20 (limitations), item 21 (generalizability), and item 25 (funding). The following 10 extensions were recommended for STROBE: item 1 (title/abstract), item 2 (background/rationale), item 7 (variables), item 8 (data sources/measurement), item 12 (statistical methods), item 14 (descriptive data), item 16 (main results), item 19 (limitations), item 21 (generalizability), and item 22 (funding). An elaboration document was created to provide examples and explanation for each extension.

CONCLUSIONS

We have developed extensions for the CONSORT and STROBE Statements that can help improve the quality of reporting for SBR.

Gestalt assessment of online educational resources may not be sufficiently reliable and consistent

PURPOSE

Online open educational resources are increasingly used in medical education, particularly blogs and podcasts. However, it is unclear whether these resources can be adequately appraised by end-users. Our goal was to determine whether gestalt-based recommendations are sufficient for emergency medicine trainees and attending physicians to reliably recommend online educational resources to others.

METHODS

Raters (33 trainees and 21 attendings in emergency medicine from North America) were asked to rate 40 blog posts according to whether, based on their gestalt, they would recommend the resource to (1) a trainee or (2) an attending physician. The ratings' reliability was assessed using intraclass correlation coefficients (ICC). Associations between groups' mean scores were assessed using Pearson's r. A repeated measures analysis of variance (RM-ANOVA) was completed to determine the effect of the level of training on gestalt recommendation scale (i. e. trainee vs. attending).

RESULTS

Trainees demonstrated poor reliability when recommending resources for other trainees (ICC = 0.21, 95% CI 0.13-0.39) and attendings (ICC = 0.16, 95% CI = 0.09-0.30). Similarly, attendings had poor reliability when recommending resources for trainees (ICC = 0.27, 95% CI 0.18-0.41) and other attendings (ICC = 0.22, 95% CI 0.14-0.35). There were moderate correlations between the mean scores for each blog post when either trainees or attendings considered the same target audience. The RM-ANOVA also corroborated that there is a main effect of the proposed target audience on the ratings by both trainees and attendings.

CONCLUSIONS

A gestalt-based rating system is not sufficiently reliable when recommending online educational resources to trainees and attendings. Trainees' gestalt ratings for recommending resources for both groups were especially unreliable. Our findings suggest the need for structured rating systems to rate online educational resources.