Examining Reliability and Validity of an Online Score (ALiEM AIR) for Rating Free Open Access Medical Education Resources

The ID Digital Institute: Building a digital education toolset and community

Organic digital education (ODE) includes online medical education content that can take various forms, such as blogs, social media, videos, podcasts, or infographics. Multimedia ODE platforms have unique benefits and have quickly become an essential part of medical education. Modern medical educators with competency in digital teaching modalities can leverage these for teaching as well as career development and dissemination of scientific research. The ID Digital Institute is a digital education program with a curriculum designed to equip infectious diseases (ID) professionals with the skills to appraise, create, curate, and integrate ODE into their teaching and career. We share the structure, content, and lessons learned from the ID Digital Institute program. We also illustrate how digital education skills can present unique opportunities to align with current and future transplant and immunocompromised host infectious diseases education efforts.

FOAM authorship: Who's teaching our learners?

Background

Free open-access medical education (FOAM) is extremely popular among learners and educators despite lacking the traditional peer review process. Despite the potential for inaccurate, low-quality, or biased content, little has been published describing FOAM authors.

Methods

We performed a cross-sectional analysis of 12 months of content from the top 25 blogs in the 2020 Social Media Index from August 2020-2021. We recorded the number of posts per site and descriptive characteristics of authors, including gender affiliation, conflicts of interest (COI) statements, and type of practice (academic, community, or hybrid).

Results

We identified 2141 posts by 1001 authors. More than half were produced by six websites: EM Docs (266), Life in the Fast Lane (232), EMCrit (188), ALiEM (185), Don't Forget the Bubbles (181), and Rebel EM (174). Most content (1680 posts, 78.5%) lacked a COI statement. Authors were mostly academic (89%), mostly held MD degrees (67.4%), and were mostly men (59.7%). Geographically, most FOAM authors reside in the United States (59.5%), Canada (22.42%), or the United Kingdom (9.4%).

Conclusions

Of all the posts in the top 25 sites in 2020, more than half came from six sites, and authors were largely North American men in academics with MD degrees. Learners, content creators, and educators should consider the ways in which a more diverse authorship pool might bring value to the FOAM educational experience.

SAEM systematic online academic resource (SOAR) review: Gastrointestinal illnesses

Background and Objectives

Free open access medical education (FOAM) has become an essential tool for emergency medicine (EM) education and can be valuable to clinicians as a point-of-care resource. The development of the revised Medical Education Translational Resources Impact and Quality (rMETRIQ) tool provides a standardized means of quality assessment. Previous entries of the Society for Academic Emergency Medicine systematic online academic resource (SOAR) series have focused on renal, endocrine, and sickle cell disorders. In this iteration, we strive to identify, curate, and describe FOAM topics specific to acute gastrointestinal (GI) illnesses.

Methods

We searched 389 keywords across 11 GI topics that were modified from the 2019 Model of the Clinical Practice of EM (EM Model) using the search engine Google FOAM and within the top 50 websites listed on Academic Life in Emergency Medicine's Social Media Index. The sites underwent preliminary screening to eliminate resources that were not relevant to EM or GI illnesses. Identified resources were evaluated with the rMETRIQ tool by five board-certified EM physicians who received rMETRIQ tool rater training.

Results

After duplicates of the initial 39,505 resources were eliminated, 8059 remained. Primary screening resulted in a final 1202 resources. The most common categories were large bowel (18%), small bowel (13%), stomach (11%), esophagus (11%), biliary (11%), and liver (10%). Many resources covered multiple topics and subtopics. The final mean intraclass correlation coefficient among the five physicians was 0.95 (95% CI 0.92-0.98) for rMETRIQ scoring. We identified 256 sites considered "high quality" with a rMETRIQ score of 16 or higher as designated in prior reviews.

Conclusions

This iteration of the SOAR review resulted in the highest number of high-quality resources compared to other SOAR reviews, with 21% of resources thus far scoring ≥ 16. A final list of high-quality resources can guide trainees, educator recommendations, and FOAM authors.

Online learning in health professions education. Part 2: Tools and practical application: AMEE Guide No. 163

Part 1 of the AMEE Guide Online learning in health professions education focused on foundational concepts such as theory, methods, and instructional design in online learning. Part 2 builds upon Part 1, introducing technology tools and applications of these foundational concepts by exploring the various levels (from beginner to advanced) of utilisation, while describing how their usage can transform Health Professions Education. This Part covers Learning Management Systems, infographics, podcasting, videos, websites, social media, online discussion forums, simulation, virtual patients, extended and virtual reality. Intertwined are other topics, such as online small group teaching, game-based learning, FOAM, online social and collaboration learning, and virtual care teaching. We end by discussing digital scholarship and emerging technologies. Combined with Part 1, the overall aim of Part 2 is to produce a comprehensive overview to help guide effective use online learning in Health Professions Education.

Performance of ChatGPT, Bard, Claude, and Bing on the Peruvian National Licensing Medical Examination: a cross-sectional study

PURPOSE

We aimed to describe the performance and evaluate the educational value of justifications provided by artificial intelligence chatbots, including GPT-3.5, GPT-4, Bard, Claude, and Bing, on the Peruvian National Medical Licensing Examination (P-NLME).

METHODS

This was a cross-sectional analytical study. On July 25, 2023, each multiple-choice question (MCQ) from the P-NLME was entered into each chatbot (GPT-3, GPT-4, Bing, Bard, and Claude) 3 times. Then, 4 medical educators categorized the MCQs in terms of medical area, item type, and whether the MCQ required Peru-specific knowledge. They assessed the educational value of the justifications from the 2 top performers (GPT-4 and Bing).

RESULTS

GPT-4 scored 86.7% and Bing scored 82.2%, followed by Bard and Claude, and the historical performance of Peruvian examinees was 55%. Among the factors associated with correct answers, only MCQs that required Peru-specific knowledge had lower odds (odds ratio, 0.23; 95% confidence interval, 0.09-0.61), whereas the remaining factors showed no associations. In assessing the educational value of justifications provided by GPT-4 and Bing, neither showed any significant differences in certainty, usefulness, or potential use in the classroom.

CONCLUSION

Among chatbots, GPT-4 and Bing were the top performers, with Bing performing better at Peru-specific MCQs. Moreover, the educational value of justifications provided by the GPT-4 and Bing could be deemed appropriate. However, it is essential to start addressing the educational value of these chatbots, rather than merely their performance on examinations.

Hunting for Pearls: A Qualitative Analysis of the Reflections of Students Creating Psychiatric Podcasts

Purpose

This paper aims to explore medical student experiences of creating a peer-to-peer psychiatry educational podcast.

Methods

During psychiatry placement, ten year-4 University of Bristol medical students created peer-educational multi-episode podcasts on psychiatric topics. Following completion, they submitted reflective essays on their experiences. Qualitative thematic analysis of these essays was completed by two independent authors. Following data familiarisation, authors independently generated codes that were collated into relevant themes. Upon reaching thematic saturation, findings were collated, and member checking was carried out to confirm the validity of findings.

Results

Themes included effective preparation, choosing content, podcast production, enhancing learning, the weight of responsibility and creating educational support networks. All students found podcast creation to be beneficial for personal learning.

Conclusion

Exploration of students' experiences creating podcasts can support clearer guidance for medical podcast production, providing opportunities for educators to optimise podcast creation efficiency and educational effectiveness.

Needs Assessment of Non-core Content for Podcasts in Emergency Medicine

Study objective The purpose of this study was to assess the use of podcasts among emergency medicine (EM) students, residents, and attending physicians for non-core content. Methods A survey was administered to medical students interested in emergency medicine, current residents of emergency medicine, and attending physicians to determine which podcasts, if any, they listen to and how often. The purpose of listening to these podcasts was also evaluated and determined it was to learn the non-core content of the specialty. The survey was administered online via multiple platforms including Google Forms (Google, Inc., Mountain View, CA), email, and Twitter (San Francisco, CA). The survey was also distributed via messaging application and word of mouth. Results There were 52 responses that were received on the survey. Of the respondents, 73.1% (N = 38) stated that they do listen to medical podcasts. Of these listeners, 21.2% (N = 11) stated that they listen to them once a year, 15.4% (N = 8) once a month, 11.5% (N = 6) once every two weeks, 19.2% (N = 10) weekly, 11.5% (N = 6) 1-2 days per week, 5.8% (N = 3) 3-4 days per week, 5.8% (N = 3) 5-6 days per week, and 9.6% (N = 5) daily. Based on the 52 responses received, 30.8% (N = 16) stated that they listen to them as a supplement to other sources of foundational content, 25% (N = 13) for general interest, 28.8% (N = 15) for entertainment, 7.7% (N = 4) as a primary resource to learn foundational content in their specialty, and 7.7% (N = 4) other. Based on the data metrics, 51.9% (N = 27) stated that 1-5 of the medical podcasts they listen to discuss the practice of emergency medicine, 44.2% (N = 23) stated that none do, and 3.8% (N = 2) stated that more than 10 do. The list of emergency medicine podcasts being listened to are as follows: EMplify, EM:RAP, EM Over Easy, EM Clerkship, Core EM, EM Basic, EMA, EM Cases, Internet Book of Critical Care, Pharm So Hard, EMCRIT, EM Board Bombs, PEM Rules, and REBEL CAST. When asked how important medical podcasts are for learning within the specialty, 44.2% (N = 23) reported not important, 26.9% (N = 14) reported somewhat important, and 28.8% (N = 15) reported very important. When asked how important medical podcasts are for learning outside the specialty, 40% (N = 20) reported not important, 48% (N = 24) reported somewhat important, and 12% (N = 6) stated very important. When asked if their program includes medical education podcasts as part of their curriculum, 18.4% (N = 9) responded yes, while 81.6% (N = 40) responded no. When asked if they believe medical education podcasts should be included as part of their curriculum, 52% (N = 26) stated yes, while 48% (N = 24) stated no. Conclusion Based on the results of the survey, roughly three-quarters of the surveyees listen to medical podcasts. Of the respondents, 38.5% (N = 20) stated that they listen to these podcasts as some form of supplement to learning content (7.7% (N = 4) primary and 30.8% (N = 16) supplementary), which explains the rising availability of the number of podcasts related to emergency medicine. Of the respondents stating that medical podcasts should be included in their curriculum, 52% (N = 26) signifies the importance of quality learning that these medical podcasts provide to their listeners.

Emergency Medicine Procedures: A Review of Approved Instructional Resources from the World of Free Open Access Medical Education

The Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) Series was created in 2014 to address the Free Open Access Medical Education (FOAM) movement's decentralized nature and lack of inherent peer review. The AIR series provides a topic-based, curated list of online educational content vetted by academic emergency medicine (EM) faculty that meets individualized interactive instruction criteria for EM trainees. Relevant FOAM resources were identified from the top 50 FOAM websites using the Social Media Index and then scored by EM faculty using a validated instrument to identify the highest quality posts related to a topic. This article reviews FOAM resources pertaining to EM procedures that were labeled as an "Approved Instructional Resource" or "Honorable Mention" using the AIR series methodology.

Developing a Web-Based Asynchronous Case Discussion Format on Social Media to Teach Clinical Reasoning: Mixed Methods Study

BACKGROUND

Case-based learning conferences are valuable to trainees, but growing clinical demands hinder consistent attendance. Social media increasingly acts as a venue for trainees to supplement their education asynchronously. We designed and implemented a web-based asynchronous clinical case discussion series on the Twitter social media platform to fill this educational gap.

OBJECTIVE

The aim of this mixed methods study is to examine the nature of interactions among web-based case discussion participants and assess local attitudes regarding the educational intervention.

METHODS

Starting in February 2018, we posted clinical vignettes to a dedicated Twitter account with the prompt "What else do you want to know?" to stimulate discussion. The authors replied in real time when case discussion participants requested additional details. Additional data about the case were posted at regular intervals to the discussion thread to advance the overall case discussion. Participants were asked to explain their reasoning and support their conclusions when appropriate. Web-based engagement was assessed using Twitter Analytics. Participants' posts were qualitatively analyzed for themes, with special attention to examples of using clinical reasoning skills. A codebook of types of participant posts and interactions was refined iteratively. Local engagement and attitudes at our institution were assessed by surveying internal medicine trainees (n=182) and faculty (n=165) after 6 months.

RESULTS

Over a 6-month period, 11 live case discussions were engaged with by users 1773 times. A total of 86 Twitter profiles spanning 22 US states and 6 countries contributed to discussions among participants and the authors. Participants from all training levels were present, ranging from students to faculty. Interactions among participants and the case moderators were most commonly driven by clinical reasoning, including hypothesis-driven information gathering, discussing the differential diagnosis, and data interpretation or organization. Of 71 respondents to the local survey, 29 (41%) reported having a Twitter account. Of the 29 respondents with Twitter accounts, 17 (59%) reported participating in the case discussions. Respondents agreed that case participation increased both their clinical reasoning skills (15/17, 88%) and clinical knowledge (13/17, 76%).

CONCLUSIONS

A social media-based serialized case discussion was a feasible asynchronous teaching method for engaging web-based learners of all levels in a clinical reasoning discussion. Further study should examine what factors drive trainee participation in web-based case discussions and under what circumstances asynchronous discussion might be preferred over in-person teaching activities.

Evolution in Congenital Cardiology Education: The Rise of Digital-Learning Tools

Technology-enhanced learning is now an established part of medical education due to its ready availability and on-demand nature. This offers new opportunities but also challenges to both learners and teachers. This review outlines the current use of social media tools and online resources in medical education with a particular emphasis on congenital cardiology. It provides strategies to the reader on how to optimize learning in the digital environment and offers guidance on how such tools can be incorporated into routine educational practice. Suggestions for developing and assessing educational material online are discussed. Lastly, the concepts of digital professionalism and digital scholarship are explored to aid medical teachers and educators employ these technologies effectively into their teaching and career development.

Evaluation of Open Access Websites for Anesthesia Education

BACKGROUND

While the prevalence of free, open access medical education resources for health professionals has expanded over the past 10 years, many educational resources for health care professionals are not publicly available or require fees for access. This lack of open access creates global inequities in the availability and sharing of information and may have the most significant impact on health care providers with the greatest need. The extent of open access online educational websites aimed for clinicians and trainees in anesthesiology worldwide is unknown. In this study, we aimed to identify and evaluate the quality of websites designed to provide open access educational resources for anesthesia trainees and clinicians.

METHODS

A PubMed search of articles published between 2009 and 2020, and a Startpage search engine web search was conducted in May 2021 to identify websites using the following inclusion criteria: (1) contain educational content relevant for anesthesia providers or trainees, (2) offer content free of charge, and (3) are written in the English language. Websites were each scored by 2 independent reviewers using a website quality evaluation tool with previous validity evidence that was modified for anesthesia (the Anesthesia Medical Education Website Quality Evaluation Tool).

RESULTS

Seventy-five articles and 175 websites were identified; 37 websites met inclusion criteria. The most common types of educational content contained in the websites included videos (66%, 25/37), text-based resources (51%, 19/37), podcasts (35%, 13/37), and interactive learning resources (32%, 12/37). Few websites described an editorial review process (24%, 9/37) or included opportunities for active engagement or interaction by learners (30%,11/37). Scores by tertile differed significantly across multiple domains, including disclosure of author/webmaster/website institution; description of an editorial review process; relevancy to residents, fellows, and faculty; comprehensiveness; accuracy; disclosure of content creation or revision; ease of access to information; interactivity; clear and professional presentation of information; and links to external information.

CONCLUSIONS

We found 37 open access websites for anesthesia education available on the Internet. Many of these websites may serve as a valuable resource for anesthesia clinicians looking for self-directed learning resources and for educators seeking to curate resources into thoughtfully integrated learning experiences. Ongoing efforts are needed to expand the number and improve the existing open access websites, especially with interactivity, to support the education and training of anesthesia providers in even the most resource-limited areas of the world. Our findings may provide recommendations for those educators and organizations seeking to fill this needed gap to create new high-quality educational websites.

Medical Education Blog and Podcast Utilization During the COVID-19 Pandemic

Introduction The coronavirus disease 2019 (COVID-19) pandemic disrupted traditional in-person learning models. Free Open Access Medical (FOAM) education resources naturally filled this void, so we evaluated how medical blog and podcast utilization changed during the early months of the pandemic. Methods Academic medical podcast and blog producers were surveyed on blog and podcast utilization immediately before (January-March 2020) and after (April-May 2020) the COVID-19 pandemic declaration and subsequent lockdown. Utilization is quantified in terms of blog post pageviews and podcast downloads. Linear regression was used to estimate the effect of publication during the COVID-19 period on 30-day downloads or pageviews. A linear mixed model was developed to confirm this relationship after adjustment for independent predictors of higher 30-day downloads or pageviews, using the podcast or blog as a random intercept. Results Compared to the pre-pandemic period, downloads and pageviews per unique blog and podcast publication significantly increased for blogs (median 30-day pageviews 802 to 1860, p<0.0001) but not for podcasts (median 30-day downloads 2726 to 1781, p=0.27). Publications that contained COVID-19 content were strongly associated with higher monthly utilization (β=7.21, 95% CI 6.29-8.14 p<0.001), and even non-COVID-19 material had higher utilization in the early pandemic (median 30-day downloads/pageviews 868 to 1380, p<0.0001). Discussion The increased blog pageviews during the early months of the COVID-19 pandemic demonstrated the important role of blogs in rapid knowledge translation. Podcasts did not experience a similar increase in utilization.

Going viral: A scoping review of the current state and impact of online research dissemination in emergency medicine

BACKGROUND

The use of free open-access medical education (FOAM) and other online knowledge dissemination methods has increased over the past decade. However, the role and impact of these tools in the knowledge translation continuum are poorly understood, potentially limiting the ability of knowledge generators to fully harness and exploit their potential. Here, we aim to comprehensively map and synthesize the literature describing the use of online tools for the dissemination of emergency medicine research.

METHODS

Using scoping review methodology, we searched the traditional literature via PubMed, CINAHL, EMBASE, ERIC, SCOPUS, and the gray literature for publications exploring online methods to disseminate new research findings. We synthesized the results and constructed a conceptual model of current research dissemination methods.

RESULTS

We included 79 out of 655 unique abstracts and articles identified in our search, 62 of which were from the traditional literature. We describe six primary domains: integration with traditional literature, measurement of dissemination, online organizations and communities of practice, professional development, quality assurance tools and techniques, and advantages and disadvantages of FOAM. For each domain we present an exemplar article and prevailing gaps in knowledge. Finally, we propose a current conceptual framework for dissemination of new research findings that describes both traditional and novel methods of dissemination.

CONCLUSIONS

This comprehensive review of the literature and current dissemination framework will empower researchers, research networks, and granting organizations to maximize their use of FOAM and other online methods to disseminate new knowledge as well as provide clinicians a better understanding of the tools and methods by which to access and implement new research findings.

Frequency of Social Media and Digital Scholarship Keywords in U.S. Medical Schools' Promotion and Tenure Guidelines

PURPOSE

Medical schools must have clear policies and procedures for promotion and tenure (P&T) of faculty. Social media and digital scholarship (SMDS) is an emerging form of scholarship capable of reaching audiences quickly, conveniently, and in a wide variety of formats. It is unclear how frequently SMDS is considered during P&T reviews. The authors sought to determine whether current P&T guidelines at medical schools consider SMDS.

METHOD

The authors acquired P&T guidelines from any U.S. Liaison Committee on Medical Education-accredited medical school (or their governing university) that were available online between October and December 2020. Using an iterative process, they developed a bank of keywords that were specific to SMDS or that could include SMDS between October and December 2020. The authors searched each school's guidelines for each keyword and determined whether the word was being used in relation to crediting faculty for SMDS in the context of P&T procedures. The primary outcome measure was the dichotomous presence or absence of SMDS-specific keywords in each school's P&T guidelines.

RESULTS

The authors acquired P&T guidelines from 145/154 (94%) medical schools. After removing duplicate documents, the authors considered 139 guidelines. The keyword bank included 59 terms, of which 49 were specific to SMDS and 10 were umbrella terms that could be inclusive of SMDS. Of the 139 guidelines, 121 (87%) contained at least 1 SMDS-specific keyword. Schools had a median of 3 SMDS-specific keywords in their P&T guidelines.

CONCLUSIONS

As the presence and impact of SMDS increase, schools should provide guidance on its role in the P&T process. Faculty should receive clear guidance on how to document quality SMDS for their promotion file.

The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources

BACKGROUND

Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations.

OBJECTIVES

We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool.

METHODS

As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability.

RESULTS

Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95).

CONCLUSIONS

We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.

Holes in the FOAM: An Analysis of Curricular Comprehensiveness in Online Educational Resources

OBJECTIVES

We sought to evaluate Free Open Access Medical Education (FOAM), defined as online educational content available free to anyone, anywhere, at any time, by classifying the most impactful FOAM content per the Social Media Index into the topics and subtopics of the American Board of Emergency Medicine's Model of the Clinical Practice of Emergency Medicine. We then analyzed FOAM's comprehensiveness by describing over- and underrepresentation among these topics and subtopics.

METHODS

First, we searched for FOAM resources based on the most recent 12 months of relevant content for each organ system from the top 50 Social Media Index sites. Next, we classified all 898 posts into its related topics or subtopics per the American Board of Emergency Medicine's Model of the Clinical Practice of Emergency Medicine. Finally, we analyzed how comprehensively FOAM covered each organ system and the frequency of posts that covered each organ system subtopic as well as identified the subtopics with the most frequent coverage.

RESULTS

The search yielded 898 FOAM posts, of which cardiology and neurology were significantly overrepresented and psychobehavioral; obstetrics and gynecology; and head, ears, eyes, nose, and throat were significantly underrepresented. Among subtopics, acute coronary syndrome had the highest subtopic coverage consisting of 55.5% of all cardiology content. Other highly represented subtopics include renal colic; diabetic ketoacidosis; sepsis; and stroke with 39, 40, 40, and 71% of each of their topic's content, respectively.

CONCLUSIONS

Although residents and programs are frequently incorporating FOAM into the educational curriculum, these materials seem to lack comprehensiveness. Educators and learners must be aware of these deficits in creating comprehensive emergency medicine curricula.

Quality Appraisal and Assurance Techniques for Free Open Access Medical Education (FOAM) Resources: A Rapid Review

Free open access medical education (FOAM) has disrupted traditional modes of knowledge translation and dissemination. These are popular resources with a wide educational reach. Nephrology has been a leader in FOAM, but many skeptics still question the accuracy and reliability of this content. Recently, quality-assurance techniques have been developed to address these concerns. These techniques may be helpful for readers to appraise the online literature and for institutions to reward the production of high-quality open educational resources. We performed a rapid review of the literature. A medical librarian conducted a systematic search of the Medline and Cumulative Index of Nursing and Allied Health Literature databases. Two independent assessors screened and selected articles, performed a hand-search of reference lists, and scored articles on their quality using the Medical Education Research Study Quality Instrument. Thirteen reports were included for the final descriptive analysis. We identified 10 quality-assessment techniques, and 4 of them having been validated. The quality of the reports was fairly high, with an average Medical Education Research Study Quality Instrument score of 11.5 of 18 (SD, 2.3; range, 7.25-14.25). The calculated Cronbach α was 0.85. There is burgeoning literature on the topic of critical appraisal of open educational resources, and, more specifically, FOAM resources. Many of the techniques used are of varying quality and developed with different intended uses and audiences. By continuing to refine these tools, we can continue not only to support and legitimize the FOAM movement, but also foster individual critical appraisal skills that increasingly are necessary in this age of information.

Content Analysis and Characterization of Medical Tweets During the Early Covid-19 Pandemic

Objective

The novel coronavirus disease 2019 (Covid-19) has infected millions worldwide and impacted the lives of many folds more. Many clinicians share new Covid-19-related resources, research, and ideas within the online Free Open Access to Medical Education (FOAM) community of practice. This study provides a detailed content and contributor analysis of Covid-19-related tweets among the FOAM community during the first months of the pandemic.

Design, Setting, and Participants

In this social media content analysis study, Twitter was searched from November 1, 2019, to March 21, 2020, for English tweets discussing Covid-19 in the FOAM community. Tweets were classified into one of 13 pre-specified content categories: original research, editorials, FOAM resource, public health, podcast or video, learned experience, refuting false information, policy discussion, emotional impact, blatantly false information, other Covid-19, and non-Covid-19. Further analysis of linked original research and FOAM resources was performed. One-thousand (1000) randomly selected contributor profiles and those deemed to have contributed false information were analyzed.

Results

The search yielded 8541 original tweets from 4104 contributors. The number of tweets in each content category were: 1557 other Covid-19 (18.2%), 1190 emotional impact (13.9%), 1122 FOAM resources (13.1%), 1111 policy discussion (13.0%), 928 advice (10.9%), 873 learned experience (10.2%), 424 non-Covid-19 (5.0%), 410 podcast or video (4.8%), 304 editorials (3.6%), 275 original research (3.2%), 245 public health (2.9%), 83 refuting false information (1.0%), and 19 blatantly false (0.2%).

Conclusions

Early in the Covid-19 pandemic, the FOAM community used Twitter to share Covid-19 learned experiences, online resources, crowd-sourced advice, and research and to discuss the emotional impact of Covid-19. Twitter also provided a forum for post-publication peer review of new research. Sharing blatantly false information within this community was infrequent. This study highlights several potential benefits from engaging with the FOAM community on Twitter.

Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study

INTRODUCTION

Podcasts are increasingly being used for medical education. Studies have found that the assessment of the quality of online resources can be challenging. We sought to determine the reliability of gestalt quality assessment of education podcasts in emergency medicine.

METHODS

An international, interprofessional sample of raters was recruited through social media, direct contact, and the extended personal network of the study team. Each participant listened to eight podcasts (selected to include a variety of accents, number of speakers, and topics) and rated the quality of that podcast on a seven-point Likert scale. Phi coefficients were calculated within each group and overall. Decision studies were conducted using a phi of 0.8.

RESULTS

A total of 240 collaborators completed all eight surveys and were included in the analysis. Attendings, medical students, and physician assistants had the lowest individual-level variance and thus the lowest number of required raters to reliably evaluate quality (phi >0.80). Overall, 20 raters were required to reliably evaluate the quality of emergency medicine podcasts.

DISCUSSION

Gestalt ratings of quality from approximately 20 health professionals are required to reliably assess the quality of a podcast. This finding should inform future work focused on developing and validating tools to support the evaluation of quality in these resources.

Consensus Guidelines for Digital Scholarship in Academic Promotion

Introduction

As scholarship moves into the digital sphere, applicant and promotion and tenure (P&T) committee members lack formal guidance on evaluating the impact of digital scholarly work. The P&T process requires the appraisal of individual scholarly impact in comparison to scholars across institutions and disciplines. As dissemination methods evolve in the digital era, we must adapt traditional P&T processes to include emerging forms of digital scholarship.

Methods

We conducted a blended, expert consensus procedure using a nominal group process to create a consensus document at the Council of Emergency Medicine Residency Directors Academic Assembly on April 1, 2019.

Results

We discussed consensus guidelines for evaluation and promotion of digital scholarship with the intent to develop specific, evidence-supported recommendations to P&T committees and applicants. These recommendations included the following: demonstrate scholarship criteria; provide external evidence of impact; and include digital peer-review roles. As traditional scholarship continues to evolve within the digital realm, academic medicine should adapt how that scholarship is evaluated. P&T committees in academic medicine are at the epicenter for supporting this changing paradigm in scholarship.

Conclusion

P&T committees can critically appraise the quality and impact of digital scholarship using specific, validated tools. Applicants for appointment and promotion should highlight and prepare their digital scholarship to specifically address quality, impact, breadth, and relevance. It is our goal to provide specific, timely guidance for both stakeholders to recognize the value of digital scholarship in advancing our field.

Changes to the ACGME Common Program Requirements and Their Potential Impact on Emergency Medicine Core Faculty Protected Time

The Accreditation Council for Graduate Medical Education (ACGME), which regulates residency and fellowship training in the United States, recently revised the minimum standards for all training programs. These standards are codified and published as the Common Program Requirements. Recent specific revisions, particularly removing the requirement ensuring protected time for core faculty, are poised to have a substantial impact on emergency medicine training programs. A group of representatives and relevant stakeholders from national emergency medicine (EM) organizations was convened to assess the potential effects of these changes on core faculty and the training of emergency physicians. We reviewed the literature and results of surveys conducted by EM organizations to examine the role of core faculty protected time. Faculty nonclinical activities contribute greatly to the academic missions of EM training programs. Protected time and reduced clinical hours allow core faculty to engage in education and research, which are two of the three core pillars of academic EM. Loss of core faculty protected time is expected to have detrimental impacts on training programs and on EM generally. We provide consensus recommendations regarding EM core faculty clinical work hour limitations to maintain protected time for educational activities and scholarship and preserve the quality of academic EM.

A Short History of Free Open Access Medical Education. The Past, Present, and Future

Free Open Access Medical education (FOAM) has taken the emergency medicine and critical care worlds by storm in the past decade. This article represents one perspective on the stages of transition for FOAM from its humble beginnings as a grassroots movement to the more recent multiauthor blogs that are described in the peer-reviewed literature. In this article, the authors describe the following four distinct waves of people within the movement, with each wave creating a new stage in the evolution of the FOAM community: Creation by the Founders, Adoption by the Enthusiasts, Structure and Formalization by the Structuralists, and Engagement and Activity by the End Users. The authors contextualize some of the phenomena that have been observed within this field and highlight challenges for the field moving forward.

The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance

Construct: This study seeks to determine validity evidence for the Quality of Assessment for Learning score (QuAL score), which was created to evaluate short qualitative comments that are related to specific scores entered into a workplace-based assessment, common within the competency-based medical education (CBME) context. Background: In the age of CBME, qualitative comments play an important role in clarifying the quantitative scores rendered by observers at the bedside. Currently there are few practical tools that evaluate mixed data (e.g. associated score-and-comment data), other than the comprehensive Completed Clinical Evaluation Report Rating tool (CCERR) that was originally derived to rate end-of-rotation reports. Approach: A multi-center, randomized cohort-based rating exercise was conducted to evaluate the rating properties of the QuAL score as compared to the CCERR. One group rated comments using the QuAL score, and the other group rated comments using the CCERR. A generalizability study (G-Study) and a decision study (D-study) were conducted to determine the number of meta-raters for a reliable rating (phi-coefficient target of >0.80). Both scores were correlated against rater's gestalt perceptions of utility for both faculty and residents reading the scores. Results: Twenty-five meta-raters from 20 sites participated in this rating exercise. The G-study revealed that the CCERR group (n = 13) rated the comments with a very high reliability (Phi = 0.97). Meanwhile, the QuAL group (n = 12) rated the comments with a similarly high reliability (Phi = 0.97). The QuAL score required only two raters to reach an acceptable target reliability of >0.80, while the CCERR required three. The QuAL score correlated with perceptions of utility (Meta-rater usefulness, Pearson's r = 0.69, p < 0.001; Perceived usefulness for trainee, r = 0.74, p < 0.001). The CCERR performed similarly, correlating with perceived faculty (r = 0.67, <0.001) and resident utility (0.79, <0.001). Conclusions: The QuAL score is reliable rating score that correlates well with perceptions of utility. The QuAL score may be useful for rating shorter comments generated by workplace-based assessments.

Academic Life in Emergency Medicine (ALiEM) Blog and Podcast Watch: Infectious Diseases

The Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) Series and Approved Instructional Resources - Professional (AIR-Pro) Series were created in 2014 and 2015, respectively, in response to the growing need to curate online educational content as well as create a nationally available curriculum that meets individualized interactive instruction criteria for emergency medicine (EM) trainees. These two online series identify high-quality educational blog and podcast content using an expert-based approach.

The AIR series is a continuously building curriculum originally based on the Council of Emergency Medicine Directors (CORD) testing schedule. In September 2019, 61 blog posts and podcasts published within the previous 12 months and relevant to infectious diseases were evaluated by eight attending physicians using the ALiEM AIR scoring instrument. In this review, we summarize the accredited posts on infectious diseases meeting our a priori quality criteria per evaluation by the reviewers.

Mapping the Expert Mind: Integration Method for Revising the ACES Medical Simulation Curriculum

PURPOSE

This article shares our experience developing an integrated curriculum for the ACES (Acute Critical Event Simulation) program. The purpose of the ACES program is to ensure that health care providers develop proficiency in the early management of critically ill patients. The program includes multiple different types of educational interventions (mostly simulation-based) and targets both specialty and family physicians practicing in tertiary and community hospitals.

METHODS

To facilitate integration between different educational interventions, we developed a knowledge repository consisting of cognitive sequence maps that make explicit the flow of cognitive activities carried out by experts facing different situations - the sequence maps then serving as the foundation upon which multimodal simulation scenarios would be built. To encourage participation of experts, we produced this repository as a peer-reviewed ebook. Five national organizations collaborated with the Royal College of Physicians and Surgeons of Canada to identify and recruit expert authors and reviewers. Foundational chapters, centered on goals/interventions, were first developed to comprehensively address most tasks conducted in the early management of a critically ill patient. Tasks from the foundational chapters were then used to complete the curriculum with situations. The curriculum development consisted of two-phases each followed by a peer-review process. In the first phase, focus groups using web-conferencing were conducted to map clinical practice approaches and in the second, authors completed the body of the chapter (e.g., introduction, definition, concepts, etc.) then provided a more detailed description of each task linked to supporting evidence.

RESULTS

Sixty-seven authors and thirty-five peer reviewers from various backgrounds (physicians, pharmacists, nurses, respiratory therapists) were recruited. On average, there were 32 tasks and 15 situations per chapter. The average number of focus group meetings needed to develop a map (one map per chapter) was 6.7 (SD ± 3.6). We found that the method greatly facilitated integration between different chapters especially for situations which are not limited to a single goal or intervention. For example, almost half of the tasks of the Hypercapnic Ventilatory Failure chapter map were borrowed from other maps with some modifications, which significantly reduced the authors' workload and enhanced content integration. This chapter was also linked to 6 other chapters.

CONCLUSIONS

To facilitate curriculum integration, we have developed a knowledge repository consisting of cognitive maps which organize time-sensitive tasks in the proper sequence; the repository serving as the foundation upon which other educational interventions are then built. While this methodology is demanding, authors welcomed the challenge given the scholarly value of their work, thus creating an interprofessional network of educators across Canada.

Systematic Online Academic Resource (SOAR) Review: Renal and Genitourinary

BACKGROUND

Online resources for emergency medicine (EM) trainees and physicians have variable quality and inconsistent coverage of core topics. In this first entry of the Society for Academic Emergency Medicine Systematic Online Academic Resource (SOAR) series, we describe the application of a systematic methodology to comprehensively identify, collate, and curate online content for topic-specific modules.

METHODS

A list of module topics and related terms was generated from the American Board of Emergency Medicine's Model of the Clinical Practice of Emergency Medicine. The authors selected "renal and genitourinary" for the first module, which contained 35 terms; all MeSH headers and colloquial synonyms related to the topic and related terms were searched both within the 100 most impactful online educational websites per the Social Media Index and the FOAMsearch.net search engine. Duplicate entries, journal articles, images, and archives were excluded. The quality of each article was rated using the revised METRIQ (rMETRIQ) score.

RESULTS

The search yielded 13,058 online resources. After 12,717 items were excluded, 341 underwent quality assessment. All renal/genitourinary topics were covered by at least one resource. The median rMETRIQ score was 11 of 21 (interquartile range = 8-14). Calculus of urinary tract was most prominently featured with 60 posts. Thirty-four posts (10% of full-text screened FOAM articles) covering 12 core topics were identified as high quality (rMETRIQ ≥ 16).

CONCLUSIONS

We demonstrated the feasibility of systematically identifying and curating FOAM resources for a specific EM topic and identified an overrepresentation of some subtopics. This curated list of resources may guide trainees, teacher recommendations, and resource producers. Further entries in the series will address other topics relevant to EM.

The Revised METRIQ Score: A Quality Evaluation Tool for Online Educational Resources

BACKGROUND

With the rapid proliferation of online medical education resources, quality evaluation is increasingly critical. The Medical Education Translational Resources: Impact and Quality (METRIQ) study evaluated the METRIQ-8 quality assessment instrument for blogs and collected feedback to improve it.

METHODS

As part of the larger METRIQ study, participants rated the quality of five blog posts on clinical emergency medicine topics using the eight-item METRIQ-8 score. Next, participants used a 7-point Likert scale and free-text comments to evaluate the METRIQ-8 score on ease of use, clarity of items, and likelihood of recommending it to others. Descriptive statistics were calculated and comments were thematically analyzed to guide the development of a revised METRIQ (rMETRIQ) score.

RESULTS

A total of 309 emergency medicine attendings, residents, and medical students completed the survey. The majority of participants felt the METRIQ-8 score was easy to use (mean ± SD = 2.7 ± 1.1 out of 7, with 1 indicating strong agreement) and would recommend it to others (2.7 ± 1.3 out of 7, with 1 indicating strong agreement). The thematic analysis suggested clarifying ambiguous questions, shortening the 7-point scale, specifying scoring anchors for the questions, eliminating the "unsure" option, and grouping-related questions. This analysis guided changes that resulted in the rMETRIQ score.

CONCLUSION

Feedback on the METRIQ-8 score contributed to the development of the rMETRIQ score, which has improved clarity and usability. Further validity evidence on the rMETRIQ score is required.

Academic Life in Emergency Medicine (ALiEM) Blog and Podcast Watch: Gastrointestinal Emergencies

The Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) Series and Approved Instructional Resources - Professional (AIR-Pro) Series were created in 2014 and 2015, respectively, in response to the growing need to curate online educational content as well as create a nationally available curriculum that meets individualized interactive instruction criteria for emergency medicine (EM) trainees. These two online series identify high-quality educational blog and podcast content using an expert-based approach. We summarize the accredited posts on gastrointestinal emergencies that met our a priori determined quality criteria per evaluation by eight experienced faculty educators in EM.

Critical Appraisal of Emergency Medicine Educational Research: The Best Publications of 2016

OBJECTIVES

The objectives were to critically appraise the emergency medicine (EM) medical education literature published in 2016 and review the highest-quality quantitative and qualitative studies.

METHODS

A search of the English language literature in 2016 querying MEDLINE, Scopus, Education Resources Information Center (ERIC), and PsychInfo identified 510 papers related to medical education in EM. Two reviewers independently screened all of the publications using previously established exclusion criteria. The 25 top-scoring quantitative studies based on methodology and all six qualitative studies were scored by all reviewers using selected scoring criteria that have been adapted from previous installments. The top-scoring articles were highlighted and trends in medical education research were described.

RESULTS

Seventy-five manuscripts met inclusion criteria and were scored. Eleven quantitative and one qualitative papers were the highest scoring and are summarized in this article.

CONCLUSION

This annual critical appraisal series highlights the best EM education research articles published in 2016.

Academic Life in Emergency Medicine (ALiEM) Blog and Podcast Watch: Renal and Genitourinary Emergencies

The Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) series and Approved Instruction Resources Professional (AIR-Pro) series were created in 2014 and 2015, respectively, to address the need for the curation of online educational content as well as a nationally available curriculum that meets individualized interactive instruction criteria. These two programs identify high-quality educational blog and podcast content using an expert-based approach. The AIR series is a continuously building curriculum originally based on the Council of Emergency Medicine Directors (CORD) testing schedule. Using the ALiEM AIR scoring instrument, 49 blog posts and podcasts relevant to renal and genitourinary emergencies published within the previous 12 months were evaluated by eight attending physicians. We summarize the 13 posts that met our a priori determined quality criteria per evaluation by the reviewers. The ALiEM Blog and Podcast Watch series identifies high-quality educational blogs and podcasts for emergency medicine clinicians through its expert panel, using a validated scoring instrument. While this article focuses on renal and genitourinary emergencies, additional AIR modules address other topics in emergency medicine. The AIR and AIR-Pro series provide post-publication accreditation and curation of recent online content to identify and recommend high-quality educational social media content for the EM clinician.

A Review of Natural Language Processing in Medical Education

Natural language processing (NLP) aims to program machines to interpret human language as humans do. It could quantify aspects of medical education that were previously amenable only to qualitative methods. The application of NLP to medical education has been accelerating over the past several years. This article has three aims. First, we introduce the reader to NLP. Second, we discuss the potential of NLP to help integrate FOAM (Free Open Access Medical Education) resources with more traditional curricular elements. Finally, we present the results of a systematic review. We identified 30 articles indexed by PubMed as relating to medical education and NLP, 14 of which were of sufficient quality to include in this review. We close by discussing potential future work using NLP to advance the field of medical education in emergency medicine.

Academic Life in Emergency Medicine Blog and Podcast Watch: Toxicologic Emergencies

The Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) Series was created in 2014 to address a lack of both curation of online educational content and a nationally available curriculum that meets individualized interactive instruction. Using an expert-based, crowdsourced approach, the AIR series identifies trustworthy, high-quality, educational blog and podcast content. Here, we summarize the content rated as high quality per our a priori criteria as evaluated by eight attending physicians.

Effective Educational Videos in Emergency Medicine

Video is a popular format for teaching and learning online. Emergency medicine (EM) has been a leader in online learning and EM educators may wish to use video to teach. The creation of online video content is fraught with pitfalls that may make videos less effective. We review notable theory and evidence regarding effective use of video for education in EM with international considerations.

The Social Media Index as an Indicator of Quality for Emergency Medicine Blogs: A METRIQ Study

STUDY OBJECTIVE

Online educational resources such as blogs are increasingly used for education by emergency medicine clinicians. The Social Media Index was developed to quantify their relative impact. The Medical Education Translational Resources: Indicators of Quality (METRIQ) study was conducted in part to determine the association between the Social Media Index score and quality as measured by gestalt and previously derived quality instruments.

METHODS

Ten blogs were randomly selected from a list of emergency medicine and critical care Web sites. The 2 most recent clinically oriented blog posts published on these blogs were evaluated with gestalt, the Academic Life in Emergency Medicine Approved Instructional Resources (ALiEM AIR) score, and the METRIQ-8 score. Volunteer raters (including medical students, emergency medicine residents, and emergency medicine attending physicians) were identified with a multimodal recruitment methodology. The Social Media Index was calculated in February 2016, November 2016, April 2017, and December 2017. Pearson's correlations were calculated between the Social Media Index and the average rater gestalt, ALiEM AIR score, and METRIQ-8 score.

RESULTS

A total of 309 of 330 raters completed all ratings (93.6%). The Social Media Index correlated moderately to strongly with the mean rater gestalt ratings (range 0.69 to 0.76) and moderately with the mean rater ALiEM AIR score (range 0.55 to 0.61) and METRIQ-8 score (range 0.53 to 0.57) during the month of the blog post's selection and for 2 years after.

CONCLUSION

The Social Media Index's correlation with multiple quality evaluation instruments over time supports the hypothesis that it is associated with overall Web site quality. It can play a role in guiding individuals to high-quality resources that can be reviewed with critical appraisal techniques.

Quality Evaluation Scores are no more Reliable than Gestalt in Evaluating the Quality of Emergency Medicine Blogs: A METRIQ Study

Construct: We investigated the quality of emergency medicine (EM) blogs as educational resources.

PURPOSE

Online medical education resources such as blogs are increasingly used by EM trainees and clinicians. However, quality evaluations of these resources using gestalt are unreliable. We investigated the reliability of two previously derived quality evaluation instruments for blogs.

APPROACH

Sixty English-language EM websites that published clinically oriented blog posts between January 1 and February 24, 2016, were identified. A random number generator selected 10 websites, and the 2 most recent clinically oriented blog posts from each site were evaluated using gestalt, the Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) score, and the Medical Education Translational Resources: Impact and Quality (METRIQ-8) score, by a sample of medical students, EM residents, and EM attendings. Each rater evaluated all 20 blog posts with gestalt and 15 of the 20 blog posts with the ALiEM AIR and METRIQ-8 scores. Pearson's correlations were calculated between the average scores for each metric. Single-measure intraclass correlation coefficients (ICCs) evaluated the reliability of each instrument.

RESULTS

Our study included 121 medical students, 88 EM residents, and 100 EM attendings who completed ratings. The average gestalt rating of each blog post correlated strongly with the average scores for ALiEM AIR (r = .94) and METRIQ-8 (r = .91). Single-measure ICCs were fair for gestalt (0.37, IQR 0.25-0.56), ALiEM AIR (0.41, IQR 0.29-0.60) and METRIQ-8 (0.40, IQR 0.28-0.59).

CONCLUSION

The average scores of each blog post correlated strongly with gestalt ratings. However, neither ALiEM AIR nor METRIQ-8 showed higher reliability than gestalt. Improved reliability may be possible through rater training and instrument refinement.

Editorial Processes in Free Open Access Medical Educational (FOAM) Resources

BACKGROUND

Much of the skepticism toward online educational resources (OERs) in emergency medicine (EM) stems from the low barrier to publishing and a perceived lack of editorial rigor. Learners and educators have demonstrated unreliable gestalt ratings of OERs, suggesting a lack of capacity to consistently appraise these resources. The development of tools to guide clinicians and learners in the selection and use of blogs and podcasts is a growing area of interest. Disclosure of editorial process was identified in previous studies as an important quality indicator for OERs. However, little is known about editorial process in online EM resources and whether it can be reliably integrated into a critical appraisal tool.

METHODS

Two reviewers assessed 100 top EM and critical care OERs for mention and description of editorial process and academic and nonacademic affiliations. Ninety-two sites were accessible for review. All sites were also contacted to attempt clarification of their editorial process. Inter-rater reliability for mention and description of editorial process was evaluated using Cohen's kappa, and the relationship between academic affiliation and disclosure of editorial process was assessed by odds ratio (OR).

RESULTS

Eleven sites mentioned an editorial process, and 10 of these sites included a description. Five of the seven sites that responded to contact also described an editorial process. Inter-rater agreement was excellent for mention (κ = 0.90) and description (κ = 1.00) of editorial process. Eighteen sites listed academic affiliations and 21 sites had nonacademic affiliations. A greater proportion of sites with academic affiliations disclosed their editorial process compared to sites without academic affiliations (OR = 5.3, 95% confidence interval [CI] = 1.3-21.0; difference in proportions of 0.40, 95% CI = 11.6-60.8).

CONCLUSIONS

Although transparency is lacking, editorial processes exist among OERs. Inter-rater reliability for disclosure of editorial process is excellent, supporting its use within critical appraisal tools.

Academic Life in Emergency Medicine Blog and Podcast Watch: Respiratory Emergencies

The Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) Series and Approved Instruction Resources Professional (AIR-Pro) Series were created in 2014 and 2015, respectively, to address the need for curation of online educational content as well as a nationally available curriculum that meets individualized interactive instruction criteria. These two programs identify high-quality educational blog and podcast content using an expert-based approach.

We summarize the accredited posts on respiratory emergencies that met our a priori determined quality criteria per evaluation by eight experienced faculty educators in emergency medicine.

Dissemination of evidence in paediatric emergency medicine: a quantitative descriptive evaluation of a 16-week social media promotion

OBJECTIVES

TRanslating Emergency Knowledge for Kids (TREKK) and Cochrane Child Health collaborate to develop knowledge products on paediatric emergency medicine topics. Via a targeted social media promotion, we aimed to increase user interaction with the TREKK and Cochrane Child Health Twitter accounts and the uptake of TREKK Bottom Line Recommendations (BLRs) and Cochrane systematic reviews (SRs).

DESIGN

Quantitative descriptive evaluation.

SETTING

We undertook this study and collected data via the internet.

PARTICIPANTS

Our target users included online healthcare providers and health consumers.

INTERVENTION

For 16 weeks, we used Twitter accounts (@TREKKca and @Cochrane_Child) and the Cochrane Child Health blog to promote 6 TREKK BLRs and 16 related Cochrane SRs. We published 1 blog post and 98 image-based tweets per week.

PRIMARY AND SECONDARY OUTCOME MEASURES

The primary outcome was user interaction with @TREKKca and @Cochrane_Child. Secondary outcomes were visits to TREKK's website and the Cochrane Child Health blog, clicks to and views of the TREKK BLRs, and Altmetric scores and downloads of Cochrane SRs.

RESULTS

Followers to @TREKKca and @Cochrane_Child increased by 24% and 15%, respectively. Monthly users of TREKK's website increased by 29%. Clicks to the TREKK BLRs increased by 22%. The BLRs accrued 59% more views compared with the baseline period. The 16 blog posts accrued 28% more views compared with the 8 previous months when no new posts were published. The Altmetric scores for the Cochrane SRs increased by ≥10 points each. The mean number of full text downloads for the promotion period was higher for nine and lower for seven SRs compared with the 16-week average for the previous year (mean difference (SD), +4.0 (22.0%)).

CONCLUSIONS

There was increased traffic to TREKK knowledge products and Cochrane SRs during the social media promotion. Quantitative evidence supports blogging and tweeting as dissemination strategies for evidence-based knowledge products.

EMS providers do not use FOAM for education

BACKGROUND

Free open access to medical education (FOAM, #FOAM) is the free availability of educational materials on various medicine topics. We hope to evaluate the use of social media and FOAM by emergency medical services (EMS) providers.

METHODS

We designed an online survey distributed to EMS providers with questions about demographics and social media/FOAM use by providers. The survey was sent to the American College of Emergency Physicians (ACEP) EMS Listserv of medical directors and was asked to be distributed to their respective agencies. The survey was designed to inquire about the providers' knowledge of FOAM and social media and their use of the above for EMS education.

RESULTS

There were 169 respondents out of a total of 523 providers yielding a response rate of 32.3%. Fifty-three percent of respondents are paramedics, 37% are EMT-Basic trained, and the remainder (16%) were "other." The minority (20%) of respondents had heard of FOAM. However, 54% of respondents had heard of "free medical education online" regarding pertinent topics. Of the total respondents who used social media for education, 31% used Facebook and 23% used blogs and podcasts as resources for online education. Only 4% of respondents stated they produced FOAM content. Seventy-six percent of respondents said they were "interested" or "very interested" in using FOAM for medical education. If FOAM provided continuing medical education (CME), 83% of respondents would be interested in using it.

CONCLUSION

Social media is not used frequently by EMS providers for the purposes of FOAM. There is interest within EMS providers to use FOAM for education, even if CME was not provided. FOAM can provide a novel area of education for EMS.

Scholarship by the Clinician-Educator in Emergency Medicine

Emergency medicine (EM) continues to grow as an academic specialty. Like most specialties, a large number of academic emergency physicians are focused on education of our graduate student learners. For promotion, clinician-educators (CEs) are required to produce scholarly work and disseminate knowledge. Although promotion requirements may vary by institution, scholarly work is a consistent requirement. Due to the clinical constraints of working in the emergency department, the unique interactions emergency physicians have with their learners, and early adoption of alternative teaching methods, EM CEs' scholarly work may not be adequately described in a traditional curriculum vitae. Using a rubric of established domains around the academic work of CEs, this article describes some of the ways EM educators address these domains. The aim of the article is to provide a guide for academic department leadership, CEs, and promotion committees about the unique ways EM has addressed the work of the CE.

Academic Life in Emergency Medicine (ALiEM) Blog and Podcast Watch: Infectious Disease Emergencies

To address the needs for curation of online educational content as well as the development of a nationally available curriculum that meets individualized interactive instruction, the Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) Series and AIR-Pro Series were created in 2014 and 2015, respectively. Using an expert-based, crowd-sourced approach, these two programs identify trustworthy, high-quality, educational blog, and podcast content.

Here, we summarize the accredited posts that met our a priori determined quality criteria and evaluated by eight attending physicians.

Curated Collections for Educators: Five Key Papers on Evaluating Digital Scholarship

Traditionally, scholarship that was recognized for promotion and tenure consisted of clinical research, bench research, and grant funding. Recent trends have allowed for differing approaches to scholarship, including digital publication. As increasing numbers of trainees and faculty turn to online educational resources, it is imperative to critically evaluate these resources. This article summarizes five key papers that address the appraisal of digital scholarship and describes their relevance to junior clinician educators and faculty developers. In May 2017, the Academic Life in Emergency Medicine Faculty Incubator program focused on the topic of digital scholarship, providing and discussing papers relevant to the topic. We augmented this list of papers with further suggestions by guest experts and by an open call via Twitter for other important papers. Through this process, we created a list of 38 papers in total on the topic of evaluating digital scholarship. In order to determine which of these papers best describe how to evaluate digital scholarship, the authorship group assessed the papers using a modified Delphi approach to build consensus. In this paper we present the five most highly rated papers from our process about evaluating digital scholarship. We summarize each paper and discuss its specific relevance to junior faculty members and to faculty developers. These papers provide a framework for assessing the quality of digital scholarship, so that junior faculty can recommend high-quality educational resources to their trainees. These papers help guide educators on how to produce high quality digital scholarship and maximize recognition and credit in respect to receiving promotion and tenure.