The Revised METRIQ Score: A Quality Evaluation Tool for Online Educational Resources

[Digital tools in residency and continuing medical education within the framework of a digital media concept]

There are currently many online resources for medical education during residency and beyond in anesthesiology, intensive care, pain, emergency and palliative medicine. From traditional textbooks and in-person events to learning platforms, apps, podcasts, simulation training and even virtual reality, there are many ways to supplement traditional residency curricula and continuing medical education. The coronavirus disease 2019 (COVID-19) pandemic has been instrumental in making medical education content more accessible and, among other things, accelerate the transfer of knowledge.To include all colleagues in the goal of life-long learning using these modern tools, we recommend the development of a digital media concept that is individually tailored to each department of anesthesiology. First, the goals of the department should be defined, e.g., can existing teaching materials be made more digitally accessible for asynchronous learning? Then, department resources should be compiled, e.g., what learning platforms are already being used and if and how social media should play a role? One or more persons should be named responsible and maintain the new concept. In this context, it is essential to develop quality criteria to properly assess the digital content.With the support of the department, conventional teaching methods can be combined with new digital possibilities in residency education and beyond. In this way, individual shift models, various levels of participation in live teaching events and different types of learners can be taken into account. These diverse digital tools can enrich the training and further education of every team member in an anesthesiology department and will accompany us well into the future.

Learning cardiac embryology on YouTube-What videos are there to view?

In modern medical curricula, embryology is typically taught through lectures, with a few institutions providing tutorials. The use of 3-D videos or animations enables students to study these embryological structures and how they change with time. The aim of this study was to assess the quality of cardiac embryology videos available on YouTube. A systematic literature review regarding the use of YouTube in teaching or learning cardiac embryology identified no papers that examined this specific question, and next, a systematic search of YouTube was performed. A total of 1200 cardiac embryology videos were retrieved using 12 specific search terms, with 370 videos retrieved under two or more search terms and excluded. A further 511 videos were excluded under additional, specific criteria. The remaining 319 videos were evaluated with the YouTube Video Assessment Criteria (UTvAC), with 121 rated as "useful." Videos on YouTube are uploaded with a wide audience in mind, from children to cardiologists, and content control is imperfect. Multiple videos were identified as duplicates of videos from original channels, typically without attribution. While 49 videos showed operations or human material, none contained an ethical statement regarding consent, and only 10 of these included an age restriction or graphical advisory. While there are useful videos for medical students studying cardiac embryology on YouTube, intuitive search strategies will also identify many with irrelevant content and of variable quality. Digital competence and search strategies are not innate skills, so educators should teach students to assess information so as to avoid overload or "filter failure."

The ID Digital Institute: Building a digital education toolset and community

Organic digital education (ODE) includes online medical education content that can take various forms, such as blogs, social media, videos, podcasts, or infographics. Multimedia ODE platforms have unique benefits and have quickly become an essential part of medical education. Modern medical educators with competency in digital teaching modalities can leverage these for teaching as well as career development and dissemination of scientific research. The ID Digital Institute is a digital education program with a curriculum designed to equip infectious diseases (ID) professionals with the skills to appraise, create, curate, and integrate ODE into their teaching and career. We share the structure, content, and lessons learned from the ID Digital Institute program. We also illustrate how digital education skills can present unique opportunities to align with current and future transplant and immunocompromised host infectious diseases education efforts.

Listen to your heart: a critical analysis of popular cardiology podcasts

Purpose

Podcasts are an increasingly popular medium for medical education in the field of cardiology. However, evidence suggests that the quality of the information presented can be variable. The aim of our study was to assess the quality of the most popular cardiology podcasts on existing podcast streaming services, using tools designed to grade online medical education.

Results

We analyzed the five most recent episodes from 28 different popular cardiology podcasts as of 20th of September, 2022 using the validated rMETRIQ and JAMA scoring tools. The median podcast length was 20 min and most episodes were hosted by professors, subspecialty discussants or consultant physicians (87.14%). Although most episodes had only essential content (85%), only a small proportion of episodes provided detailed references (12.9%), explicitly identified conflicts of interest (30.7%), described a review process (13.6%), or provided a robust discussion of the podcast's content (13.6%). We observed no consistent relationship between episode length, seniority of host or seniority of guest speaker with rMETRIQ or JAMA scores.

Conclusions

Cardiology podcasts are a valuable remote learning tool for clinicians. However, the reliability, relevance, and transparency of information provided on cardiology podcasts varies widely. Streamlined standards for evaluation are needed to improve podcast quality.

FOAM authorship: Who's teaching our learners?

Background

Free open-access medical education (FOAM) is extremely popular among learners and educators despite lacking the traditional peer review process. Despite the potential for inaccurate, low-quality, or biased content, little has been published describing FOAM authors.

Methods

We performed a cross-sectional analysis of 12 months of content from the top 25 blogs in the 2020 Social Media Index from August 2020-2021. We recorded the number of posts per site and descriptive characteristics of authors, including gender affiliation, conflicts of interest (COI) statements, and type of practice (academic, community, or hybrid).

Results

We identified 2141 posts by 1001 authors. More than half were produced by six websites: EM Docs (266), Life in the Fast Lane (232), EMCrit (188), ALiEM (185), Don't Forget the Bubbles (181), and Rebel EM (174). Most content (1680 posts, 78.5%) lacked a COI statement. Authors were mostly academic (89%), mostly held MD degrees (67.4%), and were mostly men (59.7%). Geographically, most FOAM authors reside in the United States (59.5%), Canada (22.42%), or the United Kingdom (9.4%).

Conclusions

Of all the posts in the top 25 sites in 2020, more than half came from six sites, and authors were largely North American men in academics with MD degrees. Learners, content creators, and educators should consider the ways in which a more diverse authorship pool might bring value to the FOAM educational experience.

Applying the master adaptive learner framework to just-in-time training of procedures

Background

Just-in-time training (JITT) occurs in the clinical context when learners need immediate guidance for procedures due to a lack of proficiency or the need for knowledge refreshment. The master adaptive learner (MAL) framework presents a comprehensive model of transforming learners into adaptive experts, proficient not only in their current tasks but also in the ongoing development of lifelong skills. With the evolving landscape of procedural competence in emergency medicine (EM), trainees must develop the capacity to acquire and master new techniques consistently. This concept paper will discuss using JITT to support the development of MALs in the emergency department.

Methods

In May 2023, an expert panel from the Society for Academic Emergency Medicine (SAEM) Medical Educator's Boot Camp delivered a comprehensive half-day preconference session entitled "Be the Best Teacher" at the society's annual meeting. A subgroup within this panel focused on applying the MAL framework to JITT. This subgroup collaboratively developed a practical guide that underwent iterative review and refinement.

Results

The MAL-JITT framework integrates the learner's past experiences with the educator's proficiency, allowing the educational experience to address the unique requirements of each case. We outline a structured five-step process for applying JITT, utilizing the lumbar puncture procedure as an example of integrating the MAL stages of planning, learning, assessing, and adjusting. This innovative approach facilitates prompt procedural competence and cultivates a positive learning environment that fosters acquiring adaptable learning skills with enduring benefits throughout the learner's career trajectory.

Conclusions

JITT for procedures holds the potential to cultivate a dynamic learning environment conducive to nurturing the development of MALs in EM.

SAEM systematic online academic resource (SOAR) review: Gastrointestinal illnesses

Background and Objectives

Free open access medical education (FOAM) has become an essential tool for emergency medicine (EM) education and can be valuable to clinicians as a point-of-care resource. The development of the revised Medical Education Translational Resources Impact and Quality (rMETRIQ) tool provides a standardized means of quality assessment. Previous entries of the Society for Academic Emergency Medicine systematic online academic resource (SOAR) series have focused on renal, endocrine, and sickle cell disorders. In this iteration, we strive to identify, curate, and describe FOAM topics specific to acute gastrointestinal (GI) illnesses.

Methods

We searched 389 keywords across 11 GI topics that were modified from the 2019 Model of the Clinical Practice of EM (EM Model) using the search engine Google FOAM and within the top 50 websites listed on Academic Life in Emergency Medicine's Social Media Index. The sites underwent preliminary screening to eliminate resources that were not relevant to EM or GI illnesses. Identified resources were evaluated with the rMETRIQ tool by five board-certified EM physicians who received rMETRIQ tool rater training.

Results

After duplicates of the initial 39,505 resources were eliminated, 8059 remained. Primary screening resulted in a final 1202 resources. The most common categories were large bowel (18%), small bowel (13%), stomach (11%), esophagus (11%), biliary (11%), and liver (10%). Many resources covered multiple topics and subtopics. The final mean intraclass correlation coefficient among the five physicians was 0.95 (95% CI 0.92-0.98) for rMETRIQ scoring. We identified 256 sites considered "high quality" with a rMETRIQ score of 16 or higher as designated in prior reviews.

Conclusions

This iteration of the SOAR review resulted in the highest number of high-quality resources compared to other SOAR reviews, with 21% of resources thus far scoring ≥ 16. A final list of high-quality resources can guide trainees, educator recommendations, and FOAM authors.

Impact of the Febrile Podcast and Learning Resource as an Infectious Diseases Education Platform

Background

Febrile is an infectious diseases (ID) podcast and learning platform with the aim of providing high-quality and accessible ID content for learners. We describe the use of Febrile as a resource for learning and teaching ID as well as learner satisfaction and perceived impact on clinical practice.

Methods

The Febrile platform was launched in December 2020 and includes audio podcast episodes, infographics, and detailed online summaries of adult and pediatric ID topics. Production and contributor information is summarized. Podcast, website, and social media engagement is reported from available quantitative analytics. An online anonymous survey was conducted to assess educational impact.

Results

After 3 years of operation, Febrile has produced 90 episodes and has been downloaded >460 000 times in 196 countries, with the majority of its audience (58.9%) listening from within the United States. A total of 230 participants from 30 countries and 38 US states completed the survey, of whom 79 (34.5%) were ID fellows in training and 78 (34.2%) were ID faculty physicians. Seventy-two percent of survey respondents reported visiting the website, and 82% had seen an infographic. Enhancing core ID knowledge was the primary driver for listening. Two-thirds of respondents indicated that information learned from Febrile has changed their practice, and 50% have used Febrile as a way to teach others. Febrile also led to favorable impressions of ID for those considering ID as a career.

Conclusions

Febrile is an engaging platform for ID medical education and provides a unique resource within the global ID community.

Evaluating a Health Equity Podcast for Provider Practice Change: A Cross-sectional Study

INTRODUCTION

Podcasts are effective tools for disseminating health education. This study aimed to disseminate a health equity curriculum on best practices.

METHOD

A cross-sectional descriptive study was conducted as part of the Health Equity in Pediatrics podcast series between May and June 2023. Listeners were surveyed following each episode, and both quantitative and qualitative responses of voluntary respondents were analyzed.

RESULTS

Episodes were downloaded 4,095 times. Survey respondents (n = 66) reported increased knowledge and intended practice change inspired by the podcast. Qualitative responses included themes surrounding knowledge, appreciation, and practice change.

DISCUSSION

Podcasts are easily disseminated to wide audiences and can improve health equity knowledge while inspiring practice change. This style can help listeners identify practices that suggest implicit bias and implement more equitable best practices. Future research should examine implicit bias training and standardization of health equity education using podcasts.

Online learning in health professions education. Part 2: Tools and practical application: AMEE Guide No. 163

Part 1 of the AMEE Guide Online learning in health professions education focused on foundational concepts such as theory, methods, and instructional design in online learning. Part 2 builds upon Part 1, introducing technology tools and applications of these foundational concepts by exploring the various levels (from beginner to advanced) of utilisation, while describing how their usage can transform Health Professions Education. This Part covers Learning Management Systems, infographics, podcasting, videos, websites, social media, online discussion forums, simulation, virtual patients, extended and virtual reality. Intertwined are other topics, such as online small group teaching, game-based learning, FOAM, online social and collaboration learning, and virtual care teaching. We end by discussing digital scholarship and emerging technologies. Combined with Part 1, the overall aim of Part 2 is to produce a comprehensive overview to help guide effective use online learning in Health Professions Education.

Hunting for Pearls: A Qualitative Analysis of the Reflections of Students Creating Psychiatric Podcasts

Purpose

This paper aims to explore medical student experiences of creating a peer-to-peer psychiatry educational podcast.

Methods

During psychiatry placement, ten year-4 University of Bristol medical students created peer-educational multi-episode podcasts on psychiatric topics. Following completion, they submitted reflective essays on their experiences. Qualitative thematic analysis of these essays was completed by two independent authors. Following data familiarisation, authors independently generated codes that were collated into relevant themes. Upon reaching thematic saturation, findings were collated, and member checking was carried out to confirm the validity of findings.

Results

Themes included effective preparation, choosing content, podcast production, enhancing learning, the weight of responsibility and creating educational support networks. All students found podcast creation to be beneficial for personal learning.

Conclusion

Exploration of students' experiences creating podcasts can support clearer guidance for medical podcast production, providing opportunities for educators to optimise podcast creation efficiency and educational effectiveness.

Developing a Web-Based Asynchronous Case Discussion Format on Social Media to Teach Clinical Reasoning: Mixed Methods Study

BACKGROUND

Case-based learning conferences are valuable to trainees, but growing clinical demands hinder consistent attendance. Social media increasingly acts as a venue for trainees to supplement their education asynchronously. We designed and implemented a web-based asynchronous clinical case discussion series on the Twitter social media platform to fill this educational gap.

OBJECTIVE

The aim of this mixed methods study is to examine the nature of interactions among web-based case discussion participants and assess local attitudes regarding the educational intervention.

METHODS

Starting in February 2018, we posted clinical vignettes to a dedicated Twitter account with the prompt "What else do you want to know?" to stimulate discussion. The authors replied in real time when case discussion participants requested additional details. Additional data about the case were posted at regular intervals to the discussion thread to advance the overall case discussion. Participants were asked to explain their reasoning and support their conclusions when appropriate. Web-based engagement was assessed using Twitter Analytics. Participants' posts were qualitatively analyzed for themes, with special attention to examples of using clinical reasoning skills. A codebook of types of participant posts and interactions was refined iteratively. Local engagement and attitudes at our institution were assessed by surveying internal medicine trainees (n=182) and faculty (n=165) after 6 months.

RESULTS

Over a 6-month period, 11 live case discussions were engaged with by users 1773 times. A total of 86 Twitter profiles spanning 22 US states and 6 countries contributed to discussions among participants and the authors. Participants from all training levels were present, ranging from students to faculty. Interactions among participants and the case moderators were most commonly driven by clinical reasoning, including hypothesis-driven information gathering, discussing the differential diagnosis, and data interpretation or organization. Of 71 respondents to the local survey, 29 (41%) reported having a Twitter account. Of the 29 respondents with Twitter accounts, 17 (59%) reported participating in the case discussions. Respondents agreed that case participation increased both their clinical reasoning skills (15/17, 88%) and clinical knowledge (13/17, 76%).

CONCLUSIONS

A social media-based serialized case discussion was a feasible asynchronous teaching method for engaging web-based learners of all levels in a clinical reasoning discussion. Further study should examine what factors drive trainee participation in web-based case discussions and under what circumstances asynchronous discussion might be preferred over in-person teaching activities.

#MedEd: Mapping the Current Landscape of Medical Education Discourse and Stakeholder Participation Across Social Media Platforms

Background Medical education is a constantly evolving and multifaceted field that requires ongoing discussion and innovation. Social media platforms have emerged as a popular medium for disseminating information and engaging in professional discourse among medical educators. In particular, the hashtag #MedEd has gained widespread recognition amongst individuals and organizations within the medical education community. Our objective is to gain insights into the types of information and discussions surrounding medical education, as well as the individuals or organizations involved in these conversations. Methods Searches were conducted across major social media platforms, including Twitter, Instagram, and Facebook, using the hashtag #MedEd. The top 20 posts posted on these platforms were analyzed through a reflexive thematic analysis approach utilizing the Braun and Clarke method. Furthermore, an examination was conducted on the profiles of those responsible for posting the aforementioned top posts, to ascertain the degree of participation from individuals versus organizations within the broader discourse pertaining to the topic. Results Our analysis revealed three thematic categories associated with the usage of the #MedEd hashtag, including discussions on "continuous learning and medical case presentations," "medical specialties and topics," and "medical education pedagogy." The analysis revealed that social media can serve as a valuable platform for medical education by providing access to a diverse range of learning resources, fostering collaboration and professional networking, and providing innovative teaching methods. Furthermore, profile analysis showed that individuals were more actively involved in the discussion of medical education topics on social media compared to organizations across all three platforms. Conclusion Our study highlights the significant role that social media platforms play in facilitating the exchange of information and ideas within the medical education community. The hashtag #MedEd serves as a means of connecting individuals and organizations across the globe, enabling them to engage in professional discourse and stay informed on the latest developments in the field. Our findings suggest that a better understanding of the thematic categories and stakeholders involved in medical education discussions on social media can aid educators, learners, and organizations in enhancing their engagement with this dynamic field.

Guidelines: The Do's, Don'ts and Don't Knows of Creating Open Educational Resources

Background

In medical education, there is a growing global demand for Open Educational Resources (OERs). However, OER creators are challenged by a lack of uniform standards. In this guideline, the authors curated the literature on how to produce OERs for medical education with practical guidance on the Do's, Don'ts and Don't Knows for OER creation in order to improve the impact and quality of OERs in medical education.

Methods

We conducted a rapid literature review by searching OVID MEDLINE, EMBASE, and Cochrane Central database using keywords "open educational resources" and "OER". The search was supplemented by hand searching the identified articles' references. We organized included articles by theme and extracted relevant content. Lastly, we developed recommendations via an iterative process of peer review and discussion: evidence-based best practices were designated Do's and Don'ts while gaps were designated Don't Knows. We used a consensus process to quantify evidentiary strength.

Results

The authors performed full text analysis of 81 eligible studies. A total of 15 Do's, Don't, and Don't Knows guidelines were compiled and presented alongside relevant evidence about OERs.

Discussion

OERs can add value for medical educators and their learners, both as tools for expanding teaching opportunities and for promoting medical education scholarship. This summary should guide OER creators in producing high-quality resources and pursuing future research where best practices are lacking.

Evaluation of Open Access Websites for Anesthesia Education

BACKGROUND

While the prevalence of free, open access medical education resources for health professionals has expanded over the past 10 years, many educational resources for health care professionals are not publicly available or require fees for access. This lack of open access creates global inequities in the availability and sharing of information and may have the most significant impact on health care providers with the greatest need. The extent of open access online educational websites aimed for clinicians and trainees in anesthesiology worldwide is unknown. In this study, we aimed to identify and evaluate the quality of websites designed to provide open access educational resources for anesthesia trainees and clinicians.

METHODS

A PubMed search of articles published between 2009 and 2020, and a Startpage search engine web search was conducted in May 2021 to identify websites using the following inclusion criteria: (1) contain educational content relevant for anesthesia providers or trainees, (2) offer content free of charge, and (3) are written in the English language. Websites were each scored by 2 independent reviewers using a website quality evaluation tool with previous validity evidence that was modified for anesthesia (the Anesthesia Medical Education Website Quality Evaluation Tool).

RESULTS

Seventy-five articles and 175 websites were identified; 37 websites met inclusion criteria. The most common types of educational content contained in the websites included videos (66%, 25/37), text-based resources (51%, 19/37), podcasts (35%, 13/37), and interactive learning resources (32%, 12/37). Few websites described an editorial review process (24%, 9/37) or included opportunities for active engagement or interaction by learners (30%,11/37). Scores by tertile differed significantly across multiple domains, including disclosure of author/webmaster/website institution; description of an editorial review process; relevancy to residents, fellows, and faculty; comprehensiveness; accuracy; disclosure of content creation or revision; ease of access to information; interactivity; clear and professional presentation of information; and links to external information.

CONCLUSIONS

We found 37 open access websites for anesthesia education available on the Internet. Many of these websites may serve as a valuable resource for anesthesia clinicians looking for self-directed learning resources and for educators seeking to curate resources into thoughtfully integrated learning experiences. Ongoing efforts are needed to expand the number and improve the existing open access websites, especially with interactivity, to support the education and training of anesthesia providers in even the most resource-limited areas of the world. Our findings may provide recommendations for those educators and organizations seeking to fill this needed gap to create new high-quality educational websites.

Systematic online academic resources (SOAR) review: Sickle cell disorders

Background

Free open-access medical education (FOAM) resources have become highly utilized resources in emergency medicine education. However, FOAM content often lacks the traditional peer review process, leaving quality assessment to the readers. In this systematic online academic resource (SOAR) review, we apply a systematic methodology to assess the quality of FOAM resources on sickle cell disease (SCD).

Methods

We searched keywords for SCD using FOAM Search and the top 50 FOAM websites listed on the Social Media Index. Resources found were screened using inclusion/exclusion criteria, and a total of 53 resources underwent full-text quality assessment using the revised Medical Education Translational Resources Impact and Quality (rMETRIQ) tool.

Results

The search yielded 520 resources, of which 53 met the criteria for quality assessment. A total of eight posts (15.1% of posts) were identified as high quality (rMETRIQ ≥ 16). The most commonly addressed topics within SCD topics included acute chest syndrome, acute pain crisis and general review of SCD. A total of 11 posts (21% of posts) were found to have an rMETRIQ score of less than 7, which may indicate poor quality. The most commonly identified type of resource was personalized reading (64%) and a number of posts were deemed to not have an appropriate use due to poor quality (15%).

Conclusions

We were able to systematically search a wide range of resources to identify, appraise, and organize FOAM resources on the topic of SCD. A final list of eight high-quality resources can guide trainees, educator recommendations, and FOAM authors.

Assessing on-line medical education resources: A primer for acute care medical professionals and others

The internet is increasingly used to propagate medical education, debate, and even disinformation. Therefore, this primer aims to help acute care medical professionals, as well as the public. This is because we all need to be able to critically appraise digital products, appraise content producers, and reflect upon our own on-line presence. This article discusses the challenges and opportunities associated with online medical resources. We then review Free Open Access Medical Education (FOAMed) and the key tools used to assess the trustworthiness of on-line medical products. Specifically, after discussing the pros and cons of traditional academic quality metrics, we compare and contrast the Social Media Index, the ALiEM AIR score, the Revised METRIQ Score, and gestalt. We also discuss internet search engines, peer review, and the important message behind the seemingly tongue-in-cheek Kardashian Index. Hopefully, this primer bolsters basic digital literacy and helps trainees, practitioners, and the public locate useful and reliable on-line resources. Importantly, we highlight the continued importance of traditional academic medicine and primary source publications.

Medical Education Blog and Podcast Utilization During the COVID-19 Pandemic

Introduction The coronavirus disease 2019 (COVID-19) pandemic disrupted traditional in-person learning models. Free Open Access Medical (FOAM) education resources naturally filled this void, so we evaluated how medical blog and podcast utilization changed during the early months of the pandemic. Methods Academic medical podcast and blog producers were surveyed on blog and podcast utilization immediately before (January-March 2020) and after (April-May 2020) the COVID-19 pandemic declaration and subsequent lockdown. Utilization is quantified in terms of blog post pageviews and podcast downloads. Linear regression was used to estimate the effect of publication during the COVID-19 period on 30-day downloads or pageviews. A linear mixed model was developed to confirm this relationship after adjustment for independent predictors of higher 30-day downloads or pageviews, using the podcast or blog as a random intercept. Results Compared to the pre-pandemic period, downloads and pageviews per unique blog and podcast publication significantly increased for blogs (median 30-day pageviews 802 to 1860, p<0.0001) but not for podcasts (median 30-day downloads 2726 to 1781, p=0.27). Publications that contained COVID-19 content were strongly associated with higher monthly utilization (β=7.21, 95% CI 6.29-8.14 p<0.001), and even non-COVID-19 material had higher utilization in the early pandemic (median 30-day downloads/pageviews 868 to 1380, p<0.0001). Discussion The increased blog pageviews during the early months of the COVID-19 pandemic demonstrated the important role of blogs in rapid knowledge translation. Podcasts did not experience a similar increase in utilization.

Going viral: A scoping review of the current state and impact of online research dissemination in emergency medicine

BACKGROUND

The use of free open-access medical education (FOAM) and other online knowledge dissemination methods has increased over the past decade. However, the role and impact of these tools in the knowledge translation continuum are poorly understood, potentially limiting the ability of knowledge generators to fully harness and exploit their potential. Here, we aim to comprehensively map and synthesize the literature describing the use of online tools for the dissemination of emergency medicine research.

METHODS

Using scoping review methodology, we searched the traditional literature via PubMed, CINAHL, EMBASE, ERIC, SCOPUS, and the gray literature for publications exploring online methods to disseminate new research findings. We synthesized the results and constructed a conceptual model of current research dissemination methods.

RESULTS

We included 79 out of 655 unique abstracts and articles identified in our search, 62 of which were from the traditional literature. We describe six primary domains: integration with traditional literature, measurement of dissemination, online organizations and communities of practice, professional development, quality assurance tools and techniques, and advantages and disadvantages of FOAM. For each domain we present an exemplar article and prevailing gaps in knowledge. Finally, we propose a current conceptual framework for dissemination of new research findings that describes both traditional and novel methods of dissemination.

CONCLUSIONS

This comprehensive review of the literature and current dissemination framework will empower researchers, research networks, and granting organizations to maximize their use of FOAM and other online methods to disseminate new knowledge as well as provide clinicians a better understanding of the tools and methods by which to access and implement new research findings.

Frequency of Social Media and Digital Scholarship Keywords in U.S. Medical Schools' Promotion and Tenure Guidelines

PURPOSE

Medical schools must have clear policies and procedures for promotion and tenure (P&T) of faculty. Social media and digital scholarship (SMDS) is an emerging form of scholarship capable of reaching audiences quickly, conveniently, and in a wide variety of formats. It is unclear how frequently SMDS is considered during P&T reviews. The authors sought to determine whether current P&T guidelines at medical schools consider SMDS.

METHOD

The authors acquired P&T guidelines from any U.S. Liaison Committee on Medical Education-accredited medical school (or their governing university) that were available online between October and December 2020. Using an iterative process, they developed a bank of keywords that were specific to SMDS or that could include SMDS between October and December 2020. The authors searched each school's guidelines for each keyword and determined whether the word was being used in relation to crediting faculty for SMDS in the context of P&T procedures. The primary outcome measure was the dichotomous presence or absence of SMDS-specific keywords in each school's P&T guidelines.

RESULTS

The authors acquired P&T guidelines from 145/154 (94%) medical schools. After removing duplicate documents, the authors considered 139 guidelines. The keyword bank included 59 terms, of which 49 were specific to SMDS and 10 were umbrella terms that could be inclusive of SMDS. Of the 139 guidelines, 121 (87%) contained at least 1 SMDS-specific keyword. Schools had a median of 3 SMDS-specific keywords in their P&T guidelines.

CONCLUSIONS

As the presence and impact of SMDS increase, schools should provide guidance on its role in the P&T process. Faculty should receive clear guidance on how to document quality SMDS for their promotion file.

Systematic online academic resource (SOAR) review: Endocrine, metabolic, and nutritional disorders

BACKGROUND

Free open-access medical education (FOAM) has become an integral resource for medical school and residency education. However, questions of quality and inconsistent coverage of core topics remain. In this second entry of the SAEM Systematic Online Academic Resource (SOAR) series, we describe the application of a systematic methodology to identify, curate, and describe FOAM topics specific to endocrine, metabolic, and nutritional disorders as defined by the 2016 Model of the Clinical Practice of Emergency Medicine (MCPEM).

METHODS

We developed an automated algorithm to search 264 keywords derived from nine subtopics within the MCPEM category in the FOAM Search (a customized FOAM search tool) and the Social Media index. The top 100 results were extracted for each keyword. Resources underwent a manual iterative screening process, and those relevant to endocrine, metabolic, or nutritional disorders and EM were evaluated with the revised Medical Education Translational Resources: Impact and Quality (rMETRIQ) tool.

RESULTS

The search yielded 36,346 resources, of which 756 met the criteria for quality assessment. After rMETRIQ tool training, four raters demonstrated an average measured intraclass correlation coefficient of 0.94 (95% confidence interval = 0.88 to 0.97, p < 0.001). A total of 121 posts (16% of posts) covering 25 subtopics were identified as high quality (rMETRIQ ≥16). The most covered subtopic was potassium disorders, representing 15% of all posts. Subtopics that did not have a high-quality resource identified include metabolic alkalosis, respiratory alkalosis, fluid overload, phosphorus metabolism, hyperglycemia, malabsorption, malnutrition, and thyroiditis. From most to least common, the overall target audience was junior resident (91%), PGY-1 resident (88%), senior resident (81%), clerk (64%), attending (50%), and preclerkship (9%).

CONCLUSIONS

We systematically identified, described, and curated FOAM resources for EM learners on the topic of endocrine, metabolic, and nutritional disorders. A final list of high-quality resources can guide trainees, educator recommendations, and FOAM authors.

Evaluating Effectiveness of YouTube Videos for Teaching Medical Students CPR: Solution to Optimizing Clinician Educator Workload during the COVID-19 Pandemic

(1) Background: This study aimed to evaluate the effectiveness of using a pre-existing video on CPR to support preclinical resuscitation education for medical students; (2) Methods: In total, 129 students selected to learn CPR using a pre-existing YouTube video or the conventional screencast video by their university faculties. All students responded to the pre- and post-training multiple-choice questionnaire on the basic knowledge of CPR, and, based on their responses, an analysis of covariance (ANCOVA) was conducted to assess the comparability of effectiveness across learning modalities. (3) Results: Among the students, 49 (38.0%) students selected the YouTube video to learn about CPR and were treated as the intervention group. The mean pre-test scores and post-test scores of the YouTube and the instructor's video groups were 6.43 and 6.64, and 9.06 and 9.09, respectively. After controlling for the pre-test score effects, the results of ANCOVA did not show statistically significant differences between groups (p = 0.927), indicating comparable performance between groups that used YouTube and the instructor's videos. (4) Conclusion: Utilizing YouTube videos is a useful teaching strategy for teaching CPR knowledge, which would reduce the burden on faculty of creating screencast lecture videos for online learning on resuscitation.

The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources

BACKGROUND

Free Open-Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations.

OBJECTIVES

We assessed the AAT's usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool.

METHODS

As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability.

RESULTS

Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use-understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence-based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT's average measure ICC was 0.88 (95% CI = 0.77 to 0.95).

CONCLUSIONS

We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.

Holes in the FOAM: An Analysis of Curricular Comprehensiveness in Online Educational Resources

OBJECTIVES

We sought to evaluate Free Open Access Medical Education (FOAM), defined as online educational content available free to anyone, anywhere, at any time, by classifying the most impactful FOAM content per the Social Media Index into the topics and subtopics of the American Board of Emergency Medicine's Model of the Clinical Practice of Emergency Medicine. We then analyzed FOAM's comprehensiveness by describing over- and underrepresentation among these topics and subtopics.

METHODS

First, we searched for FOAM resources based on the most recent 12 months of relevant content for each organ system from the top 50 Social Media Index sites. Next, we classified all 898 posts into its related topics or subtopics per the American Board of Emergency Medicine's Model of the Clinical Practice of Emergency Medicine. Finally, we analyzed how comprehensively FOAM covered each organ system and the frequency of posts that covered each organ system subtopic as well as identified the subtopics with the most frequent coverage.

RESULTS

The search yielded 898 FOAM posts, of which cardiology and neurology were significantly overrepresented and psychobehavioral; obstetrics and gynecology; and head, ears, eyes, nose, and throat were significantly underrepresented. Among subtopics, acute coronary syndrome had the highest subtopic coverage consisting of 55.5% of all cardiology content. Other highly represented subtopics include renal colic; diabetic ketoacidosis; sepsis; and stroke with 39, 40, 40, and 71% of each of their topic's content, respectively.

CONCLUSIONS

Although residents and programs are frequently incorporating FOAM into the educational curriculum, these materials seem to lack comprehensiveness. Educators and learners must be aware of these deficits in creating comprehensive emergency medicine curricula.

Quality Appraisal and Assurance Techniques for Free Open Access Medical Education (FOAM) Resources: A Rapid Review

Free open access medical education (FOAM) has disrupted traditional modes of knowledge translation and dissemination. These are popular resources with a wide educational reach. Nephrology has been a leader in FOAM, but many skeptics still question the accuracy and reliability of this content. Recently, quality-assurance techniques have been developed to address these concerns. These techniques may be helpful for readers to appraise the online literature and for institutions to reward the production of high-quality open educational resources. We performed a rapid review of the literature. A medical librarian conducted a systematic search of the Medline and Cumulative Index of Nursing and Allied Health Literature databases. Two independent assessors screened and selected articles, performed a hand-search of reference lists, and scored articles on their quality using the Medical Education Research Study Quality Instrument. Thirteen reports were included for the final descriptive analysis. We identified 10 quality-assessment techniques, and 4 of them having been validated. The quality of the reports was fairly high, with an average Medical Education Research Study Quality Instrument score of 11.5 of 18 (SD, 2.3; range, 7.25-14.25). The calculated Cronbach α was 0.85. There is burgeoning literature on the topic of critical appraisal of open educational resources, and, more specifically, FOAM resources. Many of the techniques used are of varying quality and developed with different intended uses and audiences. By continuing to refine these tools, we can continue not only to support and legitimize the FOAM movement, but also foster individual critical appraisal skills that increasingly are necessary in this age of information.

Content Analysis and Characterization of Medical Tweets During the Early Covid-19 Pandemic

Objective

The novel coronavirus disease 2019 (Covid-19) has infected millions worldwide and impacted the lives of many folds more. Many clinicians share new Covid-19-related resources, research, and ideas within the online Free Open Access to Medical Education (FOAM) community of practice. This study provides a detailed content and contributor analysis of Covid-19-related tweets among the FOAM community during the first months of the pandemic.

Design, Setting, and Participants

In this social media content analysis study, Twitter was searched from November 1, 2019, to March 21, 2020, for English tweets discussing Covid-19 in the FOAM community. Tweets were classified into one of 13 pre-specified content categories: original research, editorials, FOAM resource, public health, podcast or video, learned experience, refuting false information, policy discussion, emotional impact, blatantly false information, other Covid-19, and non-Covid-19. Further analysis of linked original research and FOAM resources was performed. One-thousand (1000) randomly selected contributor profiles and those deemed to have contributed false information were analyzed.

Results

The search yielded 8541 original tweets from 4104 contributors. The number of tweets in each content category were: 1557 other Covid-19 (18.2%), 1190 emotional impact (13.9%), 1122 FOAM resources (13.1%), 1111 policy discussion (13.0%), 928 advice (10.9%), 873 learned experience (10.2%), 424 non-Covid-19 (5.0%), 410 podcast or video (4.8%), 304 editorials (3.6%), 275 original research (3.2%), 245 public health (2.9%), 83 refuting false information (1.0%), and 19 blatantly false (0.2%).

Conclusions

Early in the Covid-19 pandemic, the FOAM community used Twitter to share Covid-19 learned experiences, online resources, crowd-sourced advice, and research and to discuss the emotional impact of Covid-19. Twitter also provided a forum for post-publication peer review of new research. Sharing blatantly false information within this community was infrequent. This study highlights several potential benefits from engaging with the FOAM community on Twitter.

Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study

INTRODUCTION

Podcasts are increasingly being used for medical education. Studies have found that the assessment of the quality of online resources can be challenging. We sought to determine the reliability of gestalt quality assessment of education podcasts in emergency medicine.

METHODS

An international, interprofessional sample of raters was recruited through social media, direct contact, and the extended personal network of the study team. Each participant listened to eight podcasts (selected to include a variety of accents, number of speakers, and topics) and rated the quality of that podcast on a seven-point Likert scale. Phi coefficients were calculated within each group and overall. Decision studies were conducted using a phi of 0.8.

RESULTS

A total of 240 collaborators completed all eight surveys and were included in the analysis. Attendings, medical students, and physician assistants had the lowest individual-level variance and thus the lowest number of required raters to reliably evaluate quality (phi >0.80). Overall, 20 raters were required to reliably evaluate the quality of emergency medicine podcasts.

DISCUSSION

Gestalt ratings of quality from approximately 20 health professionals are required to reliably assess the quality of a podcast. This finding should inform future work focused on developing and validating tools to support the evaluation of quality in these resources.

Consensus Guidelines for Digital Scholarship in Academic Promotion

Introduction

As scholarship moves into the digital sphere, applicant and promotion and tenure (P&T) committee members lack formal guidance on evaluating the impact of digital scholarly work. The P&T process requires the appraisal of individual scholarly impact in comparison to scholars across institutions and disciplines. As dissemination methods evolve in the digital era, we must adapt traditional P&T processes to include emerging forms of digital scholarship.

Methods

We conducted a blended, expert consensus procedure using a nominal group process to create a consensus document at the Council of Emergency Medicine Residency Directors Academic Assembly on April 1, 2019.

Results

We discussed consensus guidelines for evaluation and promotion of digital scholarship with the intent to develop specific, evidence-supported recommendations to P&T committees and applicants. These recommendations included the following: demonstrate scholarship criteria; provide external evidence of impact; and include digital peer-review roles. As traditional scholarship continues to evolve within the digital realm, academic medicine should adapt how that scholarship is evaluated. P&T committees in academic medicine are at the epicenter for supporting this changing paradigm in scholarship.

Conclusion

P&T committees can critically appraise the quality and impact of digital scholarship using specific, validated tools. Applicants for appointment and promotion should highlight and prepare their digital scholarship to specifically address quality, impact, breadth, and relevance. It is our goal to provide specific, timely guidance for both stakeholders to recognize the value of digital scholarship in advancing our field.

Medical Education Podcasts: Where We Are and Questions Unanswered

Social media, particularly podcasts, has become an influential modality within informal medical education. As podcasts continue to become more prevalent among learners of all types, clinical educators of the future must be able to help navigate this new pedagogy. Preliminary data demonstrates that medical students, residents, post-training physicians, and advanced practitioners all utilize podcasts for their own benefit. New data is discussed in the setting of the current literature on podcasting and important questions remain to determine how this new form of learning can and will be integrated into formal and informal medical curriculum.

Rethinking Residency Conferences in the Era of COVID-19

The COVID-19 pandemic requires a substantial change to the traditional approach to conference didactics. Switching to a virtual medium for conference sessions presents several challenges, particularly with regard to aspects that rely heavily on in-person components (e.g., simulation, ultrasound). This paper will discuss the challenges and strategies to address them for conference planning in the era of COVID-19.