#MedEd: exploring the relationship between altmetrics and traditional measures of dissemination in health professions education

Assessment of the dissemination of COVID-19 articles across social media: An Altmetrics Study

BACKGROUND

The use of social media assists in the distribution of information about COVID-19 to the general public and health professionals. Alternative-level metrics (Altmetrics) is an alternative method to traditional bibliometrics that assess the amount of sharing and spreading of a scientific article on social media platforms.

OBJECTIVE

Our objective was to characterize and compare traditional bibliometrics (citation-count) with newer metrics (Altmetric Attention Score) of the top 100 Altmetric scored COVID-19 articles.

METHODS

The 100 highest Altmetric Attention Score (AAS) articles were identified utilizing the Altmetric explorer in May 2020. AAS, journal name, and mentions from various social media databases (Twitter, Facebook, Wikipedia, Reddit, Mendeley, Dimension) of each article were collected. Citation-counts were collected from the Scopus database.

RESULTS

The median AAS and citation-count were 4922.50 and 24.00, respectively. Of 100 articles, The New England Journal of Medicine published the most articles at 18% (18/100). Twitter was the most frequently used social media platform with 96.3% of the mentions (985,429/1,022,975). Positive correlations were seen between AAS and citation-count (r2=.0973; P=.002).

CONCLUSIONS

Our research characterized the top 100 articles by AAS regarding COVID-19 in the Altmetric database. Altmetrics could complement with traditional citation-count when assessing the dissemination of an article regarding COVID-19.

INTERNATIONAL REGISTERED REPORT

RR2-10.2196/21408.

Evaluating the impact of a CTSA program from 2008 to 2021 through bibliometrics, social network analysis, and altmetrics

Introduction

We evaluate a CTSA program hub by applying bibliometrics, social network analysis (SNA), and altmetrics and examine the changes in research productivity, citation impact, research collaboration, and CTSA-supported research topics since our pilot study in 2017.

Methods

The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2021. We applied measures and metrics from bibliometrics, SNA, and altmetrics to the dataset. In addition, we analyzed research topics and correlations between different metrics.

Results

1154 NC TraCS-supported publications generated over 53,560 citation counts by April 2021. The average cites per year and the relative citation ratio (RCR) mean of these publications improved from 33 and 2.26 in 2017 to 48 and 2.58 in 2021. The number of involved UNC units in the most published authors' collaboration network increased from 7 (2017) to 10 (2021). NC TraCS-supported co-authorship involved 61 NC organizations. PlumX metrics identified articles with the highest altmetrics scores. About 96% NC TraCS-supported publications have above the average SciVal Topic Prominence Percentile; the average approximate potential to translate of the included publication was 54.2%; and 177 publications addressed health disparity issues. Bibliometric measures (e.g., citation counts, RCR) and PlumX metrics (i.e., Citations, Captures, and Social-Media) are positively correlated (p < .05).

Conclusion

Bibliometrics, SNA, and altmetrics offer distinctive but related perspectives to examine CTSA research performance and longitudinal growth, especially at the individual program hub level. These perspectives can help CTSAs build program foci.

Disseminating implementation science: Describing the impact of animations shared via social media

Background

Dissemination is an under-researched activity that is important to researchers and funders and may have a role in creating conditions for implementation. We aim to study the impact of two animations shared via social media upon dissemination.

Methods

We developed two short animations to increase the awareness of healthcare positional leaders of work undertaken to enhance a much-used implementation intervention. We measured both how frequently the related articles were accessed, and engagement with the research team, before and after the intervention. We analysed variation over time using statistical process control to identify both trend and periods of significant change.

Results

We found evidence that the animation increased how frequently the articles were accessed, with a significant increase (p = <0.01) during the two weeks after release of each animation. One animation was associated with an increase in positional leaders’ engagement with the research team.

Conclusions

Animations shared via social media can enhance dissemination. We describe lessons from the work to develop the intervention and support calls for work to increase the understanding and adoption of effective dissemination interventions. Our findings provide support for further work using randomised study designs.

Introducing the EMPIRE Index: A novel, value-based metric framework to measure the impact of medical publications

Article-level measures of publication impact (alternative metrics or altmetrics) can help authors and other stakeholders assess engagement with their research and the success of their communication efforts. The wide variety of altmetrics can make interpretation and comparative assessment difficult; available summary tools are either narrowly focused or do not reflect the differing values of metrics from a stakeholder perspective. We created the EMPIRE (EMpirical Publication Impact and Reach Evaluation) Index, a value-based, multi-component metric framework for medical publications. Metric weighting and grouping were informed by a statistical analysis of 2891 Phase III clinical trial publications and by a panel of stakeholders who provided value assessments. The EMPIRE Index comprises three component scores (social, scholarly, and societal impact), each incorporating related altmetrics indicating a different aspect of engagement with the publication. These are averaged to provide a total impact score and benchmarked so that a score of 100 equals the mean scores of Phase III clinical trial publications in the New England Journal of Medicine (NEJM) in 2016. Predictor metrics are defined to estimate likely long-term impact. The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore, with the societal impact component providing unique insights. Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not. Analysis of NEJM ‘notable articles’ showed that observational studies had the highest total impact and component scores, except for societal impact, for which surgical studies had the highest score. The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics and may enable medical researchers to assess the impact of publications easily and to understand what characterizes impactful research.

Delineating the field of medical education: Bibliometric research approach(es)

BACKGROUND

The field of medical education remains poorly delineated such that there is no broad consensus of articles or journals that comprise 'the field'. This lack of consensus indicates a missed opportunity for researchers to generate insights about the field that could facilitate conducting bibliometric studies and other research designs (e.g., systematic reviews) and also enable individuals to identify themselves as 'medical education researchers'. Other fields have utilised bibliometric field delineation, which is the assigning of articles or journals to a certain field in an effort to define that field.

PROCESS

In this Research Approach, three bibliometric field delineation approaches-information retrieval, core journals, and journal co-citation-are introduced. For each approach, the authors describe attempts to apply it in medical education and identify related strengths and weaknesses. Based on co-citation, the authors propose the Medical Education Journal List 24 (MEJ-24), as a starting point for delineating medical education and invite the community to collaborate on improving and potentially expanding this list.

PEARLS

As a research approach, field delineation is complicated, and there is no clear best way to delineate the field of medical education. However, recent advances in information science provide potentially fruitful approaches to deal with the field's complexity. When considering these approaches, researchers should consider collaborating with bibliometricians. Bibliometric approaches rely on available metadata for articles and journals, which necessitates that researchers examine the metadata prior to analysis to understand its strengths and weaknesses, and to assess how this might affect data interpretation. While using bibliometric approaches for field delineation is valuable, it is important to remember that these techniques are only as good as the research team's interpretation of the data, which suggests that an expanded approach is needed to better delineate medical education, an approach that includes active discussion within the medical education community.

Be in the Digital Room Where it Happens, Part I: Tweeting & Technology for Career Development

Social media has become a part of everyday life. It has changed the way we obtain and distribute information, connect, and interact with others. As the number of platforms and users grow, medical professionals have learned the value social media can have in education, research, advocacy, and clinical care initiatives. Platforms provide opportunities to network, build collaborations, and develop a reputation. This is part one of a two-part series. This article provides an overview on how social media can benefit professional career development for clinicians and researchers, as well as for advocacy to raise awareness against biases, disparities, and for patient benefit. We review challenges, limitations, and best practices for social media use by medical professionals with neurology-specific examples.

Knowledge syntheses in medical education: Meta-research examining author gender, geographic location, and institutional affiliation

Introduction

Authors of knowledge syntheses make many subjective decisions during their review process. Those decisions, which are guided in part by author characteristics, can impact the conduct and conclusions of knowledge syntheses, which assimilate much of the evidence base in medical education. To better understand the evidence base, this study describes the characteristics of knowledge synthesis authors, focusing on gender, geography, and institution.

Methods

In 2020, the authors conducted meta-research to examine authors of 963 knowledge syntheses published between 1999 and 2019 in 14 core medical education journals.

Results

The authors identified 4,110 manuscript authors across all authorship positions. On average there were 4.3 authors per knowledge synthesis (SD = 2.51, Median = 4, Range = 1–22); 79 knowledge syntheses (8%) were single-author publications. Over time, the average number of authors per synthesis increased (M = 1.80 in 1999; M = 5.34 in 2019). Knowledge syntheses were authored by slightly more females (n = 2047; 50.5%) than males (n = 2005; 49.5%) across all author positions. Authors listed affiliations in 58 countries, and 58 knowledge syntheses (6%) included authors from low- or middle-income countries. Authors from the United States (n = 366; 38%), Canada (n = 233; 24%), and the United Kingdom (n = 180; 19%) published the most knowledge syntheses. Authors listed affiliation at 617 unique institutions, and first authors represented 362 unique institutions with greatest representation from University of Toronto (n = 55, 6%). Across all authorship positions, the large majority of knowledge syntheses (n = 753; 78%) included authors from institutions ranked in the top 200 globally.

Conclusion

Knowledge synthesis author teams have grown over the past 20 years, and while there is near gender parity across all author positions, authorship has been dominated by North American researchers located at highly ranked institutions. This suggests a potential overrepresentation of certain authors with particular characteristics, which may impact the conduct and conclusions of medical education knowledge syntheses.

Scoping reviews in medical education: A scoping review

OBJECTIVES

Over the last two decades, the number of scoping reviews in core medical education journals has increased by 4200%. Despite this growth, research on scoping reviews provides limited information about their nature, including how they are conducted or why medical educators undertake this knowledge synthesis type. This gap makes it difficult to know where the field stands and may hamper attempts to improve the conduct, reporting and utility of scoping reviews. Thus, this review characterises the nature of medical education scoping reviews to identify areas for improvement and highlight future research opportunities.

METHOD

The authors searched PubMed for scoping reviews published between 1/1999 and 4/2020 in 14 medical education journals. The authors extracted and summarised key bibliometric data, the rationales given for conducting a scoping review, the research questions and key reporting elements as described in the PRISMA-ScR. Rationales and research questions were mapped to Arksey and O'Malley's reasons for conducting a scoping review.

RESULTS

One hundred and one scoping reviews were included. On average, 10.1 scoping reviews (SD = 13.1, median = 4) were published annually with the most reviews published in 2019 (n = 42). Authors described multiple reasons for undertaking scoping reviews; the most prevalent being to summarise and disseminate research findings (n = 77). In 11 reviews, the rationales for the scoping review and the research questions aligned. No review addressed all elements of the PRISMA-ScR, with few authors publishing a protocol (n = 2) or including stakeholders (n = 20). Authors identified shortcomings of scoping reviews, including lack of critical appraisal.

CONCLUSIONS

Scoping reviews are increasingly conducted in medical education and published by most core journals. Scoping reviews aim to map the depth and breadth of emerging topics; as such, they have the potential to play a critical role in the practice, policy and research of medical education. However, these results suggest improvements are needed for this role to be fully realised.

The Online Attention to Cleft Lip and Palate Research: An Altmetric Analysis

OBJECTIVES

To identify research articles related to cleft lip and/or cleft palate (CL/P) that generated the highest online attention.

METHODS

Altmetric Explorer was used to identify the 100 articles with the highest Altmetric Attention Score (AAS). Descriptive and correlation statistics were performed to study the characteristics of these articles in relation to their publication data, research type and domain, number of Mendeley readers, and dimensions citations. Citation counts were extracted from Scopus and Google Scholar.

RESULTS

The median AAS for the top 100 outputs was 22 (range from 12 to 458). The outputs were mostly discussed on Twitter (median = 8; range = 0-131). Topics discussing treatment and care for patients with CL/P accounted for 38% of the articles with the highest AAS followed by etiology and risk factors (32%). The majority of articles originated from the USA (46%) followed by Europe (16%) and the United Kingdom (15%). No significant differences were observed in AAS among different study designs, topic domains, journals' ranking and impact factor, and the number of citations in Scopus and Google Scholar.

CONCLUSIONS

Researchers should consider use of social platforms to disseminate their work among scholars and nonscholars. Altmetrics can be combined with traditional metrics for a more comprehensive assessment of research impact.

Knowledge syntheses in medical education: A bibliometric analysis

PURPOSE

This bibliometric analysis maps the landscape of knowledge syntheses in medical education. It provides scholars with a roadmap for understanding where the field has been and where it might go in the future, thereby informing research and educational practice. In particular, this analysis details the venues in which knowledge syntheses are published, the types of syntheses conducted, citation rates they produce, and altmetric attention they garner.

METHOD

In 2020, the authors conducted a bibliometric analysis of knowledge syntheses published in 14 core medical education journals from 1999 to 2019. To characterize the studies, metadata were extracted from PubMed, Web of Science, Altmetrics Explorer, and Unpaywall.

RESULTS

The authors analyzed 963 knowledge syntheses representing 3.1% of the total articles published (n = 30,597). On average, 45.9 knowledge syntheses were published annually (SD = 35.85, median = 33), and there was an overall 2620% increase in the number of knowledge syntheses published from 1999 to 2019. The journals each published, on average, a total of 68.8 knowledge syntheses (SD = 67.2, median = 41) with Medical Education publishing the most (n = 189; 19%). Twenty-one types of knowledge synthesis were identified, the most prevalent being systematic reviews (n = 341; 35.4%) and scoping reviews (n = 88; 9.1%). Knowledge syntheses were cited an average of 53.80 times (SD = 107.12, median = 19) and received a mean Altmetric Attention Score of 14.12 (SD = 37.59, median = 6).

CONCLUSIONS

There has been considerable growth in knowledge syntheses in medical education over the past 20 years, contributing to medical education's evidence base. Beyond this increase in volume, researchers have introduced methodological diversity in these publications, and the community has taken to social media to share knowledge syntheses. Implications for the field, including the impact of synthesis types and their relationship to knowledge translation, are discussed.

Assessing the Dissemination of COVID-19 Articles Across Social Media With Altmetric and PlumX Metrics: Correlational Study

BACKGROUND

The use of social media assists in the distribution of COVID-19 information to the general public and health professionals. Alternative-level metrics (ie, altmetrics) and PlumX metrics are new bibliometrics that can assess how many times a scientific article has been shared and how much a scientific article has spread within social media platforms.

OBJECTIVE

Our objective was to characterize and compare the traditional bibliometrics (ie, citation count and impact factors) and new bibliometrics (ie, Altmetric Attention Score [AAS] and PlumX score) of the top 100 COVID-19 articles with the highest AASs.

METHODS

The top 100 articles with highest AASs were identified with Altmetric Explorer in May 2020. The AASs, journal names, and the number of mentions in various social media databases of each article were collected. Citation counts and PlumX Field-Weighted Citation Impact scores were collected from the Scopus database. Additionally, AASs, PlumX scores, and citation counts were log-transformed and adjusted by +1 for linear regression, and Spearman correlation coefficients were used to determine correlations.

RESULTS

The median AAS, PlumX score, and citation count were 4922.50, 37.92, and 24.00, respectively. The New England Journal of Medicine published the most articles (18/100, 18%). The highest number of mentions (985,429/1,022,975, 96.3%) were found on Twitter, making it the most frequently used social media platform. A positive correlation was observed between AAS and citation count (r²=0.0973; P=.002), and between PlumX score and citation count (r²=0.8911; P<.001).

CONCLUSIONS

Our study demonstrated that citation count weakly correlated with AASs and strongly correlated with PlumX scores, with regard to COVID-19 articles at this point in time. Altmetric and PlumX metrics should be used to complement traditional citation counts when assessing the dissemination and impact of a COVID-19 article.

Can altmetric mentions predict later citations? A test of validity on data from ResearchGate and three social media platforms

Purpose

The main purpose of this study is to explore and validate the question “whether altmetric mentions can predict citations to scholarly articles”. The paper attempts to explore the nature and degree of correlation between altmetrics (from ResearchGate and three social media platforms) and citations.

Design/methodology/approach

A large size data sample of scholarly articles published from India for the year 2016 is obtained from the Web of Science database and the corresponding altmetric data are obtained from ResearchGate and three social media platforms (Twitter, Facebook and blog through Altmetric.com aggregator). Correlations are computed between early altmetric mentions and later citation counts, for data grouped in different disciplinary groups.

Findings

Results show that the correlation between altmetric mentions and citation counts are positive, but weak. Correlations are relatively higher in the case of data from ResearchGate as compared to the data from the three social media platforms. Further, significant disciplinary differences are observed in the degree of correlations between altmetrics and citations.

Research limitations/implications

The results support the idea that altmetrics do not necessarily reflect the same kind of impact as citations. However, articles that get higher altmetric attention early may actually have a slight citation advantage. Further, altmetrics from academic social networks like ResearchGate are more correlated with citations, as compared to social media platforms.

Originality/value

The paper has novelty in two respects. First, it takes altmetric data for a window of about 1–1.5 years after the article publication and citation counts for a longer citation window of about 3–4 years after the publication of article. Second, it is one of the first studies to analyze data from the ResearchGate platform, a popular academic social network, to understand the type and degree of correlations.

Peer review

The peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-11-2019-0364

Documenting Social Media Engagement as Scholarship: A New Model for Assessing Academic Accomplishment for the Health Professions

BACKGROUND

The traditional model of promotion and tenure in the health professions relies heavily on formal scholarship through teaching, research, and service. Institutions consider how much weight to give activities in each of these areas and determine a threshold for advancement. With the emergence of social media, scholars can engage wider audiences in creative ways and have a broader impact. Conventional metrics like the h-index do not account for social media impact. Social media engagement is poorly represented in most curricula vitae (CV) and therefore is undervalued in promotion and tenure reviews.

OBJECTIVE

The objective was to develop crowdsourced guidelines for documenting social media scholarship. These guidelines aimed to provide a structure for documenting a scholar's general impact on social media, as well as methods of documenting individual social media contributions exemplifying innovation, education, mentorship, advocacy, and dissemination.

METHODS

To create unifying guidelines, we created a crowdsourced process that capitalized on the strengths of social media and generated a case example of successful use of the medium for academic collaboration. The primary author created a draft of the guidelines and then sought input from users on Twitter via a publicly accessible Google Document. There was no limitation on who could provide input and the work was done in a democratic, collaborative fashion. Contributors edited the draft over a period of 1 week (September 12-18, 2020). The primary and secondary authors then revised the draft to make it more concise. The guidelines and manuscript were then distributed to the contributors for edits and adopted by the group. All contributors were given the opportunity to serve as coauthors on the publication and were told upfront that authorship would depend on whether they were able to document the ways in which they met the 4 International Committee of Medical Journal Editors authorship criteria.

RESULTS

We developed 2 sets of guidelines: Guidelines for Listing All Social Media Scholarship Under Public Scholarship (in Research/Scholarship Section of CV) and Guidelines for Listing Social Media Scholarship Under Research, Teaching, and Service Sections of CV. Institutions can choose which set fits their existing CV format.

CONCLUSIONS

With more uniformity, scholars can better represent the full scope and impact of their work. These guidelines are not intended to dictate how individual institutions should weigh social media contributions within promotion and tenure cases. Instead, by providing an initial set of guidelines, we hope to provide scholars and their institutions with a common format and language to document social media scholarship.

Social media in academics and research: 21st-century tools to turbocharge education, collaboration, and dissemination of research findings

The near-ubiquitous use of smartphones and the rapid emergence of free, widely used, social media platforms have combined to turbocharge the dissemination of information at a scale and speed that would have been unimaginable just a few years ago. Increasingly, internet-savvy pathologists of all ages from every corner of the world are flipping the paradigm of traditional academia by posting educational content online free of charge, unencumbered by the limitations of traditional print media and educational conferences. These platforms are being used in innovative ways, not just to disseminate research findings, but also to create new knowledge through using them to empower research collaborations. In this review, we outline ways in which social media platforms, such as Twitter, Facebook, and YouTube, are being used by pathologists to enhance academic work and facilitate the dissemination of research. We outline key differences between the various platforms with respect to pathology academics and research, and describe key areas in which these platforms have already made an impact. These include rapid dissemination of research findings to a worldwide audience, live transnational discussion of journal articles and conference proceedings, intercontinental networking between pathologists for academic purposes, free education on a global scale at minimal or no cost, and research collaborations initiated on and facilitated by social media platforms. Finally, we provide practical tips for pathologists who wish to adopt these novel 21st-century technologies to enhance their academic endeavours.

The Use of Social Media to Increase the Impact of Health Research: Systematic Review

BACKGROUND

Academics in all disciplines increasingly use social media to share their publications on the internet, reaching out to different audiences. In the last few years, specific indicators of social media impact have been developed (eg, Altmetrics), to complement traditional bibliometric indicators (eg, citation count and h-index). In health research, it is unclear whether social media impact also translates into research impact.

OBJECTIVE

The primary aim of this study was to systematically review the literature on the impact of using social media on the dissemination of health research. The secondary aim was to assess the correlation between Altmetrics and traditional citation-based metrics.

METHODS

We conducted a systematic review to identify studies that evaluated the use of social media to disseminate research published in health-related journals. We specifically looked at studies that described experimental or correlational studies linking the use of social media with outcomes related to bibliometrics. We searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), Excerpta Medica dataBASE (EMBASE), and Cumulative Index to Nursing and Allied Health Literature (CINAHL) databases using a predefined search strategy (International Prospective Register of Systematic Reviews: CRD42017057709). We conducted independent and duplicate study selection and data extraction. Given the heterogeneity of the included studies, we summarized the findings through a narrative synthesis.

RESULTS

Of a total of 18,624 retrieved citations, we included 51 studies: 7 (14%) impact studies (answering the primary aim) and 44 (86%) correlational studies (answering the secondary aim). Impact studies reported mixed results with several limitations, including the use of interventions of inappropriately low intensity and short duration. The majority of correlational studies suggested a positive association between traditional bibliometrics and social media metrics (eg, number of mentions) in health research.

CONCLUSIONS

We have identified suggestive yet inconclusive evidence on the impact of using social media to increase the number of citations in health research. Further studies with better design are needed to assess the causal link between social media impact and bibliometrics.

When will I get my paper back? A replication study of publication timelines for health professions education research

INTRODUCTION

Biomedical researchers have lamented the lengthy timelines from manuscript submission to publication and highlighted potential detrimental effects on scientific progress and scientists' careers. In 2015, Himmelstein identified the mean time from manuscript submission to acceptance in biomedicine as approximately 100 days. The length of publication timelines in health professions education (HPE) is currently unknown.

METHODS

This study replicates Himmelstein's work with a sample of 14 HPE journals published between 2008-2018. Using PubMed, 19,182 article citations were retrieved. Open metadata for each were downloaded, including the date the article was received by the journal, date the authors resubmitted revisions, date the journal accepted the article, and date of entry into PubMed. Journals without publication history metadata were excluded.

RESULTS

Publication history data were available for 55% (n = 8) of the journals sampled. The publication histories of 4,735 (25%) articles were analyzed. Mean time from: (1) author submission to journal acceptance was 180.93 days (SD = 103.89), (2) author submission to posting on PubMed was 263.55 days (SD = 157.61), and (3) journal acceptance to posting on PubMed was 83.15 days (SD = 135.72).

DISCUSSION

This study presents publication metadata for journals that openly provide it-a first step towards understanding publication timelines in HPE. Findings confirm the replicability of the original study, and the limited data suggest that, in comparison to biomedical scientists broadly, medical educators may experience longer wait times for article acceptance and publication. Reasons for these delays are currently unknown and deserve further study; such work would be facilitated by increased public access to journal metadata.

Social Media Skills for Professional Development in Psychiatry and Medicine

Social media use is increasing in the United States. Because psychiatrists and physicians are becoming more active online, Twitter is emerging as a leading platform for professional development. Social media can enhance networking, serve as a tool for mentoring trainees and colleagues, introduce journal articles to new readers, and allow psychiatrists and physicians to advocate for health care issues. Psychiatrists and physicians should observe ethical standards for digital citizenship on Twitter and other social media platforms.

Preprints: Facilitating early discovery, access, and feedback

None