Disciplinary differences of the impact of altmetric

Assessment of the dissemination of COVID-19 articles across social media: An Altmetrics Study

BACKGROUND

The use of social media assists in the distribution of information about COVID-19 to the general public and health professionals. Alternative-level metrics (Altmetrics) is an alternative method to traditional bibliometrics that assess the amount of sharing and spreading of a scientific article on social media platforms.

OBJECTIVE

Our objective was to characterize and compare traditional bibliometrics (citation-count) with newer metrics (Altmetric Attention Score) of the top 100 Altmetric scored COVID-19 articles.

METHODS

The 100 highest Altmetric Attention Score (AAS) articles were identified utilizing the Altmetric explorer in May 2020. AAS, journal name, and mentions from various social media databases (Twitter, Facebook, Wikipedia, Reddit, Mendeley, Dimension) of each article were collected. Citation-counts were collected from the Scopus database.

RESULTS

The median AAS and citation-count were 4922.50 and 24.00, respectively. Of 100 articles, The New England Journal of Medicine published the most articles at 18% (18/100). Twitter was the most frequently used social media platform with 96.3% of the mentions (985,429/1,022,975). Positive correlations were seen between AAS and citation-count (r2=.0973; P=.002).

CONCLUSIONS

Our research characterized the top 100 articles by AAS regarding COVID-19 in the Altmetric database. Altmetrics could complement with traditional citation-count when assessing the dissemination of an article regarding COVID-19.

INTERNATIONAL REGISTERED REPORT

RR2-10.2196/21408.

Evaluating Research Impact Based on Semantic Scholar Highly Influential Citations, Total Citations, and Altmetric Attention Scores: The Quest for Refined Measures Remains Illusive

Background: The evaluation of scholarly articles’ impact has been heavily based on the citation metrics despite the limitations of this approach. Therefore, the quest for meticulous and refined measures to evaluate publications’ impact is warranted. Semantic Scholar (SS) is an artificial intelligence-based database that allegedly identifies influential citations defined as “Highly Influential Citations” (HICs). Citations are considered highly influential according to SS when the cited publication has a significant impact on the citing publication (i.e., the citer uses or extends the cited work). Altmetrics are measures of online attention to research mined from activity in online tools and environments. Aims: The current study aimed to explore whether SS HICs provide an added value when it comes to measuring research impact compared to total citation counts and Altmetric Attention Score (AAS). Methods: Dimensions was used to generate the dataset for this study, which included COVID-19-related scholarly articles published by researchers affiliated to Jordanian institutions. Altmetric Explorer was selected as an altmetrics harvesting tool, while Semantic Scholar was used to extract details related to HICs. A total of 618 publications comprised the final dataset. Results: Only 4.57% (413/9029) of the total SS citations compiled in this study were classified as SS HICs. Based on SS categories of citations intent, 2626 were background citations (29.08%, providing historical context, justification of importance, and/or additional information related to the cited paper), 358 were result citations (3.97%, that extend on findings from research that was previously conducted), and 263 were method citations (2.91%, that use the previously established procedures or experiments to determine whether the results are consistent with findings in related studies). No correlation was found between HICs and AAS (r = 0.094). Manual inspection of the results revealed substantial contradictions, flaws, and inconsistencies in the SS HICs tool. Conclusions: The use of SS HICs in gauging research impact is significantly limited due to the enigmatic method of its calculation and total dependence on artificial intelligence. Along with the already documented drawbacks of total citation counts and AASs, continuous evaluation of the existing tools and the conception of novel approaches are highly recommended to improve the reliability of publication impact assessment.

Are patents linked on Twitter? A case study of Google patents

This study attempts to analyze patents as cited/mentioned documents to better understand the interest, dissemination and engagement of these documents in social environments, laying the foundations for social media studies of patents (social Patentometrics).Particularly, this study aims to determine how patents are disseminated on Twitter by analyzing three elements: tweets linking to patents, users linking to patents, and patents linked from Twitter. To do this, all the tweets containing at least one link to a full-text patent available on Google Patents were collected and analyzed, yielding a total of 126,815 tweets (and 129,001 links) to 86,417 patents. The results evidence an increase of the number of linking tweets over the years, presumably due to the creation of a standardized patent URL ID and the integration of Google Patents and Google Scholar, which took place in 2015. The engagement achieved by these tweets is limited (80.2% of tweets did not attract likes) but increasing notably since 2018. Two super-publisher twitter bot accounts (dailypatent and uspatentbot) are responsible of 53.3% of all the linking tweets, while most accounts are sporadic users linking to patent as part of a conversation. The patents most tweeted are, by far, from United States (87.5% of all links to Google Patents), mainly due to the effect of the two super-publishers. The impact of patents in terms of the number of tweets linking to them is unrelated to their year of publication, status or number of patent citations received, while controversial and media topics might be more determinant factors. However, further research is needed to better understand the topics discussed around patents on Twitter, the users involved, and the metrics attained. Given the increasing number of linking users and linked patents, this study finds Twitter as a relevant source to measure patent-level metrics, shedding light on the impact and interest of patents by the broad public.

Introducing the EMPIRE Index: A novel, value-based metric framework to measure the impact of medical publications

Article-level measures of publication impact (alternative metrics or altmetrics) can help authors and other stakeholders assess engagement with their research and the success of their communication efforts. The wide variety of altmetrics can make interpretation and comparative assessment difficult; available summary tools are either narrowly focused or do not reflect the differing values of metrics from a stakeholder perspective. We created the EMPIRE (EMpirical Publication Impact and Reach Evaluation) Index, a value-based, multi-component metric framework for medical publications. Metric weighting and grouping were informed by a statistical analysis of 2891 Phase III clinical trial publications and by a panel of stakeholders who provided value assessments. The EMPIRE Index comprises three component scores (social, scholarly, and societal impact), each incorporating related altmetrics indicating a different aspect of engagement with the publication. These are averaged to provide a total impact score and benchmarked so that a score of 100 equals the mean scores of Phase III clinical trial publications in the New England Journal of Medicine (NEJM) in 2016. Predictor metrics are defined to estimate likely long-term impact. The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore, with the societal impact component providing unique insights. Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not. Analysis of NEJM ‘notable articles’ showed that observational studies had the highest total impact and component scores, except for societal impact, for which surgical studies had the highest score. The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics and may enable medical researchers to assess the impact of publications easily and to understand what characterizes impactful research.

Assessing the Dissemination of COVID-19 Articles Across Social Media With Altmetric and PlumX Metrics: Correlational Study

BACKGROUND

The use of social media assists in the distribution of COVID-19 information to the general public and health professionals. Alternative-level metrics (ie, altmetrics) and PlumX metrics are new bibliometrics that can assess how many times a scientific article has been shared and how much a scientific article has spread within social media platforms.

OBJECTIVE

Our objective was to characterize and compare the traditional bibliometrics (ie, citation count and impact factors) and new bibliometrics (ie, Altmetric Attention Score [AAS] and PlumX score) of the top 100 COVID-19 articles with the highest AASs.

METHODS

The top 100 articles with highest AASs were identified with Altmetric Explorer in May 2020. The AASs, journal names, and the number of mentions in various social media databases of each article were collected. Citation counts and PlumX Field-Weighted Citation Impact scores were collected from the Scopus database. Additionally, AASs, PlumX scores, and citation counts were log-transformed and adjusted by +1 for linear regression, and Spearman correlation coefficients were used to determine correlations.

RESULTS

The median AAS, PlumX score, and citation count were 4922.50, 37.92, and 24.00, respectively. The New England Journal of Medicine published the most articles (18/100, 18%). The highest number of mentions (985,429/1,022,975, 96.3%) were found on Twitter, making it the most frequently used social media platform. A positive correlation was observed between AAS and citation count (r²=0.0973; P=.002), and between PlumX score and citation count (r²=0.8911; P<.001).

CONCLUSIONS

Our study demonstrated that citation count weakly correlated with AASs and strongly correlated with PlumX scores, with regard to COVID-19 articles at this point in time. Altmetric and PlumX metrics should be used to complement traditional citation counts when assessing the dissemination and impact of a COVID-19 article.

Assessment of Altmetrics and PlumX Metrics Scoring as Mechanisms to Evaluate the Top 100 Trending Hidradenitis Suppurativa Articles on Social Media: Cross-Sectional Study

Background

Dermatologists are increasingly utilizing social media platforms to disseminate scientific information. New tools, such as altmetrics and PlumX metrics, have been made available to rapidly capture the level of scientific article dissemination across social media platforms. However, no studies have been performed to assess the level of scientific article dissemination across social media regarding hidradenitis suppurativa, a disease that is still currently not well understood.

Objective

The aim of our study was to evaluate the utility of altmetrics and PlumX metrics by characterizing the top 100 “trending” hidradenitis suppurativa articles in the altmetric database by the altmetric attention score and PlumX score.

Methods

Altmetric data components of the top 100 hidradenitis suppurativa articles were extracted from the altmetric database. Article citation count was found using Web of Science. PlumX field-weighted impact scores for each article were collected from the Scopus database. Journal title, open-access status, article type, and study design of original articles were assessed. Additionally, the altmetric attention score, PlumX score, and citation count were log transformed and adjusted by +1 for linear regression, and Spearman correlation coefficients were utilized to determine correlations.

Results

Most of the top 100 “trending” hidradenitis suppurativa articles were published in JAMA Dermatology (n=27, 27%). The median altmetric attention score, PlumX score, and citation count were 25.5, 3.7, and 10.5, respectively. The most mentions regarding social media platforms came from Twitter. Although no correlation was observed between the citation count and altmetric attention score (r2=0.019, P=.17), positive correlation was observed between the citation count and PlumX score (r2=0.469, P<.001).

Conclusions

Our research demonstrated that citation count is not correlated with the altmetric attention score, but is strongly correlated with the PlumX score regarding hidradenitis suppurativa articles at this point in time. With the continual increase of social media usage by medical professionals and researchers, this study can help investigators understand the best way to captivate their audience.

Analysis of Altmetrics in Social Recognition of Neurology and Neurological Disorders

This study used Altmetric analysis to rank neurological articles and assessed the implications in relation to the social recognition of neurology and neurological disorders. An Altmetric Explorer search was conducted on 25 May 2018 for articles published in the 91 journals included in the 2015 InCites™ Journal Citation Report^®. We identified and analyzed the 100 articles with the highest Altmetric Attention Scores (AASs). A major proportion of the social impact (high AASs) was focused on neurodegenerative disorders such as dementia and neurodegenerative disorders. About half of the high-ranking articles provided academic information such as disease information (29 articles, 29%), new or advanced treatments (17%), and side effects of treatment (8%). The journal with largest number of top 100 articles was the New England Journal of Medicine (29 articles). Some of the data gathered via altmetrics can change a field of study, the public’s health, or a larger society. This is the first report on the impact of academic articles in neurological disorder on the general public living in our altered information society.

Crossref: The sustainable source of community-owned scholarly metadata

This paper describes the scholarly metadata collected and made available by Crossref, as well as its importance in the scholarly research ecosystem. Containing over 106 million records and expanding at an average rate of 11% a year, Crossref’s metadata has become one of the major sources of scholarly data for publishers, authors, librarians, funders, and researchers. The metadata set consists of 13 content types, including not only traditional types, such as journals and conference papers, but also data sets, reports, preprints, peer reviews, and grants. The metadata is not limited to basic publication metadata, but can also include abstracts and links to full text, funding and license information, citation links, and the information about corrections, updates, retractions, etc. This scale and breadth make Crossref a valuable source for research in scientometrics, including measuring the growth and impact of science and understanding new trends in scholarly communications. The metadata is available through a number of APIs, including REST API and OAI-PMH. In this paper, we describe the kind of metadata that Crossref provides and how it is collected and curated. We also look at Crossref’s role in the research ecosystem and trends in metadata curation over the years, including the evolution of its citation data provision. We summarize the research used in Crossref’s metadata and describe plans that will improve metadata quality and retrieval in the future.