General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators

Misinformation About Orthopaedic Conditions on Social Media: Analysis of TikTok and Instagram

Introduction Social media outlets such as TikTok (TT) and Instagram (IG) have surged as a method to disseminate information. More recently, healthcare professionals have targeted this space as a means to provide medical education and advice. With the ever-growing content on these applications, there is significant variability and quality of material available, which can lead to the dissemination of misinformation. This study aims to evaluate the accuracy and popularity of content on common orthopaedic pathology on TT and IG. Methods Content on TT and IG related to six common orthopaedic conditions - achilles tendon tear, ACL tear, meniscus tear, tennis elbow, rotator cuff tear, and ankle sprains - was evaluated between April and June 2022. The top ten posts for the top two associated hashtags for each condition were reviewed. The quality of each post was analyzed using the DISCERN instrument, rating each on a scale of 1 to 5. Each post was characterized by the author's profession (physician, physical therapist, chiropractor, etc.) and content type (educational, testimonial, personal, promotional, and entertainment). Popularity and engagement metrics such as "comments," "likes," and "shares" were also collected. Results There were 165,666,490 views on TT and 9,631,015 views on IG amongst the six common aforementioned orthopaedic conditions. Content created by physicians had less overall engagement (16.1%) compared to content created by non-physicians (83.9%). The quality of content on average was low (mean misinformation index 2.04 ± 1.08 (1-5)1. Physician-created posts in comparison to non-physician posts were significantly more accurate (mean misinformation index score 3.38 ± 1.12 vs 1.89 ± 0.94, p<0.0001). Conclusions Common orthopaedic conditions such as Achilles tendon tears, ACL tears, and meniscus tears are frequently the focus of content posted on TT and IG; however, this information is often not medically accurate. Increased physician engagement may help to rectify this misinformation.

Evaluative altmetrics: is there evidence for its application to research evaluation?

Introduction

Altmetrics have been demonstrated as a promising tool for analyzing scientific communication on social media. Nevertheless, its application for research evaluation remains underdeveloped, despite the advancement of research in the study of diverse scientific interactions.

Methods

This paper develops a method for applying altmetrics in the evaluation of researchers, focusing on a case study of the Environment/Ecology ESI field publications by researchers at the University of Granada. We considered Twitter as a mirror of social attention, news outlets as media, and Wikipedia as educational, exploring mentions from these three sources and the associated actors in their respective media, contextualizing them using various metrics.

Results

Our analysis evaluated different dimensions such as the type of audience, local attention, engagement generated around the mention, and the profile of the actor. Our methodology effectively provided dashboards that gave a comprehensive view of the different instances of social attention at the author level.

Discussion

The use of altmetrics for research evaluation presents significant potential, as shown by our case study. While this is a novel method, our results suggest that altmetrics could provide valuable insights into the social attention that researchers garner. This can be an important tool for research evaluation, expanding our understanding beyond traditional metrics.

Measuring altmetric events: the need for longer observation period and article level computations

Purpose

This paper aims to explore the impact of the data observation period on the computation of altmetric measures like velocity index (VI) and half-life. Furthermore, it also attempts to determine whether article-level computations are better than computations on the whole of the data for computing such measures.

Design/methodology/approach

The complete publication records for the year 2016 indexed in Web of Science and their altmetric data (original tweets) obtained from PlumX are obtained and analysed. The creation date of articles is taken from Crossref. Two time-dependent variables, namely, half-life and VI are computed. The altmetric measures are computed for all articles at different observation points, and by using whole group as well as article-level averaging.

Findings

The results show that use of longer observation period significantly changes the values of different altmetric measures computed. Furthermore, use of article-level delineation is advocated for computing different measures for a more accurate representation of the true values for the article distribution.

Research limitations/implications

The analytical results show that using different observation periods change the measured values of the time-related altmetric measures. It is suggested that longer observation period should be used for appropriate measurement of altmetric measures. Furthermore, the use of article-level delineation for computing the measures is advocated as a more accurate method to capture the true values of such measures.

Practical implications

The research work suggests that altmetric mentions accrue for a longer period than the commonly believed short life span and therefore the altmetric measurements should not be limited to observation of early accrued data only.

Social implications

The present study indicates that use of altmetric measures for research evaluation or other purposes should be based on data for a longer observation period and article-level delineation may be preferred. It contradicts the common belief that tweet accumulation about scholarly articles decay quickly.

Originality/value

Several studies have shown that altmetric data correlate well with citations and hence early altmetric counts can be used to predict future citations. Inspired by these findings, majority of such monitoring and measuring exercises have focused mainly on capturing immediate altmetric event data for articles just after the publication of the paper. This paper demonstrates the impact of the observation period and article-level aggregation on such computations and suggests to use a longer observation period and article-level delineation. To the best of the authors’ knowledge, this is the first such study of its kind and presents novel findings.

Evaluating the online impact of reporting guidelines for randomised trial reports and protocols: a cross-sectional web-based data analysis of CONSORT and SPIRIT initiatives

Reporting guidelines are tools to help improve the transparency, completeness, and clarity of published articles in health research. Specifically, the CONSORT (Consolidated Standards of Reporting Trials) and SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) statements provide evidence-based guidance on what to include in randomised trial articles and protocols to guarantee the efficacy of interventions. These guidelines are subsequently described and discussed in journal articles and used to produce checklists. Determining the online impact (i.e., number and type of links received) of these articles can provide insights into the dissemination of reporting guidelines in broader environments (web-at-large) than simply that of the scientific publications that cite them. To address the technical limitations of link analysis, here the Debug-Validate-Access-Find (DVAF) method is designed and implemented to measure different facets of the guidelines' online impact. A total of 65 articles related to 38 reporting guidelines are taken as a baseline, providing 240,128 URL citations, which are then refined, analysed, and categorised using the DVAF method. A total of 15,582 links to journal articles related to the CONSORT and SPIRIT initiatives were identified. CONSORT 2010 and SPIRIT 2013 were the reporting guidelines that received most links (URL citations) from other online objects (5328 and 2190, respectively). Overall, the online impact obtained is scattered (URL citations are received by different article URL IDs, mainly from link-based DOIs), narrow (limited number of linking domain names, half of articles are linked from fewer than 29 domain names), concentrated (links come from just a few academic publishers, around 60% from publishers), non-reputed (84% of links come from dubious websites and fake domain names) and highly decayed (89% of linking domain names were not accessible at the time of the analysis). In light of these results, it is concluded that the online impact of these guidelines could be improved, and a set of recommendations are proposed to this end.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11192-022-04542-z.

From Research Evaluation to Research Analytics. The digitization of academic performance measurement

One could think that bibliometric measurement of academic performance has always been digital since the computer-assisted invention of the Science Citation Index. Yet, since the 2000s, the digitization of bibliometric infrastructure has accelerated at a rapid pace. Citation databases are indexing an increasing variety of publication types. Altmetric data aggregators are producing data on the reception of research outcomes. Machine-readable persistent identifiers are created to unambiguously identify researchers, research organizations, and research objects; and evaluative software tools and current research information systems are constantly enlarging their functionalities to make use of these data and extract meaning from them. In this article, we analyse how these developments in evaluative bibliometrics have contributed to an extension of indicator-based research evaluation towards data-driven research analytics. Drawing on empirical material from blogs and websites as well as from research and policy papers, we discuss how interoperability, scalability, and flexibility as material specificities of digital infrastructures generate new ways of data production and their assessment, which affect the possibilities of how academic performance can be understood and (e)valuated.

Can we use link-based indicators to find highly cited publications? The case of the Trust Flow score

The Majestic’s Trust Flow (TF) is a link-based score aimed at measuring the influence of online objects (e.g. scientific publications) by considering the weighted number of links received from trusted websites. This study describes the bibliographic characteristics and impact of those publications with the highest TF score. In order to do this, 20,810 URL-based Digital Object Identifiers (DOIs) were identified and analysed. The results show that these DOIs mainly represent recent publications (57.1% of publications were published between 2010 and 2020), journal articles (93.75%) published in the first SCImago Journal Rank (SJR) quartile (81.7%), written with international collaboration (40.4%) and biased towards the field of medicine (36.9%). While the TF score is a discovering tool with the potential to be used in webometric studies to find influential publications, a few technical limitations jeopardise the general applicability of this indicator for research evaluation at the publication level.

The role of blogs and news sites in science communication during the COVID-19 pandemic

We present a brief review of literature related to blogs and news sites; our focus is on publications related to COVID-19. We primarily focus on the role of blogs and news sites in disseminating research on COVID-19 to the wider public, that is knowledge transfer channels. The review is for researchers and practitioners in scholarly communication and social media studies of science who would like to find out more about the role of blogs and news sites during the COVID-19 pandemic. From our review, we see that blogs and news sites are widely used as scholarly communication channels and are closely related to each other. That is, the same research might be reported in blogs and news sites at the same time. They both play a particular role in higher education and research systems, due to the increasing blogging and science communication activity of researchers and higher education institutions (HEIs). We conclude that these two media types have been playing an important role for a long time in disseminating research, which even increased during the COVID-19 pandemic. This can be verified, for example, through knowledge graphs on COVID-19 publications that contain a significant amount of scientific publications mentioned in blogs and news sites.

A value creation model from science-society interconnections: Archetypal analysis combining publications, survey and altmetric data

The interplay between science and society takes place through a wide range of intertwined relationships and mutual influences that shape each other and facilitate continuous knowledge flows. Stylised consequentialist perspectives on valuable knowledge moving from public science to society in linear and recursive pathways, whilst informative, cannot fully capture the broad spectrum of value creation possibilities. As an alternative we experiment with an approach that gathers together diverse science-society interconnections and reciprocal research-related knowledge processes that can generate valorisation. Our approach to value creation attempts to incorporate multiple facets, directions and dynamics in which constellations of scientific and societal actors generate value from research. The paper develops a conceptual model based on a set of nine value components derived from four key research-related knowledge processes: production, translation, communication, and utilization. The paper conducts an exploratory empirical study to investigate whether a set of archetypes can be discerned among these components that structure science-society interconnections. We explore how such archetypes vary between major scientific fields. Each archetype is overlaid on a research topic map, with our results showing the distinctive topic areas that correspond to different archetypes. The paper finishes by discussing the significance and limitations of our results and the potential of both our model and our empirical approach for further research.

Introducing the EMPIRE Index: A novel, value-based metric framework to measure the impact of medical publications

Article-level measures of publication impact (alternative metrics or altmetrics) can help authors and other stakeholders assess engagement with their research and the success of their communication efforts. The wide variety of altmetrics can make interpretation and comparative assessment difficult; available summary tools are either narrowly focused or do not reflect the differing values of metrics from a stakeholder perspective. We created the EMPIRE (EMpirical Publication Impact and Reach Evaluation) Index, a value-based, multi-component metric framework for medical publications. Metric weighting and grouping were informed by a statistical analysis of 2891 Phase III clinical trial publications and by a panel of stakeholders who provided value assessments. The EMPIRE Index comprises three component scores (social, scholarly, and societal impact), each incorporating related altmetrics indicating a different aspect of engagement with the publication. These are averaged to provide a total impact score and benchmarked so that a score of 100 equals the mean scores of Phase III clinical trial publications in the New England Journal of Medicine (NEJM) in 2016. Predictor metrics are defined to estimate likely long-term impact. The social impact component correlated strongly with the Altmetric Attention Score and the scholarly impact component correlated modestly with CiteScore, with the societal impact component providing unique insights. Analysis of fresh metrics collected 1 year after the initial dataset, including an independent sample, showed that scholarly and societal impact scores continued to increase, whereas social impact scores did not. Analysis of NEJM ‘notable articles’ showed that observational studies had the highest total impact and component scores, except for societal impact, for which surgical studies had the highest score. The EMPIRE Index provides a richer assessment of publication value than standalone traditional and alternative metrics and may enable medical researchers to assess the impact of publications easily and to understand what characterizes impactful research.

Social media usage to share information in communication journals: An analysis of social media activity and article citations

Social media has surrounded every area of life, and social media platforms have become indispensable for today’s communication. Many journals use social media actively to promote and disseminate new articles. Its use to share the articles contributes many benefits, such as reaching more people and spreading information faster. However, there is no consensus in the studies that to evaluate between tweeted and non-tweeted papers regarding their citation numbers. Therefore, it was aimed to show the effect of social media on the citations of articles in the top ten communication-based journals. For this purpose, this work evaluated original articles published in the top 10 communication journals in 2018. The top 10 communication-based journals were chosen based on SCImago Journal & Country Rank (cited in 2019). Afterward, it was recorded the traditional citation numbers (Google Scholar and Thompson-Reuters Web of Science) and social media exposure of the articles in January 2021 (nearly three years after the articles’ publication date). It was assumed that this period would allow the impact of the published articles (the citations and Twitter mentions) to be fully observed. Based on this assessment, a positive correlation between exposure to social media and article citations was observed in this study.

“I updated the <ref>”: The evolution of references in the English Wikipedia and the implications for altmetrics

With this work, we present a publicly available data set of the history of all the references (more than 55 million) ever used in the English Wikipedia until June 2019. We have applied a new method for identifying and monitoring references in Wikipedia, so that for each reference we can provide data about associated actions: creation, modifications, deletions, and reinsertions. The high accuracy of this method and the resulting data set was confirmed via a comprehensive crowdworker labeling campaign. We use the data set to study the temporal evolution of Wikipedia references as well as users’ editing behavior. We find evidence of a mostly productive and continuous effort to improve the quality of references: There is a persistent increase of reference and document identifiers (DOI, PubMedID, PMC, ISBN, ISSN, ArXiv ID) and most of the reference curation work is done by registered humans (not bots or anonymous editors). We conclude that the evolution of Wikipedia references, including the dynamics of the community processes that tend to them, should be leveraged in the design of relevance indexes for altmetrics, and our data set can be pivotal for such an effort.

Research articles promoted in embargo e-mails receive higher citations and altmetrics

In order to be able to provide thorough and timely coverage on the most recent scientific research, science journalists frequently rely on embargoed information sent to them by publishers of scientific journals. In such embargo e-mails, publishers purposefully bring selected upcoming releases to the journalists’ attention a few days in advance of their publication. Little is known on how this early highlighting of certain research articles affects their later citations or altmetrics. We present an exploratory case study with the aim of assessing the effects of such promotion activities on scientific articles’ bibliometric and altmetric indicators. In a treatment–control design, we analyze citation counts and eight types of altmetrics of 715 articles published between 2016 and 2017 whose DOIs have been mentioned in embargo e-mails and compare these to articles from the same journal issues that have not been highlighted in embargo e-mails. Descriptive statistics and Mann–Whitney-U tests reveal significant advantages for promoted articles across all regarded metrics three to four years after their publication. Particularly large differences can be seen regarding numbers of mentions in mainstream media, in blogs, on Twitter, and on Facebook. Our findings suggest that scholarly publishers exert significant influence over which research articles will receive attention and visibility in various (social) media. Also, regarding utilizations of metrics for evaluative purposes, the observed effects of promotional activities on indicators might constitute a factor of undesirable influence that currently does not receive the amount of consideration in scientometric assessments that it should receive.

Identifying and characterizing social media communities: a socio-semantic network approach to altmetrics

Altmetric indicators allow exploring and profiling individuals who discuss and share scientific literature in social media. But it is still a challenge to identify and characterize communities based on the research topics in which they are interested as social and geographic proximity also influence interactions. This paper proposes a new method which profiles social media users based on their interest on research topics using altmetric data. Social media users are clustered based on the topics related to the research publications they share in social media. This allows removing linkages which respond to social or personal proximity and identifying disconnected users who may have similar research interests. We test this method for users tweeting publications from the fields of Information Science & Library Science, and Microbiology. We conclude by discussing the potential application of this method and how it can assist information professionals, policy managers and academics to understand and identify the main actors discussing research literature in social media.

Examining the Relationship between Altmetric Score and Traditional Bibliometrics in the Ophthalmology Literature for 2013 and 2016 Cohorts

Background In this study, we reviewed a select sample of ophthalmology literature to determine if there was a correlation between Altimetric and traditional citation-based and impact factor metrics. We hypothesized that Altmetric score would more closely correlate with impact factor and citations in 2016.

Methods Journal Citation Reports for the year 2013 was used to find the 15 highest impact factor ophthalmology journals in 2013. Then Elsevier's Scopus was used to identify the 10 most cited articles from each journal for the years 2013 and 2016. Metrics for all identified articles were collected using the Altmetric Bookmarklet, and date of Twitter account creation was noted for journals with such an account. Altmetric scores, impact factor, and citation counts were tabulated for each article. Pearson's correlation coefficient (r) determined correlation of independent variables (number of citations or impact factor) with dependent variable (Altmetric score). For our Twitter analysis, account age was the independent variable and calculated correlation coefficients (r) were the dependent variable. Proportion of variance was determined with a coefficient of determination (R 2).

Results This study included 300 articles, evenly split between 2013 and 2016. Within the 2013 cohort, three journals had significant positive correlations between citation count and Altmetric score. For the 2016 cohort, both Altmetric score and citation count (r = 0.583, p < 0.001) and Altmetric score and impact factor (r = 0.183, p = 0.025) revealed significant positive correlations. In 2016, two journals were found to have significant correlations between Altmetric score and citation number. Neither year revealed a significant correlation between the age of a journal's Twitter profile and the relationship between Altmetric score and citation count. In each year, Twitter accounted for the highest number of mentions.

Conclusion The findings suggest that correlation between Altmetric score and traditional quality metric scores may be increasing. Altmetric score was correlated with impact factor and number of citations in 2016 but not 2013. At this time, Altmetrics are best used as an adjunct that is complementary but not an alternative to traditional bibliometrics for assessing academic productivity and impact.

Bibliometric Reports for Institutions: Best Practices in a Responsible Metrics Scenario

Carrying out bibliometric reports is one of the common tasks performed by librarians and practitioners within the framework of their professional duties. The emergence of novel data sources, the need to measure new research activities and the growing demand for fairer and more equitable evaluation within the framework of the Responsible Metrics movement has led to calls for a review of the traditional approaches to these types of reports. The main goal of this study is to outline a series of recommendations for bibliometricians, consultants and research support librarians when drafting bibliometric reports in their institutions. These best practices can significantly enhance the quality and utility of bibliometric reports, posing their practitioners as key players in the science management process.

Analyzing Social Media Research: A Data Quality and Research Reproducibility Perspective

Social media platforms have become very popular these days among individuals and organizations. On the one hand, organizations use social media as a potential tool to create awareness of their products among consumers, and on the other hand, social media data is useful to predict the national crisis, election polls, stock prediction, etc. However, nowadays, a debate is going on about the quality of data generated on social media platforms, whether it is relevant for prediction and generalization. The article discusses the relevance and quality of data obtained from social media in the context of research and development. Social media data quality issues may impact the generalizability and reproducibility of the results of the study. The paper explores possible reasons for quality issues in the data generated over social media platforms along with the suggestive measures to minimize them using the proposed social media data quality framework.

Analysis of trending topics in glaucoma articles from an altmetric perspective

PURPOSE

Altmetric analyses are a new way of assessing and sharing scientific knowledge. Traditional metrics and altmetric analyses highlight key publications. The primary objective of this study was to evaluate the social attention paid to highly cited articles related to glaucoma in the recent English literature and compare with traditional citation metrics.

MATERIALS AND METHODS

"Glaucoma" was entered as a search term into Thomson Reuter's Web of Science database, and all articles related to the topic in the last decade were identified. The 50 highly cited articles (T50 list) were analyzed by topic, journal name, author name, year of the publication and Altmetric Attention Score (AAS). Descriptive statistics and Spearman correlation test were determined with the use of SPSS.

RESULTS

According to bibliometric criteria, there were 31,370 eligible articles and the median (range) citation number was recorded as 181.5 (158.75-250.75). The T50 list was ranked with AASs between 176 and 0. The median AAS was 5 (2.75-10). The main subjects of the top 10 highly cited articles were mostly related to follow-up and prognostics about glaucoma (n = 3), while the main subjects of the top 10 articles with the highest AAS were related to genetics in glaucoma pathogenesis (n = 2), treatment modalities (n = 2) and pathophysiology with therapeutics of glaucoma disease (n = 2). AASs and citation number showed a positive moderate correlation (r = 0.403 p = 0.004), although AASs did not correlate with journal impact factor (r = 0.36 p = 0.01). No statistically significant correlation was found for ASSs and citation numbers with H-index of the journals on the T50 list.

CONCLUSIONS

Bibliometric-based altmetric analyses offer important but different perspectives regarding article impact. This study provides valuable information about trending topics related to glaucoma research and its impact in both the academic literature and social media CLINICAL TRIAL REGISTRATION: With regard to the data characteristics of the manuscript, which is mainly retrospective and international, the clinical trial registration process is theoretically not applicable to this study.

The Top 100 Spine Surgery Articles on Social Media: An Altmetric Study

STUDY DESIGN

Literature review.

OBJECTIVE

To discern which social media outlets contribute most to spine surgery literature dissemination and to investigate how popular articles compare to articles with most citations.

SUMMARY OF BACKGROUND DATA

Scientific literature is increasingly disseminated through social media. The Altmetric Attention Score (AAS) is defined as an automated, weighted score calculation from counts of all online attention received by various research outputs. Increasing AAS values indicate more online attention. For example, the overall top 100 Altmetric spine surgery peer-reviewed articles since 2010 had an AAS range from 78 to 1537. Among all spine surgery literature reviewed since 2010, the mean AAS was 5.3 with a median of 1.0.

METHODS

We performed an Altmetric database search of nine spine surgery journals from January 2010 to October 2019. Mean AAS was summarized alongside metrics including citation count and impact factor. We assessed correlations between AAS and online sources, readers, and citations. Journals were grouped by impact factor, and analysis-of-variance compared mean AAS. The 100 highest AAS articles were compared to the most cited.

RESULTS

13,601 articles were included. The mean AAS was 5.3, with Twitter contributing the most. The three highest associations were news (P < 0.001), Twitter (P < 0.001), and Facebook (P < 0.001). There was no significant association between impact factor and AAS. Compared with the most cited articles, the top 100 AAS articles had significantly more article types, more prospective studies, fewer retrospective studies, fewer reviews, and fewer systematic reviews (P < 0.001 for all). Spine contributed the most articles in both top 100 sets.

CONCLUSION

Our evaluation revealed Twitter, newsfeeds, and Facebook were the most significant social media outlets. Compared to articles with the most citations, the most popular articles are prospective and encompass broader study designs. Social media plays an integral role in dissemination, both within spine literature and the public sphere.

LEVEL OF EVIDENCE

3.

Sentiment analysis of tweets through Altmetrics: A machine learning approach

The purpose of the study is to (a) contribute to annotating an Altmetrics dataset across five disciplines, (b) undertake sentiment analysis using various machine learning and natural language processing–based algorithms, (c) identify the best-performing model and (d) provide a Python library for sentiment analysis of an Altmetrics dataset. First, the researchers gave a set of guidelines to two human annotators familiar with the task of related tweet annotation of scientific literature. They duly labelled the sentiments, achieving an inter-annotator agreement (IAA) of 0.80 (Cohen’s Kappa). Then, the same experiments were run on two versions of the dataset: one with tweets in English and the other with tweets in 23 languages, including English. Using 6388 tweets about 300 papers indexed in Web of Science, the effectiveness of employed machine learning and natural language processing models was measured by comparing with well-known sentiment analysis models, that is, SentiStrength and Sentiment140, as the baseline. It was proved that Support Vector Machine with uni-gram outperformed all the other classifiers and baseline methods employed, with an accuracy of over 85%, followed by Logistic Regression at 83% accuracy and Naïve Bayes at 80%. The precision, recall and F1 scores for Support Vector Machine, Logistic Regression and Naïve Bayes were (0.89, 0.86, 0.86), (0.86, 0.83, 0.80) and (0.85, 0.81, 0.76), respectively.

How much research shared on Facebook happens outside of public pages and groups? A comparison of public and private online activity around PLOS ONE papers

Despite its undisputed position as the biggest social media platform, Facebook has never entered the main stage of altmetrics research. In this study, we argue that the lack of attention by altmetrics researchers is due, in part, to the challenges in collecting Facebook data regarding activity that takes place outside of public pages and groups. We present a new method of collecting aggregate counts of shares, reactions, and comments across the platform—including users’ personal timelines—and use it to gather data for all articles published between 2015 to 2017 in the journal PLOS ONE. We compare the gathered data with altmetrics collected and aggregated by Altmetric. The results show that 58.7% of papers shared on Facebook happen outside of public spaces and that, when collecting all shares, the volume of activity approximates patterns of engagement previously only observed for Twitter. Both results suggest that the role and impact of Facebook as a medium for science and scholarly communication has been underestimated. Furthermore, they emphasize the importance of openness and transparency around the collection and aggregation of altmetrics.

A Bibliometric Analysis Using Alternative Metrics for Articles in the Annals of Rehabilitation Medicine

OBJECTIVE

To investigate the articles in the Annals of Rehabilitation Medicine (ARM) using a bibliometric analysis to verify whether there is a correlation between the topics of interest for expert groups and the public media.

METHODS

A total of 1,088 ARM articles from the third issue of 2011 to the third issue of 2019 were analyzed. We conducted a bibliometric analysis of the articles using conventional metrics (CM) and alternative metrics (AM). The CM was investigated by collating the type of publication, number of citations, and the specific field of rehabilitation medicine for each article. The AM was analyzed using the Altmetric Attention Score (AAS) provided by Altmetric, the leading AM company. The correlation between the number of citations and the AAS was tested using the Spearman rank correlation coefficient.

RESULTS

The combined ratio of original articles and case reports was over 90% in this study; however, the total distribution was significantly different compared to previous bibliometric studies (p<0.05). There were 233 articles that satisfied both conditions of at least one citation and at least one AAS point. The number of citations and the AAS were found to have a statistically significant positive linear correlation on a scatter plot (r=0.216, p=0.001).

CONCLUSION

There is a significant correlation between AM and CM, which means itis important to increase the dissemination of academic knowledge through the public media and increase the status of the journal by increasing the citation-related index.

Can altmetrics predict future citation counts in critical care medicine publications?

Introduction

Social media is increasingly used in the dissemination of medical research. Traditional measures of the impact of a paper do not account for this. Altmetrics are a measure of the dissemination of a publication via social media websites. The purpose of this study is to ascertain if the altmetric attention score of an article is a reliable measure of the impact it has in the field of critical care medicine. To this end, we investigated if a correlation exists between future citation count and altmetric attention score.

Methods

The top nine journals by impact factor in the field of critical care medicine were identified for 2014 and 2015. The 100 most cited articles from these journals were recorded to form the Scientific Impact Group, i.e. those with the greatest impact on the scientific community. The altmetric attention score was recorded for each article. The top 100 articles by altmetric attention score were also identified to form the Media Impact Group, i.e. those that generated the most online attention. Their citation counts' were recorded. Statistical analysis was performed on each group to identify a correlation between altmetric attention score and citation count.

Results

There was a moderately positive correlation in the Scientific Impact Group, with a Spearman r score of 0.4336 (P = 0.0001). A weakly positive correlation was found in the Media Impact Group, with a Spearman r score of 0.3033 (P = 0.002).

Conclusions

There is a positive correlation between traditional bibliographic metrics and altmetrics in the field of critical care medicine. Highly cited papers are more likely to generate online attention. However, papers that generate a lot of online attention are less likely to have a high citation count. Therefore, altmetric attention score is not a reliable predictor of future citation count in critical care medicine.

Crossref: The sustainable source of community-owned scholarly metadata

This paper describes the scholarly metadata collected and made available by Crossref, as well as its importance in the scholarly research ecosystem. Containing over 106 million records and expanding at an average rate of 11% a year, Crossref’s metadata has become one of the major sources of scholarly data for publishers, authors, librarians, funders, and researchers. The metadata set consists of 13 content types, including not only traditional types, such as journals and conference papers, but also data sets, reports, preprints, peer reviews, and grants. The metadata is not limited to basic publication metadata, but can also include abstracts and links to full text, funding and license information, citation links, and the information about corrections, updates, retractions, etc. This scale and breadth make Crossref a valuable source for research in scientometrics, including measuring the growth and impact of science and understanding new trends in scholarly communications. The metadata is available through a number of APIs, including REST API and OAI-PMH. In this paper, we describe the kind of metadata that Crossref provides and how it is collected and curated. We also look at Crossref’s role in the research ecosystem and trends in metadata curation over the years, including the evolution of its citation data provision. We summarize the research used in Crossref’s metadata and describe plans that will improve metadata quality and retrieval in the future.

Top 100 Publications as Measured by Altmetrics in the Field of Central Nervous System Inflammatory Demyelinating Disease

Background

Altmetrics analyze the visibility of articles in social media and estimate their impact on the general population. We performed an altmetric analysis of articles on central nervous system inflammatory demyelinating disease (CIDD) and investigated its correlation with citation analysis.

Methods

Articles in the 91 journals comprising the “clinical neurology,” “neuroscience,” and “medicine, general, and internal” Web of Science categories were searched for their relevance to the CIDD topic. The Altmetric Explorer database was used to determine the Altmetric.com Attention Score (AAS) values of the selected articles. The papers with the top 100 AAS values were characterized.

Results

Articles most frequently mentioned online were primarily published after 2014 and were published in journals with high impact factors. All articles except one were dealt with the issue of multiple sclerosis. Most were original articles, but editorials were also common. Novel treatments and risk factors are the most frequent topics. The AAS was weakly correlated with journal impact factors; however, no link was found between the AAS and the number of citations.

Conclusions

We present the top 100 most frequently mentioned CIDD articles in online media using an altmetric approach. Altmetrics can rapidly offer alternative information on the impact of research based on a broader audience and can complement traditional metrics.

Altmetrics Analysis of Archivos de Bronconeumología From 2014 to 2018

INTRODUCTION

Alternative metrics or altmetrics are non-traditional measurements of scientific production that reflect a publication's influence in social networks and similar channels of dissemination. The aim of this study was to analyze the media impact of Archivos de Bronconeumología according to 2 altmetric aggregators and website visits.

METHODS

This was an observational study of the original articles and review and consensus articles published in Archivos de Bronconeumología during the period 2014-2018. Data from the PlumX Metrics and Altmetric aggregators and visits to the Archivos de Bronconeumología website were analyzed. Five comparisons were made: by specialty area, by funding received, by number of participating centers, by document type, and by topic. In a subanalysis, altmetrics were correlated with the conventional citation system.

RESULTS

We analyzed 273 papers, of which 186 were original articles (68.1%). The papers that achieved greater media impact in the 2 aggregators analyzed, and in terms of website visits, were pulmonology papers and review and consensus articles. The mean Altmetric Attention Score was 1.9±4.4 (range 0-59), which is above average for the date of publication of the paper. A statistically significant weak to moderate correlation was identified between altmetrics and conventional citations.

CONCLUSIONS

Review articles, consensus documents, and pulmonology papers had a greater media impact. Mean Altmetric Attention Score was higher than the average based on the date of publication. A weak to moderate correlation between altmetrics and conventional citations was identified.

Comparing the topological rank of journals in Web of Science and Mendeley

Recently, there has been a surge of interest in new data emerged due to the rapid development of the information technologies in scholarly communication. Since the 2010s, altmetrics has become a common trend in scientometric research. However, researchers have not treated in much detail the question of the probability distributions underlying these new data. The principal objective of this study was to investigate one of the classic problems of scientometrics-the problem of citation and readership distributions. The study is based on the data obtained from two information systems: Web of Science and Mendeley. Here we based on the concept of the cumulative empirical distribution function to explore the differences and similarities between citations and readership counts of biological journals indexed in Web of Science and Mendeley. The basic idea was to determine, for any journal, a "size" (it is said to be the topological rank) of citation and readership empirical cumulative distributions, and then to compare distributions of the topological ranks of Web of Science and Mendeley. In order to verify our model, we employ it to the bibliometric and altmetric research of 305 biological journals indexed in Journal Citation Reports 2015. The findings show that both distributions of the topological rank of biological journals are statistically close to the Wakeby distribution. The findings presented in this study add to our understanding of information processes of the scholarly communication in the new digital environment.

Towards a second generation of 'social media metrics': Characterizing Twitter communities of attention around science

‘Social media metrics’ are bursting into science studies as emerging new measures of impact related to scholarly activities. However, their meaning and scope as scholarly metrics is still far from being grasped. This research seeks to shift focus from the consideration of social media metrics around science as mere indicators confined to the analysis of the use and visibility of publications on social media to their consideration as metrics of interaction and circulation of scientific knowledge across different communities of attention, and particularly as metrics that can also be used to characterize these communities. Although recent research efforts have proposed tentative typologies of social media users, no study has empirically examined the full range of Twitter user’s behavior within Twitter and disclosed the latent dimensions in which activity on Twitter around science can be classified. To do so, we draw on the overall activity of social media users on Twitter interacting with research objects collected from the Altmetic.com database. Data from over 1.3 million unique users, accounting for over 14 million tweets to scientific publications, is analyzed. Based on an exploratory and confirmatory factor analysis, four latent dimensions are identified: ‘Science Engagement’, ‘Social Media Capital’, ‘Social Media Activity’ and ‘Science Focus’. Evidence on the predominant type of users by each of the four dimensions is provided by means of VOSviewer term maps of Twitter profile descriptions. This research breaks new ground for the systematic analysis and characterization of social media users’ activity around science.

Rxivist.org: Sorting biology preprints using social media and readership metrics

Preprints have arrived. In increasing numbers, researchers across the life sciences are embracing the once-niche practice, shaking off decades of reluctance and posting hundreds of papers per week to preprint servers, sharing their findings with the community before embarking on the weary march through peer review. However, there are limited methods for individuals sifting through this avalanche of research to identify the preprints that are most relevant to their interests. Here, we describe Rxivist.org, a website that indexes all preprints posted to bioRxiv.org, the largest preprint server in the life sciences, and allows users to filter and sort papers based on download metrics and Twitter activity over a variety of categories and time periods. In this work, we hope to make it easier for readers to find relevant research on bioRxiv and to improve the visibility of preprints currently being read and discussed online.

This Community Page article describes Rxivist.org, a new website that indexes all preprints posted to bioRxiv.org and allows users to filter and sort papers based on download metrics and Twitter activity over a variety of categories and time periods.

Exploratory analysis of Publons metrics and their relationship with bibliometric and altmetric impact

Purpose

The purpose of this paper is to analyse the metrics provided by Publons about the scoring of publications and their relationship with impact measurements (bibliometric and altmetric indicators).

Design/methodology/approach

In January 2018, 45,819 research articles were extracted from Publons, including all their metrics (scores, number of pre and post reviews, reviewers, etc.). Using the DOI identifier, other metrics from altmetric providers were gathered to compare the scores of those publications in Publons with their bibliometric and altmetric impact in PlumX, Altmetric.com and Crossref Event Data.

Findings

The results show that: there are important biases in the coverage of Publons according to disciplines and publishers; metrics from Publons present several problems as research evaluation indicators; and correlations between bibliometric and altmetric counts and the Publons metrics are very weak (r<0.2) and not significant.

Originality/value

This is the first study about the Publons metrics at article level and their relationship with other quantitative measures such as bibliometric and altmetric indicators.

The insoluble problems of books: what does Altmetric.com have to offer?

Purpose

The purpose of this paper is to analyze the capabilities, functionalities and appropriateness of Altmetric.com as a data source for the bibliometric analysis of books in comparison to PlumX.

Design/methodology/approach

The authors perform an exploratory analysis on the metrics the Altmetric Explorer for Institutions, platform offers for books. The authors use two distinct data sets of books. On the one hand, the authors analyze the Book Collection included in Altmetric.com. On the other hand, the authors use Clarivate’s Master Book List, to analyze Altmetric.com’s capabilities to download and merge data with external databases. Finally, the authors compare the findings with those obtained in a previous study performed in PlumX.

Findings

Altmetric.com combines and orderly tracks a set of data sources combined by DOI identifiers to retrieve metadata from books, being Google Books its main provider. It also retrieves information from commercial publishers and from some Open Access initiatives, including those led by university libraries, such as Harvard Library. We find issues with linkages between records and mentions or ISBN discrepancies. Furthermore, the authors find that automatic bots affect greatly Wikipedia mentions to books. The comparison with PlumX suggests that none of these tools provide a complete picture of the social attention generated by books and are rather complementary than comparable tools.

Practical implications

This study targets different audience which can benefit from the findings. First, bibliometricians and researchers who seek for alternative sources to develop bibliometric analyses of books, with a special focus on the Social Sciences and Humanities fields. Second, librarians and research managers who are the main clients to which these tools are directed. Third, Altmetric.com itself as well as other altmetric providers who might get a better understanding of the limitations users encounter and improve this promising tool.

Originality/value

This is the first study to analyze Altmetric.com’s functionalities and capabilities for providing metric data for books and to compare results from this platform, with those obtained via PlumX.

Highly visible sepsis publications from 2012 to 2017: Analysis and comparison of altmetrics and bibliometrics

PURPOSE

We sought to delineate highly visible publications related to sepsis. Within these subsets, elements of altmetrics performance, including mentions on Twitter, and the correlation between altmetrics and conventional citation counts were ascertained.

MATERIALS AND METHODS

Three subsets of sepsis publications from 2012 to 2017 were synthesized by the overall Altmetric.com attention score, number of mentions by unique Twitter users, and conventional citation counts. For these subsets, geolocated Twitter activity was plotted on a choropleth, the lag between publication date and altmetrics mentions was characterized, and correlations were examined between altmetrics performance and normalized conventional citation counts.

RESULTS

Of 57,152 PubMed query results, Altmetric.com data was available for 28,344 (49.6%). The top 50 publications by Altmetric.com attention score and Twitter attention represented a mix of original research and other types of work, garnering attention from Twitter users in 143 countries that was highly contemporaneous with publication. Altmetrics performance and conventional citation counts were poorly correlated.

CONCLUSIONS

While unreliable to gauge impact or future citation potential, altmetrics may be valuable for parties who wish to detect and drive public awareness of research findings and may enable researchers to dynamically explore the reach of their work in novel dimensions.