Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime

Characteristics, Impact, and Trends of Healthcare Simulation in Latin America and the Caribbean: A Bibliometric Analysis

SUMMARY STATEMENT

This study highlights the growing significance of healthcare simulation in enhancing the quality and safety of patient care across Latin America and the Caribbean, by analyzing bibliometric trends and the impact of publications on simulation-based clinical training between 2012 and 2022. Leveraging the Scopus database and VOSviewer software for thesaurus interaction analysis, the research identified 610 documents, accumulating 4681 citations, thereby indicating a burgeoning interest in this field with notable publication spikes in 2017 and 2020. Brazil and the United States emerged as leading contributors, with a primary focus on "simulation training," "clinical competence," "medical education," and "education." The study observed an uptick in international collaboration, mirroring the increase in document count and citations. This bibliometric review underscores the emphasis on evaluating technical skills and clinical practices as prevailing areas of interest, highlighting Brazil's significant academic contributions, and suggesting a promising future for the implementation of clinical simulation in the region. The study advocates for continued scholarly output to align with global advancements in medical simulation, aiming to optimize patient outcomes.

Are open science practices in dentistry associated with higher Altmetric scores and citation rates?

AIM

Open science, a set of principles and practices, aims to make scientific research more accessible and accountable, benefiting scientists and society. This study evaluated whether adopting open science practices (OSPs) correlates with higher citation rates and Altmetric scores.

METHODS

A random sample of randomised clinical trials (RCTs) on dental caries published between 2000 and 2022 was selected. A systematic PubMed search identified relevant RCTs, and data on OSPs - study registration, open methodology, open software, open scripts, open analysis plan, open data, open peer review, and open access (OA) - were manually collected by two independent assessors. The Robot Reviewer tool automatically evaluated the risk of bias (RoB). Outcomes included the total number of citations and the Altmetric Attention Score. Associations between OSPs, RoB, and other explanatory variables with the outcomes were assessed using binomial negative regression analysis, and expressed as Incidence Rate Ratio (IRR; α =0.05).

RESULTS

In total, 323 papers were analysed. At least one OSP was adopted in 57.5 % (n = 186) of the articles, dropping to 39.6 % (n = 128) without OA. Papers with protocol registration (IRR: 1.45; 95 % CI: 1.15, 1.82) and OA publication (IRR: 1.24; 95 % CI: 1.01, 1.53) had higher citation rates. Conversely, papers in full OA journals had fewer citations (IRR: 0.67; 95 % CI: 0.52, 0.87). After adjusting for RoB, low-risk studies showed higher citation rates (IRR: 1.48; 95 % CI: 1.14, 1.91), while OA lost significance. For Altmetric scores, registered and OA manuscripts showed higher scores (IRR: 3.74; 95 % CI: 2.00, 7.01; IRR: 1.69; 95 % CI: 1.04, 2.75), with registration remaining significant after adjusting for RoB and impact factor (IRR: 3.71; 95 % CI: 1.97-6.99).

CONCLUSION

The adoption of OSPs demonstrated a partial correlation with citation rates and Altmetric scores in RCTs on dental caries; however, these effects are complex and seem more related to the journal's impact factor.

CLINICAL SIGNIFICANCE

The citations and the attention to clinical trials in dentistry, which could drive clinical decision-making and the elaboration of policies and recommendations, seem to be driven more by the journal's prestige than by the adoption of OSPs.

Global COVID-19 Policy Engagement With Scientific Research Information: Altmetric Data Study

BACKGROUND

Previous studies on COVID-19 scholarly articles have primarily focused on bibliometric characteristics, neglecting the identification of institutional actors that cite recent scientific contributions related to COVID-19 in the policy domain, and their locations.

OBJECTIVE

The purpose of this study was to assess the online citation network and knowledge structure of COVID-19 research across policy domains over 2 years from January 2020 to January 2022, with a particular emphasis on geographical frequency. Two research questions were addressed. The first question was related to who has been the most active in policy engagement with science and research information sharing during the COVID-19 pandemic, particularly in terms of countries and organization types. The second question was related to whether there are significant differences in the types of coronavirus research shared among countries and continents.

METHODS

The Altmetric database was used to collect policy report citations of scientific articles for 3 topic terms (COVID-19, COVID-19 vaccine, and COVID-19 variants). Altmetric provides the URLs of policy agencies that have cited COVID-19 research. The scientific articles used for Altmetric citations are extracted from journals indexed by PubMed. The numbers of COVID-19, COVID-19 vaccine, and COVID-19 variant research outputs between January 1, 2020, and January 31, 2022, were 216,787, 16,748, and 2777, respectively. The study examined the frequency of citations based on policy institutional domains, such as intergovernmental organizations, national and domestic governmental organizations, and nongovernmental organizations (think tanks and academic institutions).

RESULTS

The World Health Organization (WHO) stood out as the most notable institution citing COVID-19-related research outputs. The WHO actively sought and disseminated information regarding the COVID-19 pandemic. The COVID-19 vaccine citation network exhibited the most extensive connections in terms of degree centrality, 2-local eigenvector centrality, and eigenvector centrality among the 3 key terms. The Netherlands, the United States, the United Kingdom, and Australia were the countries that sought and shared the most information on COVID-19 vaccines, likely due to their high numbers of COVID-19 cases. Developing nations, although gaining quicker access to COVID-19 vaccine information, appeared to be relatively isolated from the enriched COVID-19 pandemic content in the global network.

CONCLUSIONS

The global scientific network ecology during the COVID-19 pandemic revealed distinct types of links primarily centered around the WHO. Western countries demonstrated effective networking practices in constructing these networks. The prominent position of the key term "COVID-19 vaccine" demonstrates that nation-states align with global authority regardless of their national contexts. In summary, the citation networking practices of policy agencies have the potential to uncover the global knowledge distribution structure as a proxy for the networking strategy employed during a pandemic.

A study of the correlation between publication delays and measurement indicators of journal articles in the social network environment-based on online data in PLOS

The development of network technique and open access has made numerous research results freely obtained online, thereby facilitating the growth of the emerging evaluation methods of Altmetrics. However, it is unknown whether the time interval from reception to publication has an impact on the evaluation indicators of articles in the social network environment. We construct a range of time-series indexes that represent the features of the evaluation indicators and then explore the correlation of acceptance delay, technical delay, and overall delay with the relevant indicators of citations, usage, sharing and discussions, and collections that are obtained from the open access journal platform PLOS. Moreover, this research also explores the differences in the correlations of the delays for the literature in six subject areas with the corresponding indicators and the discrepancies of the correlations of delays and indexes in various metric quartiles. The results of the Mann-Whitney U test reveal that the length of delays affects the performance of the literature on some indicators. This study indicates that reducing the acceptance time and final publication time of articles can improve the efficiency of knowledge diffusion through the formal academic citation channel, but in the context of social networking communication, an appropriate interval at a particular stage in the publishing process can enhance the heat of sharing, discussion, and collection of articles to a small extent, hence boosting the influence and attention received by the literature on the internet.

The inverted U-shaped relationship between knowledge diversity of researchers and societal impact

With the increasing importance of interdisciplinary research, some studies have focused on the role of reference diversity by analysing reference lists of published papers. However, the relationship between the knowledge diversity of collaborating team members and research performance has been overlooked. In this study, we measured knowledge diversity through the disciplinary attributes of collaborating authors and research performance (understood as societal impact) through altmetric data. The major findings are: (1) The relationship between interdisciplinary collaboration diversity and societal impact is not a simple linear one, showing an inverted U-shaped pattern; and (2) As the number of collaborative disciplines increases, the marginal effects diminish or even become outweighed by the costs, showing a predominance of negative influences. Hence, diversity in interdisciplinary collaboration does not always have a positive impact. Research collaborations need to take into account the cost issues associated with the diversity of member disciplines.

Outreach and Post-Publication Impact of Soil Erosion Modelling Literature

Back in the 1930s, the aphorism “publish or perish” first appeared in an academic context. Today, this phrase is becoming a harsh reality in several academic environments, and scientists are giving increasing attention to publishing and disseminating their scientific work. Soil erosion modelers make no exception. With the introduction of the bibliometric field, the evaluation of the impact of a piece of scientific work becomes more articulated. The post-publication impact of the research became an important aspect too. In this study, we analyse the outreach and the impact of the literature on soil erosion modelling using the altmetric database, i.e., Altmetric. In our analysis, we use only a small fraction (around 15%) of Global Applications of Soil Erosion Modelling Tracker (GASEMT) papers because only 257 papers out of 1697 had an Altmetric Score (AS) larger than 0. We observed that media and policy documents mentioned more frequently literature dealing with global-scale assessments and future projection studies than local-scale ones. Papers that are frequently cited by researchers do not necessarily also yield high media and policy outreach. The GASEMT papers that had an AS larger than 0 were, on average, mentioned by one policy document and five Twitter users and had 100 Mendeley readers. Only around 5% and 9% of papers with AS > 0 appeared in news articles and blogs, respectively. However, this percentage was around 45% for Twitter and policy mentions. The top GASEMT paper’s upper bound was around 1 million Twitter followers, while this number was around 10,000 for the 10th ranked GASEMT paper. The exponentially increasing trend for erosion modelling papers having an AS has been confirmed, as during the last 3 years (2014–2017), we estimated that the number of entries had doubled compared to 2011–2014 and quadrupled if we compare it with 2008–2011.

All the research that’s fit to print: Open access and the news media

The goal of the open access (OA) movement is to help everyone access scholarly research, not just those who can afford to. However, most studies looking at whether OA has met this goal have focused on whether other scholars are making use of OA research. Few have considered how the broader public, including the news media, uses OA research. I sought to answer whether the news media mentions OA articles more or less than paywalled articles by looking at articles published from 2010 through 2018 in journals across all four quartiles of the Journal Impact Factor using data obtained through Altmetric.com and Web of Science. Gold, green and hybrid OA articles all had a positive correlation with the number of news mentions received. News mentions for OA articles did see a dip in 2018, although they remained higher than those for paywalled articles.

Correlation between universities’ Altmetric Attention Scores and their performance scores in Nature Index, Leiden, Times Higher Education and Quacquarelli Symonds ranking systems

Objective:

Altmetrics are claimed to measure the scientific, societal, educational, technological and economic impacts of science. They have some of these dimensions in common with university ranking and evaluating systems. Their results are, therefore, expected to be partially convergent with the systems’. Given the importance of the scientific and non-scientific impacts of science, this study investigated the correlations of universities’ altmetrics with their total and dimensional scores in Nature Index, Leiden, Times Higher Education (THE) and Quacquarelli Symonds (QS).

Methodology:

Following a correlational design, it explored an available sample of the universities commonly ranked in the systems in 2017. The data were collected from online documents using checklists and analysed by the Spearman correlation. As Altmetric Attention Score (ASS) is efficient in that it integrates several indicators into a single one, it was used as the proxy of the universities’ social performance.

Findings:

The universities showed significant positive correlations between their ASSs and their performance scores on the total and dimensional levels, except for industry income in THE, with an insignificant correlation, and proportion of collaborative publication less than 100 km. in Leiden, with an inverse correlation. The correlations ranged from weak to marginally strong.

Conclusion:

The positive relationships between the universities’ performance and ASSs signified that there existed some similarities in what they measured. However, they were of weak-to-marginally strong powers, implying that the metrics differed in what they measured. The findings contribute to the existing knowledge by providing some evidence of convergence between university-level altmetrics and university performances in various dimensions.

Science as collaborative knowledge generation

The COVID-19 pandemic points to the need for scientists to pool their efforts in order to understand this disease and respond to the ensuing crisis. Other global challenges also require such scientific cooperation. Yet in academic institutions, reward structures and incentives are based on systems that primarily fuel the competition between (groups of) scientific researchers. Competition between individual researchers, research groups, research approaches, and scientific disciplines is seen as an important selection mechanism and driver of academic excellence. These expected benefits of competition have come to define the organizational culture in academia. There are clear indications that the overreliance on competitive models undermines cooperative exchanges that might lead to higher quality insights. This damages the well-being and productivity of individual researchers and impedes efforts towards collaborative knowledge generation. Insights from social and organizational psychology on the side effects of relying on performance targets, prioritizing the achievement of success over the avoidance of failure, and emphasizing self-interest and efficiency, clarify implicit mechanisms that may spoil valid attempts at transformation. The analysis presented here elucidates that a broader change in the academic culture is needed to truly benefit from current attempts to create more open and collaborative practices for cumulative knowledge generation.

Identifying the princes base on Altmetrics: An awakening mechanism of sleeping beauties from the perspective of social media

In science, sleeping beauties (SBs) denotes a special phenomenon of the diffusion of scientific knowledge based on citation trajectories, the awakening of which is also measured through changes in the citations index. However, the rapid advancement of social media has altered the mode of scientific communication and knowledge diffusion. This study aims to re-identify SBs and its Prince from the perspective of comprehensive indicators, which involves the analysis of Altmetrics indexes and Citation index, and investigate the awakening mechanism of A-SB to supplement the research on the awakening mechanism of SBs. By combining Ab index, we redefined the Prince, which makes A-SB receive high attention after a long Sleeping period and reflects the most prominent academic or social behavior that awakens and sustains the Awakening of A-SB. Then we conducted empirical research on the retrieved PLOS Biology collection and examined Prince after identifying the A-SB. The analysis and summary of the characteristics of the identified A-SB and Prince revealed the SBs’ awakening mechanism under the comprehensive trajectory based on Altmetrics from the three dimensions of the influence between the indicators, the overall evolution trajectory of A-SB, and literature bibliometric attributes. In the trajectory of Delayed Recognition stage of A-SB, we define the Dogsleep of SBs, which mirrors that the instability of the Sleeping of SBs will generate a specific negative impact on Prince of A-SB and Awakening intensity. Besides, the literature bibliometric attributes cannot reflect the tendency of users to read academic papers, which again proves that the traditional citation index cannot be neglected in the awakening mechanism of A-SB. Overall, this study demonstrates the addition of the Altmetrics indexes as a useful complement, illustrating the inheritance and connection between the SBs based on the comprehensive trajectory and the SBs based on the citation diffusion trajectory.

A bibliometric analysis and visualization of the Journal of Documentation: 1945–2018

Purpose

As a pioneering and influential journal in the field of library and information science (LIS), the Journal of Documentation (JDoc) needs to be evaluated from a bibliometric perspective. This study aimed at conducting a bibliometric overview and visualization of the scientific output of JDoc from its inception in 1945–2018.

Design/methodology/approach

In this bibliometric study, 2056 papers published in JDoc were analyzed. All needed data were extracted from Scopus in 9 July 2019 in CSV format. Bibliometric analyses were done in Microsoft Excel. Visualization was done by Vosviewer software and applying techniques such as co-citation, co-authorship and co-occurrence. As a limited altmetric study, JDoc highly mentioned papers and the rate of their presence in social media were extracted from Altmetric LLP, too.

Findings

There was an increasing trend in published papers and received citations. Highly cited and most influential authors in JDoc are well-known in the field. However, the contributions of developing countries and their affiliated institutions to the journal were relatively low. This is true in case of author, country and institute co-authorship patterns. Highly frequent keywords and keyword co-occurrence patterns showed that the journal considered most topics related to LIS, including newly emerged ones. The authors and sources (generally journals) cited by JDoc are all prolific and influential ones.

Originality/value

The results of this study can be beneficial to JDoc editorial team for decision making on its further development as well as helpful for researchers and practitioners interesting in LIS field to have better contact with and contributions to the journal.

Evaluating the impact of research using the altmetrics approach (case study: the field of scientometrics)

Purpose

This paper aims to assess the impact of research in the field of scientometrics by using the altmetrics (social media metrics) approach.

Design/methodology/approach

This is an applied study which uses scientometric and altmetrics methods. The research population consists of the studies and their citations published in the two core journals (Scientometrics and Journal of Informetrics) in a period of five years (included 1,738 papers and 11,504 citations). Collecting and extracting the studies directly was carried from Springer and ScienceDirect databases. The Altmetric Explorer, a service provided by Altmetric.com, was used to collect data on studies from various sources (www.altmetric.com/). The research studies with the altmetric scores were identified (included 830 papers). The altmetric scores represent the quantity and quality of attention that the study has received on social media. The association between altmetric scores and citation indicators was investigated by using correlation tests.

Findings

The findings indicated a significant, positive and weak statistical relationship between the number of citations of the studies published in the field of scientometrics and the altmetric scores of these studies, as well as the number of readers of these studies in the two social networks (Mendeley and Citeulike) with the number of their citations. In this study, there was no statistically significant relationship between the number of citations of the studies and the number of readers on Twitter. In sum, the above findings suggest that some social networks and their indices can be representations of the impact of scientific papers, similar citations. However, owing to the weakness of the correlation coefficients, the replacement of these two categories of indicators is not recommended, but it is possible to use the altmetrics indicators as complementary scientometrics indicators in evaluating the impact of research.

Originality/value

Investigating the impact of research on social media can reflect the social impact of research and can also be useful for libraries, universities, and research organizations in planning, budgeting, and resource allocation processes.

Methods for mapping the impact of social sciences and humanities—A literature review

This article explores the current literature on ‘research impact’ in the social sciences and humanities (SSH). By providing a comprehensive review of available literature, drawing on national and international experiences, we take a systematic look at the impact agenda within SSH. The primary objective of this article is to examine key methodological components used to assess research impact comparing the advantages and disadvantages of each method. The study finds that research impact is a highly complex and contested concept in the SSH literature. Drawing on the strong methodological pluralism emerging in the literature, we conclude that there is considerable room for researchers, universities, and funding agencies to establish impact assessment tools directed towards specific missions while avoiding catch-all indicators and universal metrics.

How do altmetric sources evaluate scientific collaboration? An empirical investigation for Chinese collaboration publications

Purpose

The purpose of this paper is to provide insights on the improvement of academic impact and social attention of Chinese collaboration articles from the perspective of altmetrics.

Design/methodology/approach

The authors retrieved articles which are from the Chinese Academy of Sciences (CAS) and indexed by Nature Index as sampled articles. With the methods of distribution analysis, comparative analysis and correlation analysis, authors compare the coverage differences of altmetric sources for CAS Chinese articles and CAS international articles, and analyze the correlation between the collaborative information and the altmetric indicators.

Findings

Results show that the coverage of altmetric sources for CAS international articles is greater than that for CAS Chinese articles. Mendeley and Twitter cover a higher percentage of collaborative articles than other sources studied. Collaborative information, such as number of collaborating countries, number of collaborating institutions, and number of collaborating authors, show moderate or low correlation with altmetric indicator counts. Mendeley readership has a moderate correlation with altmetric indicators like tweets, news outlets and blog posts.

Practical implications

International scientific collaboration at different levels improves attention, academic impact and social impact of articles. International collaboration and altmetrics indicators supplement each other. The results of this study can help us better understand the relationship between altmetrics indicators of articles and collaborative information of articles. It is of great significance to evaluate the influence of Chinese articles, as well as help to improve the academic impact and social attention of Chinese collaboration articles.

Originality/value

To the best of authors’ knowledge, few studies focus on the use of altmetrics to assess publications produced through Chinese academic collaboration. This study is one of a few attempts that include the number of collaborating countries, number of collaborating institutions, and number of collaborating authors of scientific collaboration into the discussion of altmetric indicators and figured out the relationship among them.

Do altmetrics correlate with citations? A study based on the 1,000 most-cited articles

Purpose

The purpose of this paper is to study the presence of highly cited papers of Nature in social media websites and tools. It also tries to examine the correlation between altmetric and bibliometric indicators.

Design/methodology/approach

This descriptive study was carried out using altmetric indicators. The research sample consisted of 1,000 most-cited articles in Nature. In February 2019, the bibliographic information of these articles was extracted from the Scopus database. Then, the titles of all articles were manually searched on Google, and by referring to the article in the journal website and altmetric institution, the data related to social media presence and altmetric score of articles were collected. The data were analyzed using Microsoft Excel and SPSS.

Findings

According to the results of the study, from 1,000 articles, 989 of them (98.9 per cent) were mentioned at least once in different social media websites and tools. The most used altmetric source in highly cited articles was Mendeley (98.9 per cent), followed by Citeulike (79.8 per cent) and Wikipedia (69.4 per cent). Most Tweets, blog posts, Facebook posts, news stories, readers in Mendeley, Citeulike and Connotea and Wikipedia citations belonged to the article titled “Mastering the game of Go with deep neural networks and tree search”. The highest altmetric score was 3,135 which belonged to this paper. Most tweeters and articles’ readers were from the USA. The membership type of the tweeters was public membership. In terms of fields of study, most readers were PhD students in Agricultural and Biological Sciences. Finally, the results of Spearman’s Correlation revealed positive significant statistical correlation between all altmetric indicators and received citations of highly cited articles (p-value = 0.0001).

Practical implications

The results of this study can help researchers, editors and editorial boards of journals better understand the importance and benefits of using social media and tools to publish articles.

Originality/value

Altmetrics is a relatively new field, and in particular, there are not many studies related to the presence of articles in various social media until now. Accordingly, in this study, a comprehensive altmetric analysis was carried out on 1000 most-cited articles of one of the world's most reliable journals.

Do articles in open access journals have more frequent altmetric activity than articles in subscription-based journals? An investigation of the research output of Finnish universities

Scientific articles available in Open Access (OA) have been found to attract more citations and online attention to the extent that it has become common to speak about OA Altmetrics Advantage. This research investigates how the OA Altmetrics Advantage holds for a specific case of research articles, namely the research outputs from universities in Finland. Furthermore, this research examines disciplinary and platform specific differences in that (dis)advantage. The new methodological approaches developed in this research focus on relative visibility, i.e. how often articles in OA journals receive at least one mention on the investigated online platforms, and relative receptivity, i.e. how frequently articles in OA journals gain mentions in comparison to articles in subscription-based journals. The results show significant disciplinary and platform specific differences in the OA advantage, with articles in OA journals within for instance veterinary sciences, social and economic geography and psychology receiving more citations and attention on social media platforms, while the opposite was found for articles in OA journals within medicine and health sciences. The results strongly support field- and platform-specific considerations when assessing the influence of journal OA status on altmetrics. The new methodological approaches used in this research will serve future comparative research into OA advantage of scientific articles over time and between countries.

Usage, captures, mentions, social media and citations of LIS highly cited papers: an altmetrics study

Purpose

The purpose of this paper is to investigate the usage, captures, mentions, social media and citations of highly cited papers of Library and information science (LIS).

Design/methodology/approach

This study is quantitative research that was conducted using scientometrics and altmetrics indicators. The research sample consists of LIS classic papers. The papers contain highly cited papers of LIS that are introduced by Google Scholar. The research data have been gathered from Google Scholar, Scopus and Plum Analytics Categories. The data analysis has been done by Excel and SPSS applications.

Findings

The data indicate that among the highly cited articles of LIS, the highest score regarding the usage, captures, mentions and social media and the most abundance of citations belong to “Citation advantage of open access articles” and “Usage patterns of collaborative tagging systems.” Based on the results of Spearman statistical tests, there is a positive significant correlation between Google Scholar Citations and all studied indicators. However, only the correlation between Google Scholar Citations with capture metrics (p-value = 0.047) and citation metrics (p-value = 0.0001) was statistically significant.

Originality/value

Altmetrics indicators can be used as complement traditional indicators of Scientometrics to study the impact of papers. Therefore, the Altmetrics knowledge of LIS researchers and experts and practicing new studies in this field will be very important.

Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data

In this study, we address the question whether (and to what extent, respectively) altmetrics are related to the scientific quality of papers (as measured by peer assessments). Only a few studies have previously investigated the relationship between altmetrics and assessments by peers. In the first step, we analyse the underlying dimensions of measurement for traditional metrics (citation counts) and altmetrics–by using principal component analysis (PCA) and factor analysis (FA). In the second step, we test the relationship between the dimensions and quality of papers (as measured by the post-publication peer-review system of F1000Prime assessments)–using regression analysis. The results of the PCA and FA show that altmetrics operate along different dimensions, whereas Mendeley counts are related to citation counts, and tweets form a separate dimension. The results of the regression analysis indicate that citation-based metrics and readership counts are significantly more related to quality, than tweets. This result on the one hand questions the use of Twitter counts for research evaluation purposes and on the other hand indicates potential use of Mendeley reader counts.

Field- and time-normalization of data with many zeros: an empirical analysis using citation and Twitter data

Thelwall (J Informetr 11(1):128–151, 2017a. 10.1016/j.joi.2016.12.002; Web indicators for research evaluation: a practical guide. Morgan and Claypool, London, 2017b) proposed a new family of field- and time-normalized indicators, which is intended for sparse data. These indicators are based on units of analysis (e.g., institutions) rather than on the paper level. They compare the proportion of mentioned papers (e.g., on Twitter) of a unit with the proportion of mentioned papers in the corresponding fields and publication years. We propose a new indicator (Mantel–Haenszel quotient, MHq) for the indicator family. The MHq is rooted in the Mantel–Haenszel (MH) analysis. This analysis is an established method, which can be used to pool the data from several 2 × 2 cross tables based on different subgroups. We investigate using citations and assessments by peers whether the indicator family can distinguish between quality levels defined by the assessments of peers. Thus, we test the convergent validity. We find that the MHq is able to distinguish between quality levels in most cases while other indicators of the family are not. Since our study approves the MHq as a convergent valid indicator, we apply the MHq to four different Twitter groups as defined by the company Altmetric. Our results show that there is a weak relationship between the Twitter counts of all four Twitter groups and scientific quality, much weaker than between citations and scientific quality. Therefore, our results discourage the use of Twitter counts in research evaluation.

A correlation comparison between Altmetric Attention Scores and citations for six PLOS journals

This study considered all articles published in six Public Library of Science (PLOS) journals in 2012 and Web of Science citations for these articles as of May 2015. A total of 2,406 articles were analyzed to examine the relationships between Altmetric Attention Scores (AAS) and Web of Science citations. The AAS for an article, provided by Altmetric aggregates activities surrounding research outputs in social media (news outlet mentions, tweets, blogs, Wikipedia, etc.). Spearman correlation testing was done on all articles and articles with AAS. Further analysis compared the stratified datasets based on percentile ranks of AAS: top 50%, top 25%, top 10%, and top 1%. Comparisons across the six journals provided additional insights. The results show significant positive correlations between AAS and citations with varied strength for all articles and articles with AAS (or social media mentions), as well as for normalized AAS in the top 50%, top 25%, top 10%, and top 1% datasets. Four of the six PLOS journals, Genetics, Pathogens, Computational Biology, and Neglected Tropical Diseases, show significant positive correlations across all datasets. However, for the two journals with high impact factors, PLOS Biology and Medicine, the results are unexpected: the Medicine articles showed no significant correlations but the Biology articles tested positive for correlations with the whole dataset and the set with AAS. Both journals published substantially fewer articles than the other four journals. Further research to validate the AAS algorithm, adjust the weighting scheme, and include appropriate social media sources is needed to understand the potential uses and meaning of AAS in different contexts and its relationship to other metrics.

Altmetric scores: short-term popularity or long-term scientific importance

Purpose

The purpose of this paper is to report on a three-year case study of the extent with which altmetrics compare to traditional metrics in certain subject areas for selected departments at the University of Cincinnati, College of Medicine (COM).

Design/methodology/approach

A three-year analysis of peer-reviewed papers and invited editorials from 2009 to 2013 written by tenure-track faculty from 20 COM departments was done to explore what subject areas received the highest altmetric scores. Research output was searched in PubMed; articles were quantified by subject area, times cited in Scopus, and its altmetric score over each of three successive years.

Findings

The topics of the highest scored altmetric papers (n = 40) sample focused on stroke, obesity, and diabetes for all three years. Analysis of high initial altmetric scores over the course of the three years shifted from a possible predictor of future impact in the second year to no indicator of long-term interest in the scientific community as the public interest waned over time.

Research limitations/implications

The authors used Scopus Times Cited and Altmetrics.com to gather data.

Originality/value

Initially assessed a total of 3,678 unique publications and worked with the 40 highest altmetric scores in subsequent years. Data showed that subjects of interest to the public receive the highest altmetric scores and the topic areas did not change over the course of the study. These initially high altmetric scores do not indicate long-term interest by the scientific community.