Early indicators of scientific impact: Predicting citations with altmetrics

A peek into the room where it happens: Quantifying ABA's influence on public policy discussions

To maximize its influence, applied behavior analysis must both create solutions and shape public policy to implement those solutions at scale. From the perspective of data-driven decision making, it is illogical to talk about seeking public policy influence without consulting evidence showing when influence has been achieved. One relevant form of evidence is the attention that behavioral solutions receive in published discussions about policy issues, and here I show how much of this attention has been earned by articles published in Journal of Applied Behavior Analysis. I also propose using the same kind of data to support finer grained analyses focusing on specific behavior problems, specific types of interventions, and the research programs of individual investigators. Although this is far from a complete account of the influence of applied behavior analysis on policy, it is better to have data than none if the goal is to transform the quest for influence on policy from a matter of speculation and casual discussion into an evidence-based practice.

Measuring altmetric events: the need for longer observation period and article level computations

Purpose

This paper aims to explore the impact of the data observation period on the computation of altmetric measures like velocity index (VI) and half-life. Furthermore, it also attempts to determine whether article-level computations are better than computations on the whole of the data for computing such measures.

Design/methodology/approach

The complete publication records for the year 2016 indexed in Web of Science and their altmetric data (original tweets) obtained from PlumX are obtained and analysed. The creation date of articles is taken from Crossref. Two time-dependent variables, namely, half-life and VI are computed. The altmetric measures are computed for all articles at different observation points, and by using whole group as well as article-level averaging.

Findings

The results show that use of longer observation period significantly changes the values of different altmetric measures computed. Furthermore, use of article-level delineation is advocated for computing different measures for a more accurate representation of the true values for the article distribution.

Research limitations/implications

The analytical results show that using different observation periods change the measured values of the time-related altmetric measures. It is suggested that longer observation period should be used for appropriate measurement of altmetric measures. Furthermore, the use of article-level delineation for computing the measures is advocated as a more accurate method to capture the true values of such measures.

Practical implications

The research work suggests that altmetric mentions accrue for a longer period than the commonly believed short life span and therefore the altmetric measurements should not be limited to observation of early accrued data only.

Social implications

The present study indicates that use of altmetric measures for research evaluation or other purposes should be based on data for a longer observation period and article-level delineation may be preferred. It contradicts the common belief that tweet accumulation about scholarly articles decay quickly.

Originality/value

Several studies have shown that altmetric data correlate well with citations and hence early altmetric counts can be used to predict future citations. Inspired by these findings, majority of such monitoring and measuring exercises have focused mainly on capturing immediate altmetric event data for articles just after the publication of the paper. This paper demonstrates the impact of the observation period and article-level aggregation on such computations and suggests to use a longer observation period and article-level delineation. To the best of the authors’ knowledge, this is the first such study of its kind and presents novel findings.

Predictable by publication: discovery of early highly cited academic papers based on their own features

Purpose

Predicting highly cited papers can enable an evaluation of the potential of papers and the early detection and determination of academic achievement value. However, most highly cited paper prediction studies consider early citation information, so predicting highly cited papers by publication is challenging. Therefore, the authors propose a method for predicting early highly cited papers based on their own features.

Design/methodology/approach

This research analyzed academic papers published in the Journal of the Association for Computing Machinery (ACM) from 2000 to 2013. Five types of features were extracted: paper features, journal features, author features, reference features and semantic features. Subsequently, the authors applied a deep neural network (DNN), support vector machine (SVM), decision tree (DT) and logistic regression (LGR), and they predicted highly cited papers 1–3 years after publication.

Findings

Experimental results showed that early highly cited academic papers are predictable when they are first published. The authors’ prediction models showed considerable performance. This study further confirmed that the features of references and authors play an important role in predicting early highly cited papers. In addition, the proportion of high-quality journal references has a more significant impact on prediction.

Originality/value

Based on the available information at the time of publication, this study proposed an effective early highly cited paper prediction model. This study facilitates the early discovery and realization of the value of scientific and technological achievements.

YouTube and science: models for research impact

Video communication has been rapidly increasing over the past decade, with YouTube providing a medium where users can post, discover, share, and react to videos. There has also been an increase in the number of videos citing research articles, especially since it has become relatively commonplace for academic conferences to require video submissions. However, the relationship between research articles and YouTube videos is not clear, and the purpose of the present paper is to address this issue. We created new datasets using YouTube videos and mentions of research articles on various online platforms. We found that most of the articles cited in the videos are related to medicine and biochemistry. We analyzed these datasets through statistical techniques and visualization, and built machine learning models to predict (1) whether a research article is cited in videos, (2) whether a research article cited in a video achieves a level of popularity, and (3) whether a video citing a research article becomes popular. The best models achieved F1 scores between 80% and 94%. According to our results, research articles mentioned in more tweets and news coverage have a higher chance of receiving video citations. We also found that video views are important for predicting citations and increasing research articles' popularity and public engagement with science.

The inverted U-shaped relationship between knowledge diversity of researchers and societal impact

With the increasing importance of interdisciplinary research, some studies have focused on the role of reference diversity by analysing reference lists of published papers. However, the relationship between the knowledge diversity of collaborating team members and research performance has been overlooked. In this study, we measured knowledge diversity through the disciplinary attributes of collaborating authors and research performance (understood as societal impact) through altmetric data. The major findings are: (1) The relationship between interdisciplinary collaboration diversity and societal impact is not a simple linear one, showing an inverted U-shaped pattern; and (2) As the number of collaborative disciplines increases, the marginal effects diminish or even become outweighed by the costs, showing a predominance of negative influences. Hence, diversity in interdisciplinary collaboration does not always have a positive impact. Research collaborations need to take into account the cost issues associated with the diversity of member disciplines.

A modified LSTM network to predict the citation counts of papers

Quantifiable predictability in the citation counts of articles is significant in scientometrics and informetrics. Many metrics based on the citation counts can evaluate the scientific impact of research articles and journals. Utilising time series models, an article’s citation counts up to the yth year after publication can be predicted by those up to the previous years. However, the typically used models cannot predict the fat tail of the actual citation distributions. Thus, based on cumulative advantage of the citation behaviour, we propose a method to predict the accumulated citation counts, by using a random number sampled from a power-law distribution to modify the results given by a recurrent neural network (RNN), long short-term memory. Extensive experiments on the data set including 17 journals in information science verified the effectiveness of our method by the good fittings on distributions and evolutionary trends of the citation counts of articles. Our method has the potential to be extended to predict other popular assessment measures such as impact factor and h-index for journals.

Can the quality of published academic journal articles be assessed with machine learning?

Formal assessments of the quality of the research produced by departments and universities are now conducted by many countries to monitor achievements and allocate performance-related funding. These evaluations are hugely time consuming if conducted by post-publication peer review and are simplistic if based on citations or journal impact factors. This article investigates whether machine learning could help reduce the burden of peer review by using citations and metadata to learn how to score articles from a sample assessed by peer review. An experiment is used to underpin the discussion, attempting to predict journal citation thirds, as a proxy for article quality scores, for all Scopus narrow fields from 2014 to 2020. The results show that these proxy quality thirds can be predicted with above baseline accuracy in all 326 narrow fields, with Gradient Boosting Classifier, Random Forest Classifier, or Multinomial Naïve Bayes being the most accurate in nearly all cases. Nevertheless, the results partly leverage journal writing styles and topics, which are unwanted for some practical applications and cause substantial shifts in average scores between countries and between institutions within a country. There may be scope for predicting articles scores when the predictions have the highest probability.

Peer Review

https://publons.com/publon/10.1162/qss_a_00185

Public Reaction to Scientific Research via Twitter Sentiment Prediction

Purpose

Social media users share their ideas, thoughts, and emotions with other users. However, it is not clear how online users would respond to new research outcomes. This study aims to predict the nature of the emotions expressed by Twitter users toward scientific publications. Additionally, we investigate what features of the research articles help in such prediction. Identifying the sentiments of research articles on social media will help scientists gauge a new societal impact of their research articles.

Design/methodology/approach

Several tools are used for sentiment analysis, so we applied five sentiment analysis tools to check which are suitable for capturing a tweet's sentiment value and decided to use NLTK VADER and TextBlob. We segregated the sentiment value into negative, positive, and neutral. We measure the mean and median of tweets’ sentiment value for research articles with more than one tweet. We next built machine learning models to predict the sentiments of tweets related to scientific publications and investigated the essential features that controlled the prediction models.

Findings

We found that the most important feature in all the models was the sentiment of the research article title followed by the author count. We observed that the tree-based models performed better than other classification models, with Random Forest achieving 89% accuracy for binary classification and 73% accuracy for three-label classification.

Research limitations

In this research, we used state-of-the-art sentiment analysis libraries. However, these libraries might vary at times in their sentiment prediction behavior. Tweet sentiment may be influenced by a multitude of circumstances and is not always immediately tied to the paper's details. In the future, we intend to broaden the scope of our research by employing word2vec models.

Practical implications

Many studies have focused on understanding the impact of science on scientists or how science communicators can improve their outcomes. Research in this area has relied on fewer and more limited measures, such as citations and user studies with small datasets. There is currently a critical need to find novel methods to quantify and evaluate the broader impact of research. This study will help scientists better comprehend the emotional impact of their work. Additionally, the value of understanding the public's interest and reactions helps science communicators identify effective ways to engage with the public and build positive connections between scientific communities and the public.

Originality/value

This study will extend work on public engagement with science, sociology of science, and computational social science. It will enable researchers to identify areas in which there is a gap between public and expert understanding and provide strategies by which this gap can be bridged.