1
|
Critchfield TS. A peek into the room where it happens: Quantifying ABA's influence on public policy discussions. J Appl Behav Anal 2024; 57:288-303. [PMID: 38247278 DOI: 10.1002/jaba.1056] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2022] [Accepted: 12/21/2023] [Indexed: 01/23/2024]
Abstract
To maximize its influence, applied behavior analysis must both create solutions and shape public policy to implement those solutions at scale. From the perspective of data-driven decision making, it is illogical to talk about seeking public policy influence without consulting evidence showing when influence has been achieved. One relevant form of evidence is the attention that behavioral solutions receive in published discussions about policy issues, and here I show how much of this attention has been earned by articles published in Journal of Applied Behavior Analysis. I also propose using the same kind of data to support finer grained analyses focusing on specific behavior problems, specific types of interventions, and the research programs of individual investigators. Although this is far from a complete account of the influence of applied behavior analysis on policy, it is better to have data than none if the goal is to transform the quest for influence on policy from a matter of speculation and casual discussion into an evidence-based practice.
Collapse
|
2
|
Hu Z, Cui J, Lin A. Identifying potentially excellent publications using a citation-based machine learning approach. Inf Process Manag 2023. [DOI: 10.1016/j.ipm.2023.103323] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/18/2023]
|
3
|
Thelwall M, Kousha K, Abdoli M, Stuart E, Makita M, Wilson P, Levitt J. Do altmetric scores reflect article quality? Evidence from the
UK
Research Excellence Framework 2021. J Assoc Inf Sci Technol 2023. [DOI: 10.1002/asi.24751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/17/2023]
Affiliation(s)
- Mike Thelwall
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
| | - Kayvan Kousha
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
| | - Mahshid Abdoli
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
| | - Emma Stuart
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
| | - Meiko Makita
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
| | - Paul Wilson
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
| | - Jonathan Levitt
- Statistical Cybermetrics and Research Evaluation Group University of Wolverhampton Wolverhampton UK
| |
Collapse
|
4
|
Karmakar M, Singh VK, Banshal SK. Measuring altmetric events: the need for longer observation period and article level computations. GLOBAL KNOWLEDGE, MEMORY AND COMMUNICATION 2023. [DOI: 10.1108/gkmc-08-2022-0203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/10/2023]
Abstract
Purpose
This paper aims to explore the impact of the data observation period on the computation of altmetric measures like velocity index (VI) and half-life. Furthermore, it also attempts to determine whether article-level computations are better than computations on the whole of the data for computing such measures.
Design/methodology/approach
The complete publication records for the year 2016 indexed in Web of Science and their altmetric data (original tweets) obtained from PlumX are obtained and analysed. The creation date of articles is taken from Crossref. Two time-dependent variables, namely, half-life and VI are computed. The altmetric measures are computed for all articles at different observation points, and by using whole group as well as article-level averaging.
Findings
The results show that use of longer observation period significantly changes the values of different altmetric measures computed. Furthermore, use of article-level delineation is advocated for computing different measures for a more accurate representation of the true values for the article distribution.
Research limitations/implications
The analytical results show that using different observation periods change the measured values of the time-related altmetric measures. It is suggested that longer observation period should be used for appropriate measurement of altmetric measures. Furthermore, the use of article-level delineation for computing the measures is advocated as a more accurate method to capture the true values of such measures.
Practical implications
The research work suggests that altmetric mentions accrue for a longer period than the commonly believed short life span and therefore the altmetric measurements should not be limited to observation of early accrued data only.
Social implications
The present study indicates that use of altmetric measures for research evaluation or other purposes should be based on data for a longer observation period and article-level delineation may be preferred. It contradicts the common belief that tweet accumulation about scholarly articles decay quickly.
Originality/value
Several studies have shown that altmetric data correlate well with citations and hence early altmetric counts can be used to predict future citations. Inspired by these findings, majority of such monitoring and measuring exercises have focused mainly on capturing immediate altmetric event data for articles just after the publication of the paper. This paper demonstrates the impact of the observation period and article-level aggregation on such computations and suggests to use a longer observation period and article-level delineation. To the best of the authors’ knowledge, this is the first such study of its kind and presents novel findings.
Collapse
|
5
|
Tang X, Zhou H, Li S. Predictable by publication: discovery of early highly cited academic papers based on their own features. LIBRARY HI TECH 2023. [DOI: 10.1108/lht-06-2022-0305] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/05/2023]
Abstract
PurposePredicting highly cited papers can enable an evaluation of the potential of papers and the early detection and determination of academic achievement value. However, most highly cited paper prediction studies consider early citation information, so predicting highly cited papers by publication is challenging. Therefore, the authors propose a method for predicting early highly cited papers based on their own features.Design/methodology/approachThis research analyzed academic papers published in the Journal of the Association for Computing Machinery (ACM) from 2000 to 2013. Five types of features were extracted: paper features, journal features, author features, reference features and semantic features. Subsequently, the authors applied a deep neural network (DNN), support vector machine (SVM), decision tree (DT) and logistic regression (LGR), and they predicted highly cited papers 1–3 years after publication.FindingsExperimental results showed that early highly cited academic papers are predictable when they are first published. The authors’ prediction models showed considerable performance. This study further confirmed that the features of references and authors play an important role in predicting early highly cited papers. In addition, the proportion of high-quality journal references has a more significant impact on prediction.Originality/valueBased on the available information at the time of publication, this study proposed an effective early highly cited paper prediction model. This study facilitates the early discovery and realization of the value of scientific and technological achievements.
Collapse
|
6
|
Deep representation learning of scientific paper reveals its potential scholarly impact. J Informetr 2023. [DOI: 10.1016/j.joi.2023.101376] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/28/2023]
|
7
|
Abstract
Video communication has been rapidly increasing over the past decade, with YouTube providing a medium where users can post, discover, share, and react to videos. There has also been an increase in the number of videos citing research articles, especially since it has become relatively commonplace for academic conferences to require video submissions. However, the relationship between research articles and YouTube videos is not clear, and the purpose of the present paper is to address this issue. We created new datasets using YouTube videos and mentions of research articles on various online platforms. We found that most of the articles cited in the videos are related to medicine and biochemistry. We analyzed these datasets through statistical techniques and visualization, and built machine learning models to predict (1) whether a research article is cited in videos, (2) whether a research article cited in a video achieves a level of popularity, and (3) whether a video citing a research article becomes popular. The best models achieved F1 scores between 80% and 94%. According to our results, research articles mentioned in more tweets and news coverage have a higher chance of receiving video citations. We also found that video views are important for predicting citations and increasing research articles' popularity and public engagement with science.
Collapse
|
8
|
Xia W, Li T, Li C. A review of scientific impact prediction: tasks, features and methods. Scientometrics 2022. [DOI: 10.1007/s11192-022-04547-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
9
|
Wang G, Gan Y, Yang H. The inverted U-shaped relationship between knowledge diversity of researchers and societal impact. Sci Rep 2022; 12:18585. [PMID: 36329084 PMCID: PMC9633593 DOI: 10.1038/s41598-022-21821-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2022] [Accepted: 10/04/2022] [Indexed: 11/05/2022] Open
Abstract
With the increasing importance of interdisciplinary research, some studies have focused on the role of reference diversity by analysing reference lists of published papers. However, the relationship between the knowledge diversity of collaborating team members and research performance has been overlooked. In this study, we measured knowledge diversity through the disciplinary attributes of collaborating authors and research performance (understood as societal impact) through altmetric data. The major findings are: (1) The relationship between interdisciplinary collaboration diversity and societal impact is not a simple linear one, showing an inverted U-shaped pattern; and (2) As the number of collaborative disciplines increases, the marginal effects diminish or even become outweighed by the costs, showing a predominance of negative influences. Hence, diversity in interdisciplinary collaboration does not always have a positive impact. Research collaborations need to take into account the cost issues associated with the diversity of member disciplines.
Collapse
Affiliation(s)
- Gaofeng Wang
- grid.59053.3a0000000121679639School of Humanities and Social Sciences, University of Science and Technology of China, Hefei, China
| | - Yetong Gan
- grid.59053.3a0000000121679639School of Humanities and Social Sciences, University of Science and Technology of China, Hefei, China
| | - Haodong Yang
- grid.59053.3a0000000121679639School of Humanities and Social Sciences, University of Science and Technology of China, Hefei, China
| |
Collapse
|
10
|
Nabavi M. An analysis of journalism articles achieving high Altmetric attention scores. LEARNED PUBLISHING 2022. [DOI: 10.1002/leap.1492] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- Majid Nabavi
- Department of Knowledge and Information Science Shiraz University Shiraz Iran
| |
Collapse
|
11
|
Du W, Li Z, Xie Z. A modified LSTM network to predict the citation counts of papers. J Inf Sci 2022. [DOI: 10.1177/01655515221111000] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/15/2022]
Abstract
Quantifiable predictability in the citation counts of articles is significant in scientometrics and informetrics. Many metrics based on the citation counts can evaluate the scientific impact of research articles and journals. Utilising time series models, an article’s citation counts up to the yth year after publication can be predicted by those up to the previous years. However, the typically used models cannot predict the fat tail of the actual citation distributions. Thus, based on cumulative advantage of the citation behaviour, we propose a method to predict the accumulated citation counts, by using a random number sampled from a power-law distribution to modify the results given by a recurrent neural network (RNN), long short-term memory. Extensive experiments on the data set including 17 journals in information science verified the effectiveness of our method by the good fittings on distributions and evolutionary trends of the citation counts of articles. Our method has the potential to be extended to predict other popular assessment measures such as impact factor and h-index for journals.
Collapse
Affiliation(s)
- Wumei Du
- College of Liberal Arts and Sciences, National University of Defense Technology, China
| | - Zhemin Li
- College of Liberal Arts and Sciences, National University of Defense Technology, China
| | - Zheng Xie
- College of Liberal Arts and Sciences, National University of Defense Technology, China
| |
Collapse
|
12
|
Komaritsa VN. Benefits of Using Open Access: Citation Analysis. AUTOMATIC DOCUMENTATION AND MATHEMATICAL LINGUISTICS 2022. [DOI: 10.3103/s0005105522040045] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
13
|
Okagbue HI, Akinsola BE, Teixeira da Silva JA. Relationship between number of downloads and three journal-based metrics of 11 subject categories among 1575 Springer Nature journals. COLLNET JOURNAL OF SCIENTOMETRICS AND INFORMATION MANAGEMENT 2022. [DOI: 10.1080/09737766.2022.2117667] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Affiliation(s)
- Hilary I. Okagbue
- Department of Mathematics, Covenant University, Canaanland, P.M.B 1023, Ota, Nigeria
| | | | | |
Collapse
|
14
|
Shahzad M, Alhoori H, Freedman R, Rahman SA. Quantifying the online long-term interest in research. J Informetr 2022. [DOI: 10.1016/j.joi.2022.101288] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
15
|
Huang S, Huang Y, Bu Y, Lu W, Qian J, Wang D. Fine-grained citation count prediction via a transformer-based model with among-attention mechanism. Inf Process Manag 2022. [DOI: 10.1016/j.ipm.2021.102799] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/28/2023]
|
16
|
Thelwall M. Can the quality of published academic journal articles be assessed with machine learning? QUANTITATIVE SCIENCE STUDIES 2022. [DOI: 10.1162/qss_a_00185] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Abstract
Abstract
Formal assessments of the quality of the research produced by departments and universities are now conducted by many countries to monitor achievements and allocate performance-related funding. These evaluations are hugely time consuming if conducted by post-publication peer review and are simplistic if based on citations or journal impact factors. This article investigates whether machine learning could help reduce the burden of peer review by using citations and metadata to learn how to score articles from a sample assessed by peer review. An experiment is used to underpin the discussion, attempting to predict journal citation thirds, as a proxy for article quality scores, for all Scopus narrow fields from 2014 to 2020. The results show that these proxy quality thirds can be predicted with above baseline accuracy in all 326 narrow fields, with Gradient Boosting Classifier, Random Forest Classifier, or Multinomial Naïve Bayes being the most accurate in nearly all cases. Nevertheless, the results partly leverage journal writing styles and topics, which are unwanted for some practical applications and cause substantial shifts in average scores between countries and between institutions within a country. There may be scope for predicting articles scores when the predictions have the highest probability.
Peer Review
https://publons.com/publon/10.1162/qss_a_00185
Collapse
|
17
|
Zhou Y, Wang R, Zeng A. Predicting the impact and publication date of individual scientists’ future papers. Scientometrics 2022. [DOI: 10.1007/s11192-022-04286-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
18
|
Huang S, Qian J, Huang Y, Lu W, Bu Y, Yang J, Cheng Q. Disclosing the relationship between citation structure and future impact of a publication. J Assoc Inf Sci Technol 2021. [DOI: 10.1002/asi.24610] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Shengzhi Huang
- School of Information Management Wuhan University Wuhan Hubei China
- Information Retrieval and Knowledge Mining Laboratory Wuhan University Wuhan Hubei China
| | - Jiajia Qian
- School of Information Management Wuhan University Wuhan Hubei China
- Information Retrieval and Knowledge Mining Laboratory Wuhan University Wuhan Hubei China
| | - Yong Huang
- School of Information Management Wuhan University Wuhan Hubei China
- Information Retrieval and Knowledge Mining Laboratory Wuhan University Wuhan Hubei China
| | - Wei Lu
- School of Information Management Wuhan University Wuhan Hubei China
- Information Retrieval and Knowledge Mining Laboratory Wuhan University Wuhan Hubei China
| | - Yi Bu
- Department of Information Management Peking University Beijing China
| | - Jinqing Yang
- School of Information Management Wuhan University Wuhan Hubei China
- Information Retrieval and Knowledge Mining Laboratory Wuhan University Wuhan Hubei China
| | - Qikai Cheng
- School of Information Management Wuhan University Wuhan Hubei China
- Information Retrieval and Knowledge Mining Laboratory Wuhan University Wuhan Hubei China
| |
Collapse
|
19
|
Public Reaction to Scientific Research via Twitter Sentiment Prediction. JOURNAL OF DATA AND INFORMATION SCIENCE 2021. [DOI: 10.2478/jdis-2022-0003] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
Abstract
Abstract
Purpose
Social media users share their ideas, thoughts, and emotions with other users. However, it is not clear how online users would respond to new research outcomes. This study aims to predict the nature of the emotions expressed by Twitter users toward scientific publications. Additionally, we investigate what features of the research articles help in such prediction. Identifying the sentiments of research articles on social media will help scientists gauge a new societal impact of their research articles.
Design/methodology/approach
Several tools are used for sentiment analysis, so we applied five sentiment analysis tools to check which are suitable for capturing a tweet's sentiment value and decided to use NLTK VADER and TextBlob. We segregated the sentiment value into negative, positive, and neutral. We measure the mean and median of tweets’ sentiment value for research articles with more than one tweet. We next built machine learning models to predict the sentiments of tweets related to scientific publications and investigated the essential features that controlled the prediction models.
Findings
We found that the most important feature in all the models was the sentiment of the research article title followed by the author count. We observed that the tree-based models performed better than other classification models, with Random Forest achieving 89% accuracy for binary classification and 73% accuracy for three-label classification.
Research limitations
In this research, we used state-of-the-art sentiment analysis libraries. However, these libraries might vary at times in their sentiment prediction behavior. Tweet sentiment may be influenced by a multitude of circumstances and is not always immediately tied to the paper's details. In the future, we intend to broaden the scope of our research by employing word2vec models.
Practical implications
Many studies have focused on understanding the impact of science on scientists or how science communicators can improve their outcomes. Research in this area has relied on fewer and more limited measures, such as citations and user studies with small datasets. There is currently a critical need to find novel methods to quantify and evaluate the broader impact of research. This study will help scientists better comprehend the emotional impact of their work. Additionally, the value of understanding the public's interest and reactions helps science communicators identify effective ways to engage with the public and build positive connections between scientific communities and the public.
Originality/value
This study will extend work on public engagement with science, sociology of science, and computational social science. It will enable researchers to identify areas in which there is a gap between public and expert understanding and provide strategies by which this gap can be bridged.
Collapse
|
20
|
Huang H, Zhu D, Wang X. Evaluating scientific impact of publications: combining citation polarity and purpose. Scientometrics 2021. [DOI: 10.1007/s11192-021-04183-8] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
21
|
|