A peek into the room where it happens: Quantifying ABA's influence on public policy discussions

To maximize its influence, applied behavior analysis must both create solutions and shape public policy to implement those solutions at scale. From the perspective of data-driven decision making, it is illogical to talk about seeking public policy influence without consulting evidence showing when influence has been achieved. One relevant form of evidence is the attention that behavioral solutions receive in published discussions about policy issues, and here I show how much of this attention has been earned by articles published in Journal of Applied Behavior Analysis. I also propose using the same kind of data to support finer grained analyses focusing on specific behavior problems, specific types of interventions, and the research programs of individual investigators. Although this is far from a complete account of the influence of applied behavior analysis on policy, it is better to have data than none if the goal is to transform the quest for influence on policy from a matter of speculation and casual discussion into an evidence-based practice.

Evaluating Research Impact Based on Semantic Scholar Highly Influential Citations, Total Citations, and Altmetric Attention Scores: The Quest for Refined Measures Remains Illusive

Background: The evaluation of scholarly articles’ impact has been heavily based on the citation metrics despite the limitations of this approach. Therefore, the quest for meticulous and refined measures to evaluate publications’ impact is warranted. Semantic Scholar (SS) is an artificial intelligence-based database that allegedly identifies influential citations defined as “Highly Influential Citations” (HICs). Citations are considered highly influential according to SS when the cited publication has a significant impact on the citing publication (i.e., the citer uses or extends the cited work). Altmetrics are measures of online attention to research mined from activity in online tools and environments. Aims: The current study aimed to explore whether SS HICs provide an added value when it comes to measuring research impact compared to total citation counts and Altmetric Attention Score (AAS). Methods: Dimensions was used to generate the dataset for this study, which included COVID-19-related scholarly articles published by researchers affiliated to Jordanian institutions. Altmetric Explorer was selected as an altmetrics harvesting tool, while Semantic Scholar was used to extract details related to HICs. A total of 618 publications comprised the final dataset. Results: Only 4.57% (413/9029) of the total SS citations compiled in this study were classified as SS HICs. Based on SS categories of citations intent, 2626 were background citations (29.08%, providing historical context, justification of importance, and/or additional information related to the cited paper), 358 were result citations (3.97%, that extend on findings from research that was previously conducted), and 263 were method citations (2.91%, that use the previously established procedures or experiments to determine whether the results are consistent with findings in related studies). No correlation was found between HICs and AAS (r = 0.094). Manual inspection of the results revealed substantial contradictions, flaws, and inconsistencies in the SS HICs tool. Conclusions: The use of SS HICs in gauging research impact is significantly limited due to the enigmatic method of its calculation and total dependence on artificial intelligence. Along with the already documented drawbacks of total citation counts and AASs, continuous evaluation of the existing tools and the conception of novel approaches are highly recommended to improve the reliability of publication impact assessment.

Sexually transmitted infections among key populations in India: A protocol for systematic review

BACKGROUND

Sexually transmitted infections (STIs) are one of the leading causes of health, and economic burdens in the developing world, leading to considerable morbidity, mortality, and stigma. The incidence and prevalence of the four curable STIs viz. syphilis, gonorrhoea, chlamydia, and trichomoniasis vary remarkably across different geographical locations. In India, the prevalence of four curable STI among general populations is in between 0 to 3.9 percent. However, it is assumed that STI prevalence is much higher among subpopulations practicing high-risk behaviour. Like men who have sex with men (MSM), transgender (TG), injecting drug users (IDU), and female sex workers (FSW).

OBJECTIVES

In the present circumstances, a systematic review is necessary to integrate the available data from previously published peer-reviewed articles and published reports from several competent authorities to provide the prevalence and geographical distribution of the four curable STIs among the key population of India.

METHODS

All available articles will be retrieved from PubMed, Google Scholar, Cochrane database, Scopus, Science Direct, and the Global Health network using the appropriate search terms. The data will be extracted through data extraction form as per PICOS (population, intervention, comparison, outcome, study design) framework. Risk of bias and quality assessment will be performed according to the situation with the help of available conventional protocol.

DISCUSSION

The future systematic review, generated from the present protocol, may provide evidence of the prevalence and geographical distribution of the four curable STIs among the key population of India. We hope that the findings of the future systematic review will strengthen the existing surveillance system in India, to determine the above-mention STIs prevalence among key populations in India. Protocol registration number: International Prospective Register for Systematic Reviews (PROSPERO) number CRD42022346470.

Relevance assessments, bibliometrics, and altmetrics: a quantitative study on PubMed and arXiv

Relevance is a key element for analyzing bibliometrics and information retrieval (IR). In both domains, relevance decisions are discussed theoretically and sometimes evaluated in empirical studies. IR research is often based on test collections for which explicit relevance judgments are made, while bibliometrics is based on implicit relevance signals like citations or other non-traditional quantifiers like altmetrics. While both types of relevance decisions share common concepts, it has not been empirically investigated how they relate to each other on a larger scale. In this work, we compile a new dataset that aligns IR relevance judgments with traditional bibliometric relevance signals (and altmetrics) for life sciences and physics publications. The dataset covers PubMed and arXiv articles, for which relevance judgments are taken from TREC Precision Medicine and iSearch, respectively. It is augmented with bibliometric data from the Web of Science and Altmetrics. Based on the reviewed literature, we outline a mental framework supporting the answers to our research questions. Our empirical analysis shows that bibliometric (implicit) and IR (explicit) relevance signals are correlated. Likewise, there is a high correlation between biblio- and altmetrics, especially for documents with explicit positive relevance judgments. Furthermore, our cross-domain analysis demonstrates the presence of these relations in both research fields.

The prevalence and impact of university affiliation discrepancies between four bibliographic databases – Scopus, Web of Science, Dimensions, and Microsoft Academic

Research managers benchmarking universities against international peers face the problem of affiliation disambiguation. Different databases have taken separate approaches to this problem and discrepancies exist between them. Bibliometric data sources typically conduct a disambiguation process that unifies variant institutional names and those of its sub-units so that researchers can then search all records from that institution using a single unified name. This study examined affiliation discrepancies between Scopus, Web of Science, Dimensions, and Microsoft Academic for 18 Arab universities over a five-year period. We confirmed that digital object identifiers (DOIs) are suitable for extracting comparable scholarly material across databases and quantified the affiliation discrepancies between them. A substantial share of records assigned to the selected universities in any one database were not assigned to the same university in another. The share of discrepancy was higher in the larger databases, Dimensions and Microsoft Academic. The smaller, more selective databases, Scopus and especially Web of Science tended to agree to a greater degree with affiliations in the other databases. Manual examination of affiliation discrepancies showed they were caused by a mixture of missing affiliations, unification differences, and assignation of records to the wrong institution.

Peer Review

https://publons.com/publon/10.1162/qss_a_00175

Article-Level Metrics

In the era of digitization and Open Access, article-level metrics are increasingly employed to distinguish influential research works and adjust research management strategies. Tagging individual articles with digital object identifiers allows exposing them to numerous channels of scholarly communication and quantifying related activities. The aim of this article was to overview currently available article-level metrics and highlight their advantages and limitations. Article views and downloads, citations, and social media metrics are increasingly employed by publishers to move away from the dominance and inappropriate use of journal metrics. Quantitative article metrics are complementary to one another and often require qualitative expert evaluations. Expert evaluations may help to avoid manipulations with indiscriminate social media activities that artificially boost altmetrics. Values of article metrics should be interpreted in view of confounders such as patterns of citation and social media activities across countries and academic disciplines.

The Pertinent Literature of Enhanced Recovery after Surgery Programs: A Bibliometric Approach

Background and Objectives: The programs of enhanced recovery after surgery are the new revolution in surgical departments; however, features of this concept have not been systematically explored. Therefore, the purpose of this study was to explore Enhanced recovery after surgery (ERAS)-related research using bibliometric analysis. Materials and Methods: The search strategy of ERAS programs was conducted in the Web of Science database. Bibliometric analysis was further performed by Excel and Bibliometrix software. The relationship between citation counts and Mendeley readers was assessed by linear regression analysis. Results: 8539 studies from 1994-2019 were included in the present research, with reporting studies originating from 91 countries using 18 languages. The United States (US) published the greatest number of articles. International cooperation was discovered in 82 countries, with the most cooperative country being the United Kingdom. Henrik Kehlet was found to have published the highest number of studies. The journal Anesthesia and Analgesia had the largest number of articles. Linear regression analysis presented a strong positive correlation between citations and Mendeley readers. Most research was related to gastrointestinal surgery in this field. Conclusion: This bibliometric analysis shows the current status of ERAS programs from multiple perspectives, and it provides reference and guidance to scholars for further research.

COVID-19 publications: Database coverage, citations, readers, tweets, news, Facebook walls, Reddit posts

The COVID-19 pandemic requires a fast response from researchers to help address biological, medical, and public health issues to minimize its impact. In this rapidly evolving context, scholars, professionals, and the public may need to identify important new studies quickly. In response, this paper assesses the coverage of scholarly databases and impact indicators during March 21, 2020 to April 18, 2020. The rapidly increasing volume of research is particularly accessible through Dimensions, and less through Scopus, the Web of Science, and PubMed. Google Scholar’s results included many false matches. A few COVID-19 papers from the 21,395 in Dimensions were already highly cited, with substantial news and social media attention. For this topic, in contrast to previous studies, there seems to be a high degree of convergence between articles shared in the social web and citation counts, at least in the short term. In particular, articles that are extensively tweeted on the day first indexed are likely to be highly read and relatively highly cited 3 weeks later. Researchers needing wide scope literature searches (rather than health-focused PubMed or medRxiv searches) should start with Dimensions (or Google Scholar) and can use tweet and Mendeley reader counts as indicators of likely importance.

A case study exploring associations between popular media attention of scientific research and scientific citations

The association between mention of scientific research in popular media (e.g., the mainstream media or social media platforms) and scientific impact (e.g., citations) has yet to be fully explored. The purpose of this study was to clarify this relationship, while accounting for some other factors that likely influence scientific impact (e.g., the reputations of the scientists conducting the research and academic journal in which the research was published). To accomplish this purpose, approximately 800 peer-reviewed articles describing original research were evaluated for scientific impact, popular media attention, and reputations of the scientists/authors and publication venue. A structural equation model was produced describing the relationship between non-scientific impact (popular media) and scientific impact (citations), while accounting for author/scientist and journal reputation. The resulting model revealed a strong association between the amount of popular media attention given to a scientific research project and corresponding publication and the number of times that publication is cited in peer-reviewed scientific literature. These results indicate that (1) peer-reviewed scientific publications receiving more attention in non-scientific media are more likely to be cited than scientific publications receiving less popular media attention, and (2) the non-scientific media is associated with the scientific agenda. These results may inform scientists who increasingly use popular media to inform the general public and scientists concerning their scientific work. These results might also inform administrators of higher education and research funding mechanisms, who base decisions partly on scientific impact.

An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics

Sufficient data presence is one of the key preconditions for applying metrics in practice. Based on both Altmetric.com data and Mendeley data collected up to 2019, this paper presents a state-of-the-art analysis of the presence of 12 kinds of altmetric events for nearly 12.3 million Web of Science publications published between 2012 and 2018. Results show that even though an upward trend of data presence can be observed over time, except for Mendeley readers and Twitter mentions, the overall presence of most altmetric data is still low. The majority of altmetric events go to publications in the fields of Biomedical and Health Sciences, Social Sciences and Humanities, and Life and Earth Sciences. As to research topics, the level of attention received by research topics varies across altmetric data, and specific altmetric data show different preferences for research topics, on the basis of which a framework for identifying hot research topics is proposed and applied to detect research topics with higher levels of attention garnered on certain altmetric data source. Twitter mentions and policy document citations were selected as two examples to identify hot research topics of interest of Twitter users and policy-makers, respectively, shedding light on the potential of altmetric data in monitoring research trends of specific social attention.

Google Books, Scopus, Microsoft Academic and Mendeley for impact assessment of doctoral dissertations: A multidisciplinary analysis of the UK

A research doctorate normally culminates in publishing a dissertation reporting a substantial body of novel work. In the absence of a suitable citation index, this article explores the relative merits of alternative methods for the large-scale assessment of dissertation impact, using 150,740 UK doctoral dissertations from 2009–2018. Systematic methods for this were designed for Google Books, Scopus, Microsoft Academic, and Mendeley. Less than 1 in 8 UK doctoral dissertations had at least one Scopus (12%), Microsoft Academic (11%), or Google Books citation (9%), or at least one Mendeley reader (5%). These percentages varied substantially by subject area and publication year. Google Books citations were more common in the Arts and Humanities (18%), whereas Scopus and Microsoft Academic citations were more numerous in Engineering (24%). In the Social Sciences, Google Books (13%) and Scopus (12%) citations were important and in Medical Sciences, Scopus and Microsoft Academic citations to dissertations were rare (6%). Few dissertations had Mendeley readers (from 3% in Science to 8% in the Social Sciences) and further analysis suggests that Google Scholar finds more citations, but does not report information about all dissertations within a repository and is not a practical tool for large-scale impact assessment

Mendeley reader counts for US computer science conference papers and journal articles

Although bibliometrics are normally applied to journal articles when used to support research evaluations, conference papers are at least as important in fast-moving computing-related fields. It is therefore important to assess the relative advantages of citations and altmetrics for computing conference papers to make an informed decision about which, if any, to use. This paper compares Scopus citations with Mendeley reader counts for conference papers and journal articles that were published between 1996 and 2018 in 11 computing fields and that had at least one US author. The data showed high correlations between Scopus citation counts and Mendeley reader counts in all fields and most years, but with few Mendeley readers for older conference papers and few Scopus citations for new conference papers and journal articles. The results therefore suggest that Mendeley reader counts have a substantial advantage over citation counts for recently published conference papers due to their greater speed, but are unsuitable for older conference papers.

Disciplinary differences of the impact of altmetric

The main objective of this work was to group altmetric indicators according to their relationships and detect disciplinary differences with regard to altmetric impact in a set of 3793 research articles published in 2013. Three of the most representative altmetric providers (Altmetric, PlumX and Crossref Event Data) and Scopus were used to extract information about these publications and their metrics. Principal component analysis was used to summarize the information on these metrics and detect groups of indicators. The results show that these metrics can be grouped into three components: social media, gathering metrics from social networks and online media; usage, including metrics on downloads and views; and citations and saves, grouping metrics related to research impact and saves in bookmarking sites. With regard to disciplinary differences, articles in the General category attract more attention from social media, Social Sciences articles have higher usage than Physical Sciences, and General articles are more cited and saved than Health Sciences and Social Sciences articles.

Comparing the topological rank of journals in Web of Science and Mendeley

Recently, there has been a surge of interest in new data emerged due to the rapid development of the information technologies in scholarly communication. Since the 2010s, altmetrics has become a common trend in scientometric research. However, researchers have not treated in much detail the question of the probability distributions underlying these new data. The principal objective of this study was to investigate one of the classic problems of scientometrics-the problem of citation and readership distributions. The study is based on the data obtained from two information systems: Web of Science and Mendeley. Here we based on the concept of the cumulative empirical distribution function to explore the differences and similarities between citations and readership counts of biological journals indexed in Web of Science and Mendeley. The basic idea was to determine, for any journal, a "size" (it is said to be the topological rank) of citation and readership empirical cumulative distributions, and then to compare distributions of the topological ranks of Web of Science and Mendeley. In order to verify our model, we employ it to the bibliometric and altmetric research of 305 biological journals indexed in Journal Citation Reports 2015. The findings show that both distributions of the topological rank of biological journals are statistically close to the Wakeby distribution. The findings presented in this study add to our understanding of information processes of the scholarly communication in the new digital environment.

Visibilidad e impacto altmétrico de los investigadores de la Universidad de Antioquia: metodología aplicable a universidades

Resumen Este trabajo es resultado de uno de los componentes de una investigación macro que busca crear un Modelo para identifi car el grado de vinculación de una universidad con su entorno, en este caso, aplicado a la Universidad de Antioquia (Medellín-Colombia). Este componente se refi ere a la visibilidad e impacto de los investigadores desde la perspectiva y datos que ofrecen las altmetrics.Para ello, se analizan los datos altmétricos de 1.032 investigadores de seis áreas del conocimiento, considerando diferentes plataformas académicas, profesionales y sociales, además de una plataforma integradora de indicadores altmétricos. Aunque se realiza esta medición para investigadores de la Universidad de Antioquia, la metodología general para la captura e interpretación de datos puede aplicarse a otras universidades que comparten características de investigación y de comunicación científi ca y que, a partir de las altmetrics, desean identifi car la visibilidad e impacto de sus investigadores y la vinculación misma de lainstitución con su entorno, para complementar, de esta manera, las mediciones bibliométricas tradicionales u otras mediciones del entorno de la universidad, como lo considera, por ejemplo, el Manual de Valencia.

On the quest for currencies of science

Purpose

The introduction of “altmetrics” as new tools to analyze scientific impact within the reward system of science has challenged the hegemony of citations as the predominant source for measuring scientific impact. Mendeley readership has been identified as one of the most important altmetric sources, with several features that are similar to citations. The purpose of this paper is to perform an in-depth analysis of the differences and similarities between the distributions of Mendeley readership and citations across fields.

Design/methodology/approach

The authors analyze two issues by using in each case a common analytical framework for both metrics: the shape of the distributions of readership and citations, and the field normalization problem generated by differences in citation and readership practices across fields. In the first issue the authors use the characteristic scores and scales method, and in the second the measurement framework introduced in Crespo et al. (2013).

Findings

There are three main results. First, the citations and Mendeley readership distributions exhibit a strikingly similar degree of skewness in all fields. Second, the results on “exchange rates (ERs)” for Mendeley readership empirically supports the possibility of comparing readership counts across fields, as well as the field normalization of readership distributions using ERs as normalization factors. Third, field normalization using field mean readerships as normalization factors leads to comparably good results.

Originality/value

These findings open up challenging new questions, particularly regarding the possibility of obtaining conflicting results from field normalized citation and Mendeley readership indicators; this suggests the need for better determining the role of the two metrics in capturing scientific recognition.