Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?

MAATrica: a measure for assessing consistency and methods in medicinal and nutraceutical chemistry papers

The growing number of scientific papers and document sources underscores the need for methods capable of evaluating the quality of publications. Researchers who are looking for relevant papers for their studies need ways to assess the scientific value of these documents. One approach involves using semantic search engines that can automatically extract important knowledge from the growing body of text. In this study, we introduce a new metric called "MAATrica," which serves as the foundation for an innovative method designed to evaluate research papers. MAATrica offers a new way to analyze and categorize text, focusing on the consistency of research documents in the life sciences, particularly in the fields of medicinal and nutraceutical chemistry. This method utilizes semantic descriptions to cover in silico experiments, as well as in vitro and in vivo essays. Created to aid in evaluation processes like peer review, MAATrica uses toolkits and semantic applications to build the proposed measure, identify scientific entities, and gather information. We have applied MAATrica to roughly 90,000 papers and present our findings here.

How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal

BACKGROUND

Peer review represents a cornerstone of the scientific process, yet few studies have evaluated its association with scientific impact. The objective of this study is to assess the association of peer review scores with measures of impact for manuscripts submitted and ultimately published.

METHODS

3173 manuscripts submitted to Regional Anesthesia & Pain Medicine (RAPM) between August 2018 and October 2021 were analyzed, with those containing an abstract included. Articles were categorized by topic, type, acceptance status, author demographics and open-access status. Articles were scored based on means for the initial peer review where each reviewer's recommendation was assigned a number: 5 for 'accept', 3 for 'minor revision', 2 for 'major revision' and 0 for 'reject'. Articles were further classified by whether any reviewers recommended 'reject'. Rejected articles were analyzed to determine whether they were subsequently published in an indexed journal, and their citations were compared with those of accepted articles when the impact factor was <1.4 points lower than RAPM's 5.1 impact factor. The main outcome measure was the number of Clarivate citations within 2 years from publication. Secondary outcome measures were Google Scholar citations within 2 years and Altmetric score.

RESULTS

422 articles met inclusion criteria for analysis. There was no significant correlation between the number of Clarivate 2-year review citations and reviewer rating score (r=0.038, p=0.47), Google Scholar citations (r=0.053, p=0.31) or Altmetric score (p=0.38). There was no significant difference in 2-year Clarivate citations between accepted (median (IQR) 5 (2-10)) and rejected manuscripts published in journals with impact factors >3.7 (median 5 (2-7); p=0.39). Altmetric score was significantly higher for RAPM-published papers compared with RAPM-rejected ones (median 10 (5-17) vs 1 (0-2); p<0.001).

CONCLUSIONS

Peer review rating scores were not associated with citations, though the impact of peer review on quality and association with other metrics remains unclear.

Bibliometrics, the "Clinical Diagnostics" of Research Impact Comment on "'We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research': A Research Impact Evaluation at a Regional Australian Hospital and Health Service"

Following the Townsville Hospital and Health Service (THHS) strategic revision as a "research-based" institution, Brown et al have investigated the impact of THHS research, and its key drivers, based on 15 stakeholder interviews and two quantitative indicators. This commentary argues that the quantitative analyses and findings would have benefitted from applying evaluative bibliometrics, hopefully, conducted by experienced bibliometricians. We present the potential of bibliometrics for assessing the scholarly impact of research, as well as a few examples of its application to the case of THHS, for informing research policies and strategies.

Excellence or Misconduct: How the Visibility of Team Leaders Impacts the Research Project Competition in the Republic of Moldova?

Introduction. Distributing public funds to the “best” researchers is a key element of the science policy. Evaluation is a fundamental activity for the allocation of competitive funding. The flaws of peer review have led to increased interest in the use of bibliometric indicators for the evaluation of the research project proposals.Problem Statement. The advantajes and advance of bibliometrc is stimulated interest toward the correlation of peer review and applicants’ bibliometric indicators. The results of such studies are different and heterogeneous. Such studies are insufficient in Eastern Europe.Purpose. To establish the correlation between peer review and bibliometric indicators of project team leaders within the call for research projects in Moldova, which are financed from public funds for 2020—2023.Material and Methods. Statistical correlation of the results of national competition of R&D proposals (evaluation and funding) and the bibliometrics indicators of project team leaders (publications ant patents); analytical analysis of the contextual factors influencing this correlation.Results. The results of the analysis have shown a positive, albeit weak correlation between the scores assigned by experts and the previous performances of leaders. The most significant relation is between the call results and the Hirsh index in Web of Science and Scopus databases. However, the projects proposed by the most cited researchers in WoS and Scopus or the founders of scientific schools did not receive funding.Conclusions. The analysis of the national R&D competition has proved that previous scientific performance of team leaders influenced the evaluation results and the funding of project proposals. However, these dependencies are not linear and seem to be affected by the conflicts of interest and “old boys” schemes. This fact calls for significant changes of the process: ensuring the transparency, the involvement of foreign experts and the use of bibliometric indicators in evaluation.

Sentinel animals for monitoring the environmental lead exposure: combination of traditional review and visualization analysis

In nature, certain animals share a common living environment with humans, thus these animals have become biomonitors of health effects related to various environmental exposures. As one of the most toxic environmental chemicals, lead (Pb) can cause detriment health effects to animals, plants, and even humans through different exposure pathways such as atmosphere, soil, food, water, and dust, etc. Sentinel animals played an "indicative" role in the researches of environmental pollution monitoring and human health. In order to comprehend the usage of sentinel animals in the indication of environmental Pb pollution and human Pb exposure completely, a combination of traditional review and visualization analysis based on CiteSpace literature was used to review earlier researches in this study. In the first instance, present researches on exposure sources and exposure pathways of Pb were summarized briefly, and then the studies using sentinel animals to monitor environmental heavy metal pollution and human health were combed. Finally, visualization software CiteSpace 5.8.R3 was used to explore and analyze the hotspots and frontiers of lead exposure and sentinel animals researches at home and abroad. The results showed that certain mammals were good indicators for human lead exposure. Sentinel animals had been widely used to monitor the ecological environment and human lead exposure. Among them, the blood lead levels of small mammals, particularly for domestic dogs and cats, had a significant correlation with the blood lead levels of human living in the same environment. It indicated that certain biological indicators in animals can be used as surrogates to monitor human body exposure to heavy metals. This study also explored the challenges and perspectives that may be faced in sentinel animal research, in order to provide a certain theoretical basis and train of thought guidance for future research.

Insights into Circular Horticulture: Knowledge Diffusion, Resource Circulation, One Health Approach, and Greenhouse Technologies

The integration of the circular economy in agriculture has promoted sustainable innovation in food production systems such as horticulture. The present paper illustrates how horticulture is transitioning to the circular economy. This research field's performance approaches and trends were assessed through a bibliometric and text-mining analysis of the literature. Our findings revealed that circular horticulture is a recent research field that is constantly growing. Its approach has been neither systemic nor integrative but fragmented. Bioeconomy, urban agriculture, recycled nutrients, biochar, fertigation, and desalination have been positioned as research hotspots. Vegetables and fruits are the most studied crops. Resource circulation has focused primarily on biowaste recovery to provide benefits such as biofertilizers and linear-substrate substitutes, and on water reuse for the establishment of hydroponic systems. The One Health approach is scarcely explored and, therefore, weakly articulated, wherein the absence of assessment methodologies encompassing the health of ecosystems, animals, and people is a notable limitation. Science-policy interfaces between One Health and food systems need to be improved. Lastly, greenhouse technologies are aligned with bioenergy, sustainable materials, and sensing technologies. Challenges and directions for future research have been raised to promote the redesign of horticultural production systems, integrating long-term circularity.

Publishing Industry: A Bibliometric Analysis of the Scientific Production Indexed in Scopus

The general goal of this work is to carry out a bibliometric analysis of the scientific production in the publishing industry between 2012 and 2022. For this purpose, the following research posed the following questions: (i) what are the leading academic publications that collect scientific production around the publishing industry? (ii) who are the most productive and influential authors in research on the publishing industry? (iii) from which countries do the published academic works come?, and (iv) in which universities are research on the publishing industry concentrated? This research used the information available in Scopus to address this bibliometric analysis. The analysis conducted in this work is exploratory, descriptive, and quantitative, based on the techniques and tools of bibliometric analysis of the documents stored in the Scopus bibliographic database. This article highlights that research on the global book publishing market is interdisciplinary and, therefore, highly cross-cutting. The economic dimension of the publishing process, and the history and culture of the book dominated the study subjects. There is also a growing trend of research on the impact of new technologies on the value chain and book distribution, without forgetting the increasing studies on new business models in the publishing industry.

Research Trends on Climate Change and Circular Economy from a Knowledge Mapping Perspective

The circular economy (CE) has been proposed as a potentially significant catalyst to enhance the current response to the global climate crisis. The objective of this study was to investigate the scientific literature of the research between climate change and CE adopting a knowledge mapping approach. Based on a total of 789 peer-reviewed publications extracted from Scopus, we found that research on climate change and CE is continually growing and interdisciplinary in nature. Europe notably leads scientific production. Keyword evolution shows that CE has been influenced by more lines of research than climate change. We also found that waste management is the CE approach most associated with climate change, mitigation is the climate action most impacted by CE, and food is the most reported greenhouse gas (GHG)-emitting material. However, there are knowledge gaps in the integration of the social dimension, the promotion of climate change adaptation, and the association of sustainable development goal (SDG) 13. Finally, we identified four potentially valuable directions for future studies: (i) CE practices, (ii) bioeconomy, (iii) climate and energy, and (iv) sustainability and natural resources, in which carbon recovery technologies, green materials, regional supply chains, circular agriculture models, and nature-based solutions are promising themes.

Do extraordinary science and technology scientists balance their publishing and patenting activities?

This study investigated whether 12 scientists who had received the National Medal of Science and the National Medal of Technology and Innovation balanced publishing and patenting activities. The results demonstrated that although the scientist were recognized for their contributions to science and technology, the majority of recipients were not prolific researchers, and some were not influential. Notably, one scientist had not been granted a single patent. This indicated that scientific and technological contributions may not necessarily correspond with influential scientific publications and patents. Moreover, only two scientists had filed for patents before publishing, and they also invested more time developing technological inventions. Most recipients were science- or technology-oriented scientists. Few scientists balanced their publishing and patenting activities, and demonstrated excellent research and technology performance.

Producción científica sobre ansiedad bibliotecaria: un análisis bibliométrico y cienciométrico desde Scopus

La ansiedad bibliotecaria (AB) es el estado de incertidumbre que experimentan algunos usuarios cuando acuden a la biblioteca, cuyo estudio comenzó en los años 80, pero que carece de un análisis métrico actualizado. El objetivo de este estudio fue hacer un análisis bibliométrico y cienciométrico de los artículos sobre AB indizados en la base de datos Scopus entre 1989 y 2018. Se analizaron 118 publicaciones según indicadores de producción, colaboración, impacto y mapas bibliométricos. Se encontró que las publicaciones sobre AB han aumentado progresivamente, que la colaboración de autores fue baja, que una revista influyente en el área fue Library Review, y que los autores con más impacto fueron Onwuegbuzie y Jiao. Asimismo, destacaron palabras clave sobre la formación del usuario y cuatro frentes de investigación: instrumentos de medición, resultados, marcos teóricos y tipos de unidades de análisis. Dichos resultados sugieren que la AB es un tema fructífero de investigación.

Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World

Nowadays, the importance of bibliographic databases (DBs) has increased enormously, as they are the main providers of publication metadata and bibliometric indicators universally used both for research assessment practices and for performing daily tasks. Because the reliability of these tasks firstly depends on the data source, all users of the DBs should be able to choose the most suitable one. Web of Science (WoS) and Scopus are the two main bibliographic DBs. The comprehensive evaluation of the DBs’ coverage is practically impossible without extensive bibliometric analyses or literature reviews, but most DBs users do not have bibliometric competence and/or are not willing to invest additional time for such evaluations. Apart from that, the convenience of the DB’s interface, performance, provided impact indicators and additional tools may also influence the users’ choice. The main goal of this work is to provide all of the potential users with an all-inclusive description of the two main bibliographic DBs by gathering the findings that are presented in the most recent literature and information provided by the owners of the DBs at one place. This overview should aid all stakeholders employing publication and citation data in selecting the most suitable DB.