Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise

A global exploratory comparison of country self-citations 1996-2019

Self-citations are a key topic in evaluative bibliometrics because they can artificially inflate citation-related performance indicators. Recently, self-citations defined at the largest scale, i.e., country self-citations, have started to attract the attention of researchers and policymakers. According to a recent research, in fact, the anomalous trends in the country self-citation rates of some countries, such as Italy, have been induced by the distorting effect of citation metrics-centered science policies. In the present study, we investigate the trends of country self-citations in 50 countries over the world in the period 1996-2019 using Scopus data. Results show that for most countries country self-citations have decreased over time. 12 countries (Colombia, Egypt, Indonesia, Iran, Italy, Malaysia, Pakistan, Romania, Russian Federation, Saudi Arabia, Thailand, and Ukraine), however, exhibit different behavior, with anomalous trends of self-citations. We argue that these anomalies should be attributed to the aggressive science policies adopted by these countries in recent years, which are all characterized by direct or indirect incentives for citations. Our analysis confirms that when bibliometric indicators are integrated into systems of incentives, they are capable of affecting rapidly and visibly the citation behavior of entire countries.

Excellence or Misconduct: How the Visibility of Team Leaders Impacts the Research Project Competition in the Republic of Moldova?

Introduction. Distributing public funds to the “best” researchers is a key element of the science policy. Evaluation is a fundamental activity for the allocation of competitive funding. The flaws of peer review have led to increased interest in the use of bibliometric indicators for the evaluation of the research project proposals.Problem Statement. The advantajes and advance of bibliometrc is stimulated interest toward the correlation of peer review and applicants’ bibliometric indicators. The results of such studies are different and heterogeneous. Such studies are insufficient in Eastern Europe.Purpose. To establish the correlation between peer review and bibliometric indicators of project team leaders within the call for research projects in Moldova, which are financed from public funds for 2020—2023.Material and Methods. Statistical correlation of the results of national competition of R&D proposals (evaluation and funding) and the bibliometrics indicators of project team leaders (publications ant patents); analytical analysis of the contextual factors influencing this correlation.Results. The results of the analysis have shown a positive, albeit weak correlation between the scores assigned by experts and the previous performances of leaders. The most significant relation is between the call results and the Hirsh index in Web of Science and Scopus databases. However, the projects proposed by the most cited researchers in WoS and Scopus or the founders of scientific schools did not receive funding.Conclusions. The analysis of the national R&D competition has proved that previous scientific performance of team leaders influenced the evaluation results and the funding of project proposals. However, these dependencies are not linear and seem to be affected by the conflicts of interest and “old boys” schemes. This fact calls for significant changes of the process: ensuring the transparency, the involvement of foreign experts and the use of bibliometric indicators in evaluation.

Peer review in research assessment and data analysis of Italian publications in SSD M-STO/08 (Archival science, bibliography, library science)

Since the introduction of research assessment systems at institutional level in the 1980s, the ongoing debate on the roles and functions of peer review and bibliometrics has been vivid and lively. In the first part of the article, the main lines over time of this debate are traced, and a reflection on the epistemic functions of peer review and citations is proposed. In Italy, the first research assessment exercise (VTR) was based on peer review only, while the following ones (VQR) were based on different methods for bibliometric disciplines and non-bibliometric disciplines, namely bibliometric indicators and peer review. Starting from a data analysis on Italian publications, and using as a sample data from M-STO/08 (Archival science, bibliography and library science) area, the essay shows some trends and changes in publication habits in HSS. Conclusions open a perspective on revitalization of peer review as a solid qualitative method for research assessment.

The impact of Italian performance-based research funding systems on the intensity of international research collaboration

The study of national research assessment exercises serves to evaluate the effectiveness of policies versus their objectives and to improve the formulation of future initiatives. The aim of the current study is to verify whether the introduction of the first performance-based research funding in Italy, based on the 2004–10 VQR assessment, achieved the intended objective of inducing greater international collaboration on the part of researchers. For this, we apply a bibliometric approach based on the observation of coauthorships in Italian and worldwide scientific publications over a 14-year period, beginning in the target years of the VQR assessment. Through an Interrupted Time Series Analysis, we compare the expected and observed patterns of international coauthorship for Italy and the rest of the world. Although the rate of internationalization of Italian research is seen to be increasing, whether this is a consequence of the VQR incentives, or rather part of a global phenomenon of recourse to international collaboration in response to the increasingly complex scientific challenges, it is open to interpretation.

Just an artifact? The concordance between peer review and bibliometrics in economics and statistics in the Italian research assessment exercise

During the Italian research assessment exercise (2004–2010), the governmental agency (ANVUR) in charge of its realization performed an experiment on the concordance between peer review and bibliometrics at an individual article level. The computed concordances were at most weak for science, technology, engineering, and mathematics. The only exception was the moderate concordance found for the area of economics and statistics. In this paper, the disclosed raw data of the experiment are used to shed light on the anomalous results obtained for economics and statistics. In particular, the data permit us to document that the protocol of the experiment adopted for economics and statistics was different from the one used in the other areas. Indeed, in economics and statistics the same group of scholars developed the bibliometric ranking of journals for evaluating articles, managing peer reviews and forming the consensus groups for deciding the final scores of articles after having received the referee’s reports. This paper shows that the highest level of concordance in economics and statistics was an artifact mainly due to the role played by consensus groups in boosting the agreement between bibliometrics and peer review.

Do extraordinary science and technology scientists balance their publishing and patenting activities?

This study investigated whether 12 scientists who had received the National Medal of Science and the National Medal of Technology and Innovation balanced publishing and patenting activities. The results demonstrated that although the scientist were recognized for their contributions to science and technology, the majority of recipients were not prolific researchers, and some were not influential. Notably, one scientist had not been granted a single patent. This indicated that scientific and technological contributions may not necessarily correspond with influential scientific publications and patents. Moreover, only two scientists had filed for patents before publishing, and they also invested more time developing technological inventions. Most recipients were science- or technology-oriented scientists. Few scientists balanced their publishing and patenting activities, and demonstrated excellent research and technology performance.

The different responses of universities to introduction of performance-based research funding

Governments and organizations design performance-based research funding systems (PBRFS) for strategic aims, such as to selectively allocate scarce resources and stimulate research efficiency. In this work, we analyze the relative change in research productivity of Italian universities after the introduction of such a system, featuring financial and reputational incentives. Using a bibliometric approach, we compare the relative research performance of universities before and after the introduction of PBRFS, at the overall, discipline and field levels. The findings show convergence in the universities’ performance, due above all to the remarkable improvement of the lowest performers. Geographically, the universities of the south (vs central and northern Italy) achieved the greatest improvement in relative performance. The methodology, and results, should be of use to university management and policy-makers.

Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World

Nowadays, the importance of bibliographic databases (DBs) has increased enormously, as they are the main providers of publication metadata and bibliometric indicators universally used both for research assessment practices and for performing daily tasks. Because the reliability of these tasks firstly depends on the data source, all users of the DBs should be able to choose the most suitable one. Web of Science (WoS) and Scopus are the two main bibliographic DBs. The comprehensive evaluation of the DBs’ coverage is practically impossible without extensive bibliometric analyses or literature reviews, but most DBs users do not have bibliometric competence and/or are not willing to invest additional time for such evaluations. Apart from that, the convenience of the DB’s interface, performance, provided impact indicators and additional tools may also influence the users’ choice. The main goal of this work is to provide all of the potential users with an all-inclusive description of the two main bibliographic DBs by gathering the findings that are presented in the most recent literature and information provided by the owners of the DBs at one place. This overview should aid all stakeholders employing publication and citation data in selecting the most suitable DB.

On the agreement between bibliometrics and peer review: Evidence from the Italian research assessment exercises

This paper analyzes the concordance between bibliometrics and peer review. It draws evidence from the data of two experiments of the Italian governmental agency for research evaluation. The experiments were performed by the agency for validating the adoption in the Italian research assessment exercises of a dual system of evaluation, where some outputs were evaluated by bibliometrics and others by peer review. The two experiments were based on stratified random samples of journal articles. Each article was scored by bibliometrics and by peer review. The degree of concordance between the two evaluations is then computed. The correct setting of the experiments is defined by developing the design-based estimation of the Cohen's kappa coefficient and some testing procedures for assessing the homogeneity of missing proportions between strata. The results of both experiments show that for each research areas of science, technology, engineering and mathematics the degree of agreement between bibliometrics and peer review is-at most-weak at an individual article level. Thus, the outcome of the experiments does not validate the use of the dual system of evaluation in the Italian research assessments. More in general, the very weak concordance indicates that metrics should not replace peer review at the level of individual article. Hence, the use of the dual system in a research assessment might worsen the quality of information compared to the adoption of peer review only or bibliometrics only.

Are Italian research assessment exercises size-biased?

Research assessment exercises have enjoyed ever-increasing popularity in many countries in recent years, both as a method to guide public funds allocation and as a validation tool for adopted research support policies. Italy’s most recently completed evaluation effort (VQR 2011–14) required each university to submit to the Ministry for Education, University, and Research (MIUR) 2 research products per author (3 in the case of other research institutions), chosen in such a way that the same product is not assigned to two authors belonging to the same institution. This constraint suggests that larger institutions, where collaborations among colleagues may be more frequent, could suffer a size-related bias in their evaluation scores. To validate our claim, we investigate the outcome of artificially splitting Sapienza University of Rome, one of the largest universities in Europe, in a number of separate partitions, according to several criteria, noting significant score increases for several partitioning scenarios.

Determinants of quality of research environment: An assessment of the environment submissions in the UK’s Research Excellence Framework in 2014

One of the assessed research elements in the UK’s Research Excellence Framework (REF) exercise in 2014 was the research environment. The quality of the research environment was assessed by expert peer reviewers who were given a set of quantitative factors to support their decision making. However, there is no systematic procedure to integrate this quantitative information into the evaluation process. This article evaluates the relevance of quantitative factors in explaining the assessed quality of the research environment. Findings suggest submitting units with high external research income generation tend to have a better research environment evaluation in almost all the assessed subject areas. The importance given by reviewers to similar quantitative factors was distinctively different in two units of assessment (UoA) in which the evaluation criteria were the same, which highlights the internal inconsistency of the peer review evaluation. Our findings also confirm the existence of the ‘halo effect’ in some UoA where submitting units that belong to the Russell group and have sub-panel members in the REF exercise obtained higher scores even after controlling for the assessed quantitative factors.

On the evolution of the glass ceiling in Italian academia: the case of economics

Following an international trend, Italy has reformed its university system, especially concerning methods and tools for research evaluation, which are increasingly focused on a number of bibliometric indexes. To study the effects of these changes, we analyze the changing profiles of economists who have won competitions for full professorship in the last few decades in the country. We concentrate on individual characteristics and on scientific production. We show that the identification of a univocal and standardized concept of "research quality" within the new research assessments has progressively imposed a strategy of "homologation," especially for women. We find that women economists are at a higher risk of discrimination than their male colleagues and thus they are more likely to conform their research activities to the standardized profile imposed by the gender-blind application of simplistic bibliometric methods.

Citation gaming induced by bibliometric evaluation: A country-level comparative analysis

It is several years since national research evaluation systems around the globe started making use of quantitative indicators to measure the performance of researchers. Nevertheless, the effects on these systems on the behavior of the evaluated researchers are still largely unknown. For investigating this topic, we propose a new inwardness indicator able to gauge the degree of scientific self-referentiality of a country. Inwardness is defined as the proportion of citations coming from the country over the total number of citations gathered by the country. A comparative analysis of the trends for the G10 countries in the years 2000-2016 reveals a net increase of the Italian inwardness. Italy became, both globally and for a large majority of the research fields, the country with the highest inwardness and the lowest rate of international collaborations. The change in the Italian trend occurs in the years following the introduction in 2011 of national regulations in which key passages of professional careers are governed by bibliometric indicators. A most likely explanation of the peculiar Italian trend is a generalized strategic use of citations in the Italian scientific community, both in the form of strategic author self-citations and of citation clubs. We argue that the Italian case offers crucial insights on the constitutive effects of evaluation systems. As such, it could become a paradigmatic case in the debate about the use of indicators in science-policy contexts.

Theoretical research without projects

We propose a funding scheme for theoretical research that does not rely on project proposals, but on recent past scientific productivity. Given a quantitative figure of merit on the latter and the total research budget, we introduce a number of policies to decide the allocation of funds in each grant call. Under some assumptions on scientific productivity, some of such policies are shown to converge, in the limit of many grant calls, to a funding configuration that is close to the maximum total productivity of the whole scientific community. We present numerical simulations showing evidence that these schemes would also perform well in the presence of statistical noise in the scientific productivity and/or its evaluation. Finally, we prove that one of our policies cannot be cheated by individual research units. Our work must be understood as a first step towards a mathematical theory of the research activity.