The objectivity of national research foundation peer review in South Africa assessed against bibliometric indexes

Evaluation of the national research foundation-rated researchers’ output at a South African university

Purpose

The purpose of this paper is to establish the research impact of the National Research Foundation (NRF)-rated researchers’ output at the North-West University (NWU), South Africa, from 2006 to 2017.

Design/methodology/approach

The study used bibliometrics and altmetrics methods to determine the production of research outputs and the impact of NWU’s NRF-rated researchers’ publications. Various tools including Google Scholar (GS), Web of Science (WoS), Scopus, ResearchGate (RG) and Mendeley were used to collect data. The citations in the three bibliographic databases were used as proxy for academic impact, while reads and readerships in RG and Mendeley were used to determine societal impact of the researchers. The Statistical Package for the Social Sciences (SPSS) was used to test the relationship between citations in the three bibliographic databases and reads and readerships in RG and Mendeley.

Findings

The main findings were that the majority of NWU’s NRF-rated researchers’ publications emanated from GS, followed by Scopus and then WoS. GS output also had more citations. There were 6,026 research outputs in RG which were read for 676,919 times and 5,850 in Mendeley with 142,621 readerships. Correlations between RG and all three bibliographic databases’ citations were scant. Strong relationships between the three bibliographic databases’ citations and Mendeley readerships were found.

Practical implications

Academic librarians who interact with researchers who would like to predict future academic impact of their documents can be advised to consider Mendeley readerships with some level of confidence compared to RG reads. These results point to the importance of constant self-evaluation by researchers to ensure that they have balanced profiles across the three main bibliographic databases that are also considered for ratings. These results point to the relevancy of GS to evaluate research beyond the academy.

Social implications

The fact that researchers are contributing research that seeks to improve the general welfare of the population (beyond the academy) is a positive sign as society look up to researchers and research to solve their socio-economic problems. Social media play an important role as they serve as indicators that indicators point to wider research impacts and wider access by many different groups of people including the members of society at large. They point to research that is accessible to not only researchers and those who have access to their research but also the society at large.

Originality/value

Although the practice of rating researchers is common in different research ecosystems, the researchers could not find any evidence of studies conducted using a combination of bibliometrics and altmetrics to asses rated researchers’ output. This study covers and compares social impact based on data obtained from two academic social media sites and three main bibliographic databases (GS, Scopus and WoS).

Peer review versus the h-index for evaluation of individual researchers in the biological sciences

Past performance is a key consideration when rationalising the allocation of grants and other opportunities to individual researchers. The National Research Foundation of South Africa (NRF) has long used a highly structured system of ‘rating’ the past performance of individual researchers. This system relies heavily on peer review, and has seldom been benchmarked against bibliometric measures of research performance such as Hirsch’s h-index. Here I use data for about 600 rated researchers in the biological sciences to evaluate the extent to which outcomes of peer review correspond to bibliometric measures of research performance. The analysis revealed that values of the h-index based on the Scopus database are typically 5–20 for researchers placed in the NRF’s C rating category (‘established’), 20–40 for those in the B rating category (‘considerable international recognition’) and >40 for those in the A rating category (‘leading international scholars’). Despite concerns that citation patterns differ among disciplines, the mean h-index per rating category was remarkably consistent across five different disciplines in the biological sciences, namely animal sciences, plant sciences, ecology, microbiology and biochemistry/genetics. This observation suggests that the NRF rating system is equitable in the sense that the outcomes of peer review are generally consistent with bibliometric measures of research performance across different disciplines in the biological sciences. However, the study did reveal some notable discrepancies which could reflect either bias in the peer-review process or shortcomings in the bibliometric measures, or both.