Research assessment by percentile-based double rank analysis

The link between countries’ economic and scientific wealth has a complex dependence on technological activity and research policy

We studied the research performance of 69 countries by considering two different types of new knowledge: incremental (normal) and fundamental (radical). In principle, these two types of new knowledge should be assessed at two very different levels of citations, but we demonstrate that a simpler assessment can be performed based on the total number of papers (P) and the ratio of the number of papers in the global top 10% of most cited papers divided to the total number of papers (Ptop 10%/P). P represents the quantity, whereas the Ptop 10%/P ratio represents the efficiency. In ideal countries, P and the Ptop 10%/P ratio are linked to the gross domestic product (GDP) and GDP the per capita, respectively. Only countries with high Ptop 10%/P ratios participate actively in the creation of fundamental new knowledge and have Noble laureates. In real countries, the link between economic and scientific wealth can be modified by the technological activity and the research policy. We discuss how technological activity may decrease the Ptop 10%/P ratio while only slightly affecting the capacity to create fundamental new knowledge; in such countries, many papers may report incremental innovations that do not drive the advancement of knowledge. Japan is the clearest example of this, although there are many less extreme examples. Independently of technological activity, research policy has a strong influence on the Ptop 10%/P ratio, which may be higher or lower than expected from the GDP per capita depending on the success of the research policy.

Might Europe one day again be a global scientific powerhouse? Analysis of ERC publications suggests it will not be possible without changes in research policy

Numerous EU documents praise the excellence of EU research without empirical evidence and in contradiction of academic studies. We investigated research performance in two fields of high socioeconomic importance, advanced technology and basic medical research, in two sets of European countries, Germany, France, Italy, and Spain (GFIS), and the UK, Netherlands, and Switzerland (UKNCH). Despite their historical and geographical proximity, research performance in GFIS is much lower than in UKNCH, and well below the world average. Funding from the European Research Council (ERC) greatly improves performance in both GFIS and UKNCH, but ERC-GFIS publications are less cited than ERC-UKNCH publications. We conclude that research performance in GFIS and in other EU countries is intrinsically low, even in highly selected and generously funded projects. The technological and economic future of the EU depends on improving research, which requires structural changes in research policy within the EU, and in most EU countries.

Like-for-like bibliometric substitutes for peer review: Advantages and limits of indicators calculated from the ep index

The use of bibliometric indicators would simplify research assessments. The 2014 Research Excellence Framework (REF) is a peer review assessment of UK universities, whose results can be taken as benchmarks for bibliometric indicators. In this study, we use the REF results to investigate whether the ep index and a top percentile of most cited papers could substitute for peer review. The probability that a random university’s paper reaches a certain top percentile in the global distribution of papers is a power of the ep index, which can be calculated from the citation-based distribution of university’s papers in global top percentiles. Making use of the ep index in each university and research area, we calculated the ratios between the percentage of 4-star-rated outputs in REF and the percentages of papers in global top percentiles. Then, we fixed the assessment percentile so that the mean ratio between these two indicators across universities is 1.0. This method was applied to four units of assessment in REF: Chemistry, Economics and Econometrics joined to Business and Management Studies, and Physics. Some relevant deviations from the 1.0 ratio could be explained by the evaluation procedure in REF or by the characteristics of the research field; other deviations need specific studies by experts in the research area. These results indicate that in many research areas the substitution of a top percentile indicator for peer review is possible. However, this substitution cannot be made straightforwardly; more research is needed to establish the conditions of the bibliometric assessment.