CERN: Past performance and future prospects

Machine learning in scientific grant review: algorithmically predicting project efficiency in high energy physics

As more objections have been raised against grant peer-review for being costly and time-consuming, the legitimate question arises whether machine learning algorithms could help assess the epistemic efficiency of the proposed projects. As a case study, we investigated whether project efficiency in high energy physics (HEP) can be algorithmically predicted based on the data from the proposal. To analyze the potential of algorithmic prediction in HEP, we conducted a study on data about the structure (project duration, team number, and team size) and outcomes (citations per paper) of HEP experiments with the goal of predicting their efficiency. In the first step, we assessed the project efficiency using Data Envelopment Analysis (DEA) of 67 experiments conducted in the HEP laboratory Fermilab. In the second step, we employed predictive algorithms to detect which team structures maximize the epistemic performance of an expert group. For this purpose, we used the efficiency scores obtained by DEA and applied predictive algorithms - lasso and ridge linear regression, neural network, and gradient boosted trees - on them. The results of the predictive analyses show moderately high accuracy (mean absolute error equal to 0.123), indicating that they can be beneficial as one of the steps in grant review. Still, their applicability in practice should be approached with caution. Some of the limitations of the algorithmic approach are the unreliability of citation patterns, unobservable variables that influence scientific success, and the potential predictability of the model.

Applied Social Science, Policy Science, and the Federal Government

This article examines competing strategies for supporting and utilizing applied social science and the policy sciences in public decisions and program operations. It argues that tension and conflict are the normal, expected state of affairs between decision makers and the social science and policy science communities. This is so because of very different strategic perspectives governing the scientific standing, decision utility, and political "morality"of applied social research and policy research. The article suggests conflict can be dampened, but never completely eliminated, by two things: (1) sequential research designs that simultaneously produce both partial scientific truths and information useful to decision makers, and (2) more attention to overall quality control and utilization possibilities. However, such improvements require changes in the incentive and value systems of social scientists, policy scientists, and decision makers. The engineering of change m these three communities is itself a formidable unsolved problem.

Eminence of scientists in the light of the h-index and other scientometric indicators

Scientometrics cannot offer a simple consistent method for measuring the scientific eminence of individuals. The h-index method introduced by Hirsch was found applicable for evaluating publications of senior scientists with similar publishing features, only. Some simple methods — using the number of citations and journal papers, and the number of citations obtained by the most frequently cited papers — are suggested and tested to demonstrate the advantages and disadvantages of such indexes. The results indicate that calculating scientometric indexes for individuals, self-citations should be excluded and the effect of the different bibliometric features of the field should be taken into account. The correctness of the indexes used for evaluating journal papers of individuals should be investigated also on the individual level.