Definition of a model based on bibliometric indicators for assessing applicants to academic positions

ALIS: A novel metric in lineage-independent evaluation of scholars

Evaluative bibliometrics often attempts to explore various methods to measure individual scholarly influence. Scholarly independence (SI) is a unique indicator that can be used to understand and assess the research performances of individual scholars. The SI is a rare quality that most funding agencies and universities seek during funding decisions or hiring processes. We propose author lineage independent score (ALIS), a unique model to measure SI of a scholar by using his or her academic genealogy tree as the underlying graph structure. The analysis is performed on real data of 100 authors, collected from the Web of Science (WoS) and the Mathematics Genealogy Project. The analysis is further validated on a larger scale, on a simulated sample of 10,000 authors. The simulation exercise is the proof-of-concept for scalability of the metric and the proposed optimisation model. ALIS exploits genealogical relationships between scholars and their mentors and collaborating communities and constructs an influence scoring model based on the Genealogy tree structure of the respective scholars. The implications from the theoretical model are found to be profound in tracing known and recursive citation patterns among peers. The genealogy tree is used to investigate the advisor–advisee relationship and lays the foundation for defining metrics used to calculate the various indicators such as non-genealogy citations (NGCs), non-community citations (NCCs) and other citation quotient (OCQ). As these indicators/parameters are novel and thus not readily accessible, algorithms are written to compute these indicator values for the scholars under study.

Predicting the results of evaluation procedures of academics

BACKGROUND

The 2010 reform of the Italian university system introduced the National Scientific Habilitation (ASN) as a requirement for applying to permanent professor positions. Since the CVs of the 59,149 candidates and the results of their assessments have been made publicly available, the ASN constitutes an opportunity to perform analyses about a nation-wide evaluation process.

OBJECTIVE

The main goals of this paper are: (i) predicting the ASN results using the information contained in the candidates' CVs; (ii) identifying a small set of quantitative indicators that can be used to perform accurate predictions.

APPROACH

Semantic technologies are used to extract, systematize and enrich the information contained in the applicants' CVs, and machine learning methods are used to predict the ASN results and to identify a subset of relevant predictors.

RESULTS

For predicting the success in the role of associate professor, our best models using all and the top 15 predictors make accurate predictions (F-measure values higher than 0.6) in 88% and 88.6% of the cases, respectively. Similar results have been achieved for the role of full professor.

EVALUATION

The proposed approach outperforms the other models developed to predict the results of researchers' evaluation procedures.

CONCLUSIONS

Such results allow the development of an automated system for supporting both candidates and committees in the future ASN sessions and other scholars' evaluation procedures.

Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland

This article discusses the Polish Journal Ranking, which is used in the research evaluation system in Poland. In 2015, the ranking, which represents all disciplines, allocated 17,437 journals into three lists: A, B, and C. The B list constitutes a ranking of Polish journals that are indexed neither in the Web of Science nor the European Reference Index for the Humanities. This ranking was built by evaluating journals in three dimensions: formal, bibliometric, and expert-based. We have analysed data on 2035 Polish journals from the B list. Our study aims to determine how an expert-based evaluation influenced the results of final evaluation. In our study, we used structural equation modelling, which is regression based, and we designed three pairs of theoretical models for three fields of science: (1) humanities, (2) social sciences, and (3) engineering, natural sciences, and medical sciences. Each pair consisted of the full model and the reduced model (i.e., the model without the expert-based evaluation). Our analysis revealed that the multidimensional evaluation of local journals should not rely only on the bibliometric indicators, which are based on the Web of Science or Scopus. Moreover, we have shown that the expert-based evaluation plays a major role in all fields of science. We conclude with recommendations that the formal evaluation should be reduced to verifiable parameters and that the expert-based evaluation should be based on common guidelines for the experts.