Calculating the excellence shift: How efficiently do institutions produce highly cited papers?

Evaluating Research Efficiency in Indonesian Higher Education Institution

The Indonesian government has evaluated the research performance of universities, whose measurement process is projected into resources, management, outputs, and revenues to determine the provision of incentives, grants, and program funding to universities. However, efficiency calculations have shown that the outputs and competition-based incentives that drive scientific productivity are more complex. The most competitive systems must also be the most productive when considering resources. This study aimed to analyze the research efficiency in the Indonesian higher education system. The efficiency was analyzed by maximizing the 13 product outputs from the research budget and university staff. The result was then compared with the existing performance measurement analysis. Data Envelopment Analysis (DEA) was used to evaluate the efficiency based on the data of 47 universities in the Mandiri cluster and 144 in the Utama cluster for the 2014-2018 period. These findings showed that about 68% of universities have an efficiency value of 1 for the Mandiri group, almost 40% in the Utama group, and 41% for the two groups combined. Additionally, this study compared the efficiency analysis and the impact of the performance evaluation. The comparison showed that adding efficiency or productivity factors in the performance evaluation assessment produced a more accurate result.

Comparison of scientometric achievements at PhD and scientific output ten years later for 4,790 academic researchers

Introduction

PhD is the highest awarded degree offered by universities in different disciplines. Owners of a PhD can teach at universities, start independent research and receive a higher salary while further building a scientific career. We examined whether the publication output before the PhD degree has a correlation with subsequent research activities.

Methods

We downloaded publication and citation data from the Hungarian Scientific Bibliography for Hungarian researchers who obtained PhD between the ages of 24 and 45. The researchers were grouped into eleven scientific sections. We examined the number of Q1 publications published in the previous 5 years, the H-index, the total number of citations for the last complete year, and the biological age of the researcher. Each parameter was computed for the year at which the PhD was obtained and ten years later. Pre-PhD publications (and citations for these) were excluded when assessing post-PhD track records. Spearman rank correlation and Kruskal-Wallis test were computed.

Results

We analyzed all together 4,790 researchers. We obtained a positive correlation between the number of Q1 publications before and after PhD (corr. coeff. = 0.21–0.54, p<0.01 in all sections), between the H-index before and after PhD (corr. coeff. = 0.32–0.56, p<0.01 in all sections), and between the citations received before and after PhD (corr. coeff. = 0.34–0.51, p<0.01 in all sections). All three metrics measured ten years after the PhD were negatively correlated with the age of the researcher at the time of obtaining the PhD (number of publications corr. coeff. = -0.09–0.22, p<0.05; H-index corr. coeff. = -0.09–0.29, p<0.08; number of citations corr. coeff. = -0.14–0.30, p<0.01). Among all disciplines, Philosophy and History and Engineering sciences show the strongest correlation between pre- and post-PhD output. When running multiple regression analysis for all three metrics as dependent variables and the number of articles, the H-index, the number of citations in the year of the PhD, the calendar year of PhD, and the gender of the researcher as independent variables, the number of articles and the H-index in the year of PhD reached the strongest positive correlations while gender had a negative correlation.

Conclusions

We independently evaluated pre- and post-PhD publication performance. In connection with age, the discipline-specific reference values of scientometric parameters at the time of obtaining the PhD can help to select candidates for postdoctoral grants and positions.

How efficient are German life sciences? Econometric evidence from a latent class stochastic output distance model

This article investigates the technical efficiency in German higher education while accounting for possible heterogeneity in the production technology. We investigate whether a latent class model would identify the different sub-disciplines of life sciences in a sample of biology and agricultural units based on technological differences. We fit a latent class stochastic frontier model to estimate the parameters of an output distance function formulation of the production technology to investigate if a technological separation is meaningful along sub-disciplinary lines. We apply bootstrapping techniques for model validation. Our analysis relies on evaluating a unique dataset that matches information on higher educational institutions provided by the Federal Statistical Office of Germany with the bibliometric information extracted from the ISI Web of Science Database. The estimates indicate that neglecting to account for the possible existence of latent classes leads to a biased perception of efficiency. A classification into a research-focused and teaching-focused decision-making unit improves model fit compared to the pooled stochastic frontier model. Additionally, research-focused units have a higher median technical efficiency than teaching-focused units. As the research focus is more prevalent in the biology subsample an analysis not considering the potential existence of latent classes might misleadingly give the appearance of a higher mean efficiency of biology. In fact, we find no evidence of a difference in the mean technical efficiencies for German agricultural sciences and biology using the latent class model.

The graduation shift of German universities of applied sciences

In research into higher education, the evaluation of completion and dropout rates has generated a steady stream of interest for decades. While most studies only calculate quotes using student and graduate numbers for both phenomena, we propose to additionally consider the budget available to universities. We transfer the idea of the excellence shift indicator [1] from the research to the teaching area, in particular to the completion rate of educational entities. The graduation shift shows the institutions’ ability to produce graduates as measured against their basic academic teaching efficiency. It is an important advantage of the graduation shift that it avoids the well-known heterogeneity problem in efficiency measurements. Our study is based on German universities of applied science. Given their politically determined focus on education, this dataset is well-suited for introducing and evaluating the graduation shift. Using a comprehensive dataset covering the years 2008 to 2013, we show that the graduation shift produces results, which correlate closely with the results of the well-known graduation rate and standard Data Envelopment Analysis (DEA). Compared to the graduation rate, the graduation shift is preferable because it allows to take the budget of institutions into account. Compared to the DEA, the computation of the graduation shift is easy, the results are robust, and non-economists can understand them results. Thus, we recommend the graduation shift as an alternative method of efficiency measurement in the teaching area.

Factors influencing the scientific performance of Momentum grant holders: an evaluation of the first 117 research groups

The Momentum program launched in 2009 provides funding of up to 1 million Euro to establish new, independent research groups at Hungarian academic institutions. Here, our aim was to determine factors associated with the scientific output of these research groups. Publication data were downloaded from the Hungarian Scientific Work Archive (www.mtmt.hu), impact factor data were obtained from Thomson Reuters (jcr.incites.thomsonreuters.com), and journal ranks were extracted from the Scimago Journal Rank database (www.scimagojr.com). Investigated input features for each grant holder include gender, degree, targeted category, international mobility, international grants, number of publications, total number of citations, H-index, best publications, impact factors in the last 2 years, and assessment scores provided by the experts. Evaluated performance indicators include cumulative impact factor, number of D1 publications, and number of first/last author D1 publications during the grant running time. Grant holders’ publication output increased by 23 and 52% for life sciences and material sciences researchers. Scientific performance was independent from gender, degree, international grants, category applied for, and citations received for the best pre-grant publication. Those with international mobility had significantly lower scientific output (yearly impact factor, number of D1 publications, number of first/last author publications). Scores received from grant review experts were independent from later publication activity. The strongest correlations were observed between scientific output and total number of citations, H-index, and impact factor in the last 2 years pre-grant. In summary, group leaders with a dynamic publication track record were able to attain the most additional momentum. Our results can help accelerate and improve future grant review processes.