Measuring changes in publication patterns in a context of performance-based research funding systems: the case of educational research in the University of Gothenburg (2005–2014)

Evaluating Research Efficiency in Indonesian Higher Education Institution

The Indonesian government has evaluated the research performance of universities, whose measurement process is projected into resources, management, outputs, and revenues to determine the provision of incentives, grants, and program funding to universities. However, efficiency calculations have shown that the outputs and competition-based incentives that drive scientific productivity are more complex. The most competitive systems must also be the most productive when considering resources. This study aimed to analyze the research efficiency in the Indonesian higher education system. The efficiency was analyzed by maximizing the 13 product outputs from the research budget and university staff. The result was then compared with the existing performance measurement analysis. Data Envelopment Analysis (DEA) was used to evaluate the efficiency based on the data of 47 universities in the Mandiri cluster and 144 in the Utama cluster for the 2014-2018 period. These findings showed that about 68% of universities have an efficiency value of 1 for the Mandiri group, almost 40% in the Utama group, and 41% for the two groups combined. Additionally, this study compared the efficiency analysis and the impact of the performance evaluation. The comparison showed that adding efficiency or productivity factors in the performance evaluation assessment produced a more accurate result.

On the possibilities of presenting the research performance of an institute over a long period of time: the case of the Institute of Information Science at the University of Graz in Austria

In this paper, we demonstrate how the research performance of a university institute (department) over a long period of time can be presented and evaluated. Using the example of an information science institute at a German-speaking university, namely the (former) Institute of Information Science at the University of Graz in Austria, we present the research performance of this institute over the entire duration of its existence (33 years) in different ways. In order to be able to contextualize its performance, we compare it with that of some related institutions from all over the world. Due to the high effort involved in collecting data and the lack of data availability, the comparison must be limited to a period of a few years and—with regard to the institutions from non-German-speaking countries—to the Web of Science as data source. In this international comparison, the institute in the focus of the study shows relatively poor results. As can be seen, the choice of the data source has a major influence on the evaluation results. Especially for institutes from non-English-speaking countries with publications in their respective national languages, an exclusive use of international databases, such as Web of Science or Scopus, cannot fully consider the whole research performance. The use of personal publication lists or local research databases seems almost indispensable in these cases. A major novelty of this article is the handling of a very long evaluation period and the discussion of different ways of subdividing it. With regard to the presentation of the results, in the case of a long observation period, not only should annual and overall results be presented, but also multi-year comparisons be performed. In this way, year-by-year fluctuations can be smoothed out, and longer-term developments can be well represented.

National Lists of Scholarly Publication Channels: An Overview and Recommendations for Their Construction and Maintenance

Purpose

This paper presents an overview of different kinds of lists of scholarly publication channels and of experiences related to the construction and maintenance of national lists supporting performance-based research funding systems. It also contributes with a set of recommendations for the construction and maintenance of national lists of journals and book publishers.

Design/methodology/approach

The study is based on analysis of previously published studies, policy papers, and reported experiences related to the construction and use of lists of scholarly publication channels.

Findings

Several countries have systems for research funding and/or evaluation, that involve the use of national lists of scholarly publication channels (mainly journals and publishers). Typically, such lists are selective (do not include all scholarly or non-scholarly channels) and differentiated (distinguish between channels of different levels and quality). At the same time, most lists are embedded in a system that encompasses multiple or all disciplines. This raises the question how such lists can be organized and maintained to ensure that all relevant disciplines and all types of research are adequately represented.

Research limitation

The conclusions and recommendations of the study are based on the authors’ interpretation of a complex and sometimes controversial process with many different stakeholders involved.

Practical implications

The recommendations and the related background information provided in this paper enable mutual learning that may feed into improvements in the construction and maintenance of national and other lists of scholarly publication channels in any geographical context. This may foster a development of responsible evaluation practices.

Originality/value

This paper presents the first general overview and typology of different kinds of publication channel lists, provides insights on expert-based versus metrics-based evaluation, and formulates a set of recommendations for the responsible construction and maintenance of publication channel lists.

It is not all about performance: Importance of the funding formula in the allocation of performance-based research funding in England

Performance-based research funding systems have become popular over the last decades. One of the main reasons for these funding systems is to allow funding bodies to allocate public research funds more effectively based on the assessed quality. However, the performance-based research funding received by higher education institutes (HEIs) not only depends on the quality of research activity carried out but also on the funding formula used by funding bodies. This article examines the funding formula used by Research England (RE) and assesses the effect of this formula on quality-related research (QR) funding allocation using data of mainstream QR funding allocation for the 2017–18 period. RE’s funding formula includes some value judgements by policymakers such as allocation of fourfold QR funding to ‘world-leading’ research compared to ‘internationally-excellent’ research, and the use of different subject cost weights. These value judgements play an important role in the allocation of QR funding beyond the assessed quality of research. This article finds that changes in some of these value judgments such as allocation of threefold (rather than fourfold) QR funding to world-leading research compared to internationally excellent research, or the use of alternative subject cost weights lead to major changes in the allocation of QR funding to different subject areas and HEIs. Results suggest that these value judgments are also important beyond the assessed quality of research, and that consultation of different subject areas and HEIs about these decisions and re-evaluation of some of these value judgements are needed for a more accountable distribution of QR funding.