Measurement and determinants of academic research efficiency: a systematic review of the evidence

Research output of radiologists in Australia and New Zealand: Strengths, weaknesses and future directions

INTRODUCTION

Clinical radiology is a popular career. However, academic radiology in Australia and New Zealand (ANZ) has not traditionally been a strength of the specialty which has a focus on clinical medicine and has been influenced by corporatisation of the specialty. The aim of this study was to review the source(s) of radiologist-led research in Australia and New Zealand, to identify areas of relative deficiency and propose plans to improve research output.

METHODS

A manual search was performed of all manuscripts in seven popular ANZ journals, where the corresponding or senior author was a radiologist. Publications between January 2017 and April 2022 were included.

RESULTS

There were 285 manuscripts from ANZ radiologists during the study period. This equates to 10.7 manuscripts per 100 radiologists based on RANZCR census data. Radiologists in Northern Territory, Victoria, Western Australia, South Australia and the Australian Capital Territory all produced manuscripts above the corrected mean incidence rate of 10.7 manuscripts per 100 radiologists. However, locations including Tasmania, New South Wales, New Zealand and Queensland were below the mean. The majority of manuscripts arose from public teaching hospitals with accredited trainees (86%), and there were a higher proportion of manuscripts published by female radiologists (11.5 compared to 10.4 per 100 radiologists).

CONCLUSION

Radiologists in ANZ are academically active; however, interventions aimed at increasing output could be targeted at certain locations and/or areas within a busy private sector. Time, culture, infrastructure and research support are vital, but personal motivation is also extremely important.

S&T resource allocation considering both performance and potential: The case of Chinese research institutes

The ex-post assessment of institutional performance has been applied to allocate scientific and technological (S&T) resource to universities and public research institutes. However, over-emphasis on particular types of performance could lead to unintended results and harm the science system. This research assesses the performance of these universities and public research institutes using ‘technical efficiency’ and their potential using ‘capacity utilization’, which are obtained by data envelopment analysis methods. Moreover, a comprehensive S&T resource allocation framework is proposed, where the organizations can be classified into four groups according to their performance and potential assessment results. An empirical study is conducted using the data of 58 Chinese research institutes from 2011 to 2018. Results indicate different patterns in the distribution and evolution of the performance and potential of these research institutes. The approaches proposed by this research are expected to complement existing performance-based S&T resource allocations.

Digitalization and academic research: knowing of and using digital services and software to develop scientific papers

Purpose

This paper explores how digitalization affects the academic research publication process by taking into account the perspective of management scholars. It provides an overview of the digital professional services dedicated to academic research, and investigates academics' awareness of, the impact on the publication process of, and scholars' expectations regarding digital services and software.

Design/methodology/approach

This explorative study adopted a qualitative approach by performing direct observations of websites regarding digital professional research services and in-depth interviews with national and international management scholars.

Findings

The multiple digital professional services dedicated to academic research enable authors to develop a scientific paper independently or with the support of professionals. The scholars' awareness regarding the digital services and software was limited, because of both the plethora of options on the market and the frequent use of the same digital tools over time. In impact terms, these tools enable scholars to improve research quality and to increase productivity. However, the negative effects led scholars to express different expectations about how they can be improved and what difficulties should be overcome to favor the publication process.

Practical implications

The results of this study provide suggestions both for scholars who engage in academic research and digital services and software providers.

Originality/value

To the best of the authors' knowledge, this is the first study to examine the ongoing development of digitalization in support of the research publication process from the perspective of academics.

Relationship Between Research Culture and Research Activity of Medical Doctors: A Survey and Audit

PURPOSE

To describe the research capacity and culture, and research activity (publications and new projects) of medical doctors across a health service and determine if the research activity of specialty groups correlated with their self-reported "team" level research capacity and culture.

METHODS

Cross-sectional, observational survey and audit of medical doctors at a tertiary health service in Queensland. The Research Capacity and Culture (RCC) validated survey was used to measure self-reported research capacity/culture at organisation, team and individual levels, and presence of barriers and facilitators to research. An audit of publications and ethically approved research projects was used to determine research activity.

RESULTS

Approximately, 10% of medical doctors completed the survey (n= 124). Overall, median scores on the RCC were 5 out of 10 for organisational level, 5.5 for specialty level, and 6 for individual level capacity and culture; however, specialty-level scores varied significantly between specialty groups (range 3.1-7.8). Over 80% of participants reported lack of time and other work roles taking priority as barriers to research. One project was commenced per year for every 12.5 doctors employed in the health service, and one article was published for every 7.5. There was a positive association between a team's number of publications and projects and their self-reported research capacity and culture on the RCC. This association was stronger for publications.

CONCLUSION

Health service research capacity building interventions may need a tailored approach for different specialty teams to accommodate for varying baselines of capacity and activity. When evaluating these initiatives, a combination of research activity and subjective self-report measures may be complementary.

The systemic approach as an instrument to evaluate higher education systems: Opportunities and challenges

This article aims to provide a systemic instrument to evaluate the functioning of higher education systems. Despite systemic instruments have had a strong impact on the management of public policy systems in fields such as health and innovation, higher education has not been widely discussed in applying this type of instrument. Herein lies the main gap that we want to close. The ultimate purpose of the evaluation instrument introduced here is thus to provide information for decision-makers, so these can identify the strengths/weaknesses in the functioning of their respective higher education systems from a systemic perspective. To achieve the previous goal, we apply the methodological guidelines of the integrative review of the literature. An integrative review of the literature was chosen because it guides the extraction of quantitative evidence from the literature and its classification, with the purpose of integrating the results into an analytical framework. This resulting analytical framework is what we have labelled as the systemic evaluation instrument. The article makes three contributions to the literature. First, the different types of higher education institutions considered in the literature and the higher education systems analysis scales are evidenced. Second, we identify the capacities and functions examined by the literature so that higher education institutions and higher education systems can fulfil their missions. Third, a systemic evaluation framework for higher education institutions and higher education systems is presented. The article concludes with a discussion of the opportunities and challenges associated to the implementation of such a systemic framework for policymaking.

How efficient are German life sciences? Econometric evidence from a latent class stochastic output distance model

This article investigates the technical efficiency in German higher education while accounting for possible heterogeneity in the production technology. We investigate whether a latent class model would identify the different sub-disciplines of life sciences in a sample of biology and agricultural units based on technological differences. We fit a latent class stochastic frontier model to estimate the parameters of an output distance function formulation of the production technology to investigate if a technological separation is meaningful along sub-disciplinary lines. We apply bootstrapping techniques for model validation. Our analysis relies on evaluating a unique dataset that matches information on higher educational institutions provided by the Federal Statistical Office of Germany with the bibliometric information extracted from the ISI Web of Science Database. The estimates indicate that neglecting to account for the possible existence of latent classes leads to a biased perception of efficiency. A classification into a research-focused and teaching-focused decision-making unit improves model fit compared to the pooled stochastic frontier model. Additionally, research-focused units have a higher median technical efficiency than teaching-focused units. As the research focus is more prevalent in the biology subsample an analysis not considering the potential existence of latent classes might misleadingly give the appearance of a higher mean efficiency of biology. In fact, we find no evidence of a difference in the mean technical efficiencies for German agricultural sciences and biology using the latent class model.

Does a Country/Region’s Economic Status Affect Its Universities’ Presence in International Rankings?

Purpose

Study how economic parameters affect positions in the Academic Ranking of World Universities’ top 500 published by the Shanghai Jiao Tong University Graduate School of Education in countries/regions with listed higher education institutions.

Design/methodology/approach

The methodology used capitalises on the multi-variate characteristics of the data analysed. The multi-colinearity problem posed is solved by running principal components prior to regression analysis, using both classical (OLS) and robust (Huber and Tukey) methods.

Findings

Our results revealed that countries/regions with long ranking traditions are highly competitive. Findings also showed that some countries/regions such as Germany, United Kingdom, Canada, and Italy, had a larger number of universities in the top positions than predicted by the regression model. In contrast, for Japan, a country where social and economic performance is high, the number of ARWU universities projected by the model was much larger than the actual figure. In much the same vein, countries/regions that invest heavily in education, such as Japan and Denmark, had lower than expected results.

Research limitations

Using data from only one ranking is a limitation of this study, but the methodology used could be useful to other global rankings.

Practical implications

The results provide good insights for policy makers. They indicate the existence of a relationship between research output and the number of universities per million inhabitants. Countries/regions, which have historically prioritised higher education, exhibited highest values for indicators that compose the rankings methodology; furthermore, minimum increase in welfare indicators could exhibited significant rises in the presence of their universities on the rankings.

Originality/value

This study is well defined and the result answers important questions about characteristics of countries/regions and their higher education system.

The graduation shift of German universities of applied sciences

In research into higher education, the evaluation of completion and dropout rates has generated a steady stream of interest for decades. While most studies only calculate quotes using student and graduate numbers for both phenomena, we propose to additionally consider the budget available to universities. We transfer the idea of the excellence shift indicator [1] from the research to the teaching area, in particular to the completion rate of educational entities. The graduation shift shows the institutions’ ability to produce graduates as measured against their basic academic teaching efficiency. It is an important advantage of the graduation shift that it avoids the well-known heterogeneity problem in efficiency measurements. Our study is based on German universities of applied science. Given their politically determined focus on education, this dataset is well-suited for introducing and evaluating the graduation shift. Using a comprehensive dataset covering the years 2008 to 2013, we show that the graduation shift produces results, which correlate closely with the results of the well-known graduation rate and standard Data Envelopment Analysis (DEA). Compared to the graduation rate, the graduation shift is preferable because it allows to take the budget of institutions into account. Compared to the DEA, the computation of the graduation shift is easy, the results are robust, and non-economists can understand them results. Thus, we recommend the graduation shift as an alternative method of efficiency measurement in the teaching area.

How Efficiently Do Elite US Universities Produce Highly Cited Papers?

While output and impact assessments were initially at the forefront of institutional research evaluations, efficiency measurements have become popular in recent years. Research efficiency is measured by indicators that relate research output to input. The additional consideration of research input in research evaluation is obvious, since the output depends on the input. The present study is based on a comprehensive dataset with input and output data for 50 US universities. As input, we used research expenses, and as output the number of highly-cited papers. We employed Data Efficiency Analysis (DEA), Free Disposal Hull (FDH) and two more robust models: the order-m and order-α approaches. The results of the DEA and FDH analysis show that Harvard University and Boston College can be called especially efficient compared to the other universities. While the strength of Harvard University lies in its high output of highly-cited papers, the strength of Boston College is its small input. In the order-α and order-m frameworks, Harvard University remains efficient, but Boston College becomes super-efficient. We produced university rankings based on adjusted efficiency scores (subsequent to regression analyses), in which single covariates (e.g., the disciplinary profile) are held constant.

Organizational Sustainability Practices: A Study of the Firms Listed by the Corporate Sustainability Index

Organizational sustainability (OS) has been guiding the decision-making process of managers in order to generate competitive advantage. This paper aims to identify the sustainable practices performed by large corporations in the implementation of OS. Reports with actions performed by large organizations and their reach in the three pillars of sustainability—environmental, economic, and social dimensions—are disclosed to their main stakeholders, based on short, medium and long-term sustainable goals. These reports often reflect the progress of OS or the progress made toward them. However, few studies investigate the sustainable practices adopted by firms and their reproducibility. A search was performed in reports selected from the firms listed by the Corporate Sustainability Index (CSI) from 2012–2016, belonging to the Brazilian stock market in services sector of the economy and employed the Global Reporting Initiative (GRI) methodology. The results showed the strategic planning involving infrastructure, environment, human resources, product innovation, organizational management and deadline setting acted as the baseline for the implementation of the practices found. The findings will guide the managers´ decisions in the development of their strategic planning, based on practical and objective results.