Goodhart's Law and Performance Indicators in Higher Education

Reward systems for cohort data sharing: An interview study with funding agencies

Data infrastructures are being constructed to facilitate cohort data sharing. These infrastructures are anticipated to increase the rate of data sharing. However, the lack of data sharing has also been framed as being the consequence of the lack of reputational or financial incentives for sharing. Some initiatives try to confer value onto data sharing by making researchers' individual contributions to research visible (i.e., contributorship) or by quantifying the degree to which research data has been shared (e.g., data indicators). So far, the role of downstream evaluation and funding distribution systems for reputational incentives remains underexplored. This interview study documents the perspectives of members of funding agencies on, amongst other elements, incentives for data sharing. Funding agencies are adopting narrative CVs to encourage evaluation of diverse research outputs and display diversity in researchers' profiles. This was argued to diminish the focus on quantitative indicators of scientific productivity. Indicators related to open science dimensions may be reintroduced if they are fully developed. Shifts towards contributorship models for research outputs are seen as complementary to narrative review.

Development and Evaluation of a Faculty-Based Accredited Continuing Professional Development Route for Teaching and Learning

This article characterizes and evaluates the development of an accredited, in-house, faculty-based teaching recognition scheme aimed at supporting clinicians and academics to achieve Advance HE Fellowship recognition. The scheme takes 6 to 24 months to complete and forms part of an institution-wide scheme. The evaluation covered 44 months, collecting data on participation rates across the school and 21 semi-structured interviews across 16 staff participants. We describe the outcomes measured alongside key perceived benefits and challenges to support the implementation of similar schemes elsewhere. Across 130 academic staff, there was 61% engagement. In interviews, 11 participants characterized benefits in terms of changes to their teaching, such as adopting new strategies for differing class sizes, and highlighted the benefit of accessible and context-specific development opportunities designed specifically for STEMM (science, technology, engineering, mathematics, and medicine) practitioners and clinicians. Motivations for participating were mainly intrinsic (69%), with international professional recognition also featured (61%, n = 10). Of the 23 participants who withdrew, the largest subgroup (39%) withdrew because they had left the institution, and 35% withdrew because of a lack of time, which encompassed a range of issues. We outline recommendations for implementing similar schemes including protected time, accessible development opportunities, and support for mentors.

Speeding up to keep up: exploring the use of AI in the research process

There is a long history of the science of intelligent machines and its potential to provide scientific insights have been debated since the dawn of AI. In particular, there is renewed interest in the role of AI in research and research policy as an enabler of new methods, processes, management and evaluation which is still relatively under-explored. This empirical paper explores interviews with leading scholars on the potential impact of AI on research practice and culture through deductive, thematic analysis to show the issues affecting academics and universities today. Our interviewees identify positive and negative consequences for research and researchers with respect to collective and individual use. AI is perceived as helpful with respect to information gathering and other narrow tasks, and in support of impact and interdisciplinarity. However, using AI as a way of ‘speeding up—to keep up’ with bureaucratic and metricised processes, may proliferate negative aspects of academic culture in that the expansion of AI in research should assist and not replace human creativity. Research into the future role of AI in the research process needs to go further to address these challenges, and ask fundamental questions about how AI might assist in providing new tools able to question the values and principles driving institutions and research processes. We argue that to do this an explicit movement of meta-research on the role of AI in research should consider the effects for research and researcher creativity. Anticipatory approaches and engagement of diverse and critical voices at policy level and across disciplines should also be considered.

Researcher engagement in policy deemed societally beneficial yet unrewarded

Maintaining the continued flow of benefits from science, as well as societal support for science, requires sustained engagement between the research community and the general public. On the basis of data from an international survey of 1092 participants (634 established researchers and 458 students) in 55 countries and 315 research institutions, we found that institutional recognition of engagement activities is perceived to be undervalued relative to the societal benefit of those activities. Many researchers report that their institutions do not reward engagement activities despite institutions' mission statements promoting such engagement. Furthermore, institutions that actually measure engagement activities do so only to a limited extent. Most researchers are strongly motivated to engage with the public for selfless reasons, which suggests that incentives focused on monetary benefits or career progress may not align with researchers' values. If institutions encourage researchers' engagement activities in a more appropriate way - by moving beyond incentives - they might better achieve their institutional missions and bolster the crucial contributions of researchers to society.

Building the infrastructure to make science metrics more scientific

Research leaders, policy makers and science strategists need evidence to support decision-making around research funding investment, policy and strategy.  In recent years there has been a rapid expansion in the data sources available that shed light onto aspects of research quality, excellence, use, re-use and attention, and engagement. This is at a time when the modes and routes to share and communicate research findings and data are also changing.  In this opinion piece, we outline a series of considerations and interventions that are needed to ensure that research metric development is accompanied by appropriate scrutiny and governance, to properly support the needs of research assessors and decision-makers, while securing the confidence of the research community. Key among these are: agreed 'gold standards' around datasets and methodologies; full transparency around the calculation and derivation of research-related indicators; and a strategy and roadmap to take the discipline of scientific indicators and research assessment to a more robust and sustainable place.

Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition

Over the last 50 years, we argue that incentives for academic scientists have become increasingly perverse in terms of competition for research funding, development of quantitative metrics to measure performance, and a changing business model for higher education itself. Furthermore, decreased discretionary funding at the federal and state level is creating a hypercompetitive environment between government agencies (e.g., EPA, NIH, CDC), for scientists in these agencies, and for academics seeking funding from all sources-the combination of perverse incentives and decreased funding increases pressures that can lead to unethical behavior. If a critical mass of scientists become untrustworthy, a tipping point is possible in which the scientific enterprise itself becomes inherently corrupt and public trust is lost, risking a new dark age with devastating consequences to humanity. Academia and federal agencies should better support science as a public good, and incentivize altruistic and ethical outcomes, while de-emphasizing output.

The role of administrative data in the big data revolution in social science research

The term big data is currently a buzzword in social science, however its precise meaning is ambiguous. In this paper we focus on administrative data which is a distinctive form of big data. Exciting new opportunities for social science research will be afforded by new administrative data resources, but these are currently under appreciated by the research community. The central aim of this paper is to discuss the challenges associated with administrative data. We emphasise that it is critical for researchers to carefully consider how administrative data has been produced. We conclude that administrative datasets have the potential to contribute to the development of high-quality and impactful social science research, and should not be overlooked in the emerging field of big data.