‘Impact’ in the proposals for the UK's Research Excellence Framework: Shifting the boundaries of academic autonomy

Assessing research excellence: Evaluating the Research Excellence Framework

Performance-based research funding systems have been extensively used around the globe to allocate funds across higher education institutes (HEIs), which led to an increased amount of literature examining their use. The UK’s Research Excellence Framework (REF) uses a peer-review process to evaluate the research environment, research outputs and non-academic impact of research produced by HEIs to produce a more accountable distribution of public funds. However, carrying out such a research evaluation is costly. Given the cost and that it is suggested that the evaluation of each component is subject to bias and has received other criticisms, this article uses correlation and principal component analysis to evaluate REF’s usefulness as a composite evaluation index. As the three elements of the evaluation—environment, impact and output—are highly and positively correlated, the effect of the removal of an element from the evaluation leads to relatively small shifts in the allocation of funds and in the rankings of HEIs. As a result, future evaluations may consider the removal of some elements of the REF or reconsider a new way of evaluating different elements to capture organizational achievement rather than individual achievements.

Translating knowledge in new entrepreneurial ventures: the role of business plan development

Purpose

This paper aims to analyse the role of business plan development as a knowledge translation tool, especially for the creation of start-ups. In a complex knowledge ecosystem populated by multiple diverse and autonomous actors (such as potential entrepreneurs, local companies, local public entities and business consultants) bonded together by a joint search for valuable knowledge, business plan development can work as a powerful enabler for the translation of knowledge.

Design/methodology/approach

The study uses a qualitative multi-case study approach by examining the results of a public programme devoted to the creation of new entrepreneurial ventures. The authors analysed 418 complete business plans and followed up with all the participants with an interview. In total, 40 cases were investigated more in detail.

Findings

Results show how business plan development can function as a bridge between academic, theoretical and general knowledge on start-up creation on the one hand and practical contextualised activities of potential entrepreneurs on the other.

Practical implications

The process of knowledge translation is crucial to ensure that relevant knowledge coming from both the inside (the entrepreneur) and outside (the stakeholders) of the organisation is effectively applied. To facilitate the translation process, key knowledge users should be supported in contextualising and making sense of the research knowledge. Initiatives carried out by local entities and other actors, gathering several stakeholders to develop business plans, can become valuable opportunities to facilitate the translation process for start-up development.

Originality/value

The paper contributes to knowledge management and knowledge translation literature by demonstrating the role of business plan development as an effective knowledge translation enabler. It also adds to the understanding of innovation management and entrepreneurial education by proving the relevance of the translation of knowledge for the creation of new business ventures.

Science as collaborative knowledge generation

The COVID-19 pandemic points to the need for scientists to pool their efforts in order to understand this disease and respond to the ensuing crisis. Other global challenges also require such scientific cooperation. Yet in academic institutions, reward structures and incentives are based on systems that primarily fuel the competition between (groups of) scientific researchers. Competition between individual researchers, research groups, research approaches, and scientific disciplines is seen as an important selection mechanism and driver of academic excellence. These expected benefits of competition have come to define the organizational culture in academia. There are clear indications that the overreliance on competitive models undermines cooperative exchanges that might lead to higher quality insights. This damages the well-being and productivity of individual researchers and impedes efforts towards collaborative knowledge generation. Insights from social and organizational psychology on the side effects of relying on performance targets, prioritizing the achievement of success over the avoidance of failure, and emphasizing self-interest and efficiency, clarify implicit mechanisms that may spoil valid attempts at transformation. The analysis presented here elucidates that a broader change in the academic culture is needed to truly benefit from current attempts to create more open and collaborative practices for cumulative knowledge generation.

Co-option, control and criticality: the politics of relevance regimes for the future of political science

Over the last 20 years, the notion of relevance vis-à-vis political science became not only a subject of academic debates but also a domain of practice, largely due to the developments in the research funding, increasingly referred to as the 'impact agenda'. In this article, we explore how the growing focus on socio-economic impact as the assessment criterion of research funding shapes the discipline of political science itself—its knowledge production, dissemination and the emergent forms of accountability of political scientists. The article presents the results of a major international study that has examined the emergence of ‘impact agendas’ across 33 countries. We report on the changing idea of relevance of political science through the lens of its strategic ambiguity and historical evolution. We then explore these broader trends through an in-depth analysis of the UK as an ‘extreme case’ and a blueprint for funding system reforms. These developments, we argue, are not a mere funding policy innovation but rather a paradigm-level change, reshaping the position of political science in society as well as the types of scholarship that are possible and incentivised.

The Language of Sustainable Tourism as a Proxy Indicator of Quality

Sustainable tourism (ST) has recently become the mainstream of the tourism industry and, accordingly, has influenced contemporary tourism research. However, ST is not just theories about indications and contraindications of global travel, but also a specific language that needs mastering to take sustainability work forward. In other words, what research receives recognition depends on the proficiency in how the articulation in research proposals and within assessment under the heading of “research impact”. The aim of this paper is to investigate how tourism research gains recognition within research evaluation, by investigating the national research appraisal in the United Kingdom (Research Excellence Framework). By using content analysis, we disentangle the rhetorical choices and narrative constructions within researchers’ impact claims. Our findings suggest that researchers adopt a rhetorical style that implies causality and promotes good outcomes facilitating ST. However, the structure of the assessment format enforces an articulation of sustainable research impact without stating the methodological limitations of that such claim. Therefore, the rhetorical choices of ST researchers merely represent a proxy indicator of the claimed impact. We conclude that the lack of rigor in accounting for the impact of ST research may inadvertently restrict attaining ST.

How academic researchers select collaborative research projects: a choice experiment

Although many studies have been conducted on the drivers of and barriers to research collaborations, current literature provides limited insights into the ways in which individual researchers choose to engage in different collaborative projects. Using a choice experiment, we studied the factors that drive this choice using a representative sample of 3145 researchers from Western Europe and North America who publish in English. We find that for most researchers, the expected publication of research in scientific journals deriving from a project is the most decisive factor driving their collaboration choices. Moreover, most respondents prefer to collaborate with other partners than industry. However, different factors’ influence varies across groups of researchers. These groups are characterised as going for the ‘puzzle’ (60% of the sample), the ‘ribbon’ (33%) or the ‘gold’ (8%), i.e., primarily oriented toward intellectual goals, recognition or money, respectively. This heterogeneity shows that a combination of interventions will be required for governments aiming to promote university–industry collaborations.

Changing research on research evaluation: A critical literature review to revisit the agenda

The current range and volume of research evaluation-related literature is extensive and incorporates scholarly and policy/practice-related perspectives. This reflects academic and practical interest over many decades and trails the changing funding and reputational modalities for universities, namely increased selectivity applied to institutional research funding streams and the perceived importance of university rankings and other reputational devices. To make sense of this highly diverse body of literature, we undertake a critical review of over 350 works constituting, in our view, the ‘state-of-the-art’ on institutional performance-based research evaluation arrangements (PREAs). We focus on PREAs because they are becoming the predominant means world-wide to allocate research funds and accrue reputation for universities. We highlight the themes addressed in the literature and offer critical commentary on the balance of scholarly and policy/practice-related orientations. We then reflect on five limitations to the state-of-the-art and propose a new agenda, and a change of perspective, to progress this area of research in future studies.

Playing the fields: Theorizing research impact and its assessment

How research is assessed affects what types of knowledge are valued, incentivized, and rewarded. An increasingly important element of contemporary research evaluation is the measurement of the wider impact of research (e.g. benefit to society, culture or economy). Although the measurement of impact has been highly contested, the area is under-theorized and dominated by pragmatic research policy imperatives. Informed by a sociological perspective, this article intervenes in this context by reframing research impact as the attainment and maintenance of capital (i.e. symbolic power or status) in various fields beyond academia. It argues that research impact occurs at the intersection of these fields of power. The article shows that impact involves various combinations of capital from the scholarly field, the field of politics, the field of application, the media field, and the economic field, which provide credibility, authority, utility, visibility, and weight, respectively. In exploring the forms of worth and value that underpin the pursuit of legitimacy in these fields, the article provides a new theoretical framework for understanding research impact and its assessment.

It is not all about performance: Importance of the funding formula in the allocation of performance-based research funding in England

Performance-based research funding systems have become popular over the last decades. One of the main reasons for these funding systems is to allow funding bodies to allocate public research funds more effectively based on the assessed quality. However, the performance-based research funding received by higher education institutes (HEIs) not only depends on the quality of research activity carried out but also on the funding formula used by funding bodies. This article examines the funding formula used by Research England (RE) and assesses the effect of this formula on quality-related research (QR) funding allocation using data of mainstream QR funding allocation for the 2017–18 period. RE’s funding formula includes some value judgements by policymakers such as allocation of fourfold QR funding to ‘world-leading’ research compared to ‘internationally-excellent’ research, and the use of different subject cost weights. These value judgements play an important role in the allocation of QR funding beyond the assessed quality of research. This article finds that changes in some of these value judgments such as allocation of threefold (rather than fourfold) QR funding to world-leading research compared to internationally excellent research, or the use of alternative subject cost weights lead to major changes in the allocation of QR funding to different subject areas and HEIs. Results suggest that these value judgments are also important beyond the assessed quality of research, and that consultation of different subject areas and HEIs about these decisions and re-evaluation of some of these value judgements are needed for a more accountable distribution of QR funding.

Engaging with research impact assessment for an environmental science case study

Impact assessment is embedded in many national and international research rating systems. Most applications use the Research Impact Pathway to track inputs, activities, outputs and outcomes of an invention or initiative to assess impact beyond scholarly contributions to an academic research field (i.e., benefits to environment, society, economy and culture). Existing approaches emphasise easy to attribute 'hard' impacts, and fail to include a range of 'soft' impacts that are less easy to attribute, yet are often a dominant part of the impact mix. Here, we develop an inclusive 3-part impact mapping approach. We demonstrate its application using an environmental initiative.

Kafka’s ‘Before the Law’: The participation of the subject in its subjectification

This paper presents a close encounter between the literary works of Franz Kafka and a core topic in organizational theories of power, namely the participation of subjects in their own subjectification. In discussing ‘In the Cathedral’, the penultimate chapter of The Trial by Franz Kafka, the paper develops three central aspects of Kafka’s text: reflexivity as a form of entanglement with power, self-slander complementing formal involvement, and humour as a form of freedom. These aspects are mirrored against the example of performance evaluation to complement and enrich the theoretical debate about subjectification more generally. The paper and its contributions serve as a corrective to approaches that overemphasize either the possibilities of resistance, for example through reflexivity, or the impotence of the subject in the face of power.

Increasing research impact with citizen science: The influence of recruitment strategies on sample diversity

Despite the fact that citizen engagement in research is widely practised and regarded as one of the keys to maximizing the impact of research and innovation, empirical evidence on the value, potential and possibilities of engaging a broad diversity of citizens in practice is scant. The purpose of our article is twofold: (1) to provide more insight into the value and opportunities of engaging audiences that typically are not engaged with science and (2) to explore the effect of a targeted recruitment strategy versus a generic recruitment strategy on the profile, motivation and retainment of citizen science volunteers. Our empirical research is based on five citizen science projects in the domain of surface and drinking water research in the Netherlands. This article finds that using a targeted recruitment strategy, it is possible and worth to recruit a diverse sample of citizen science volunteers.

Science diplomacy and transnational governance impact

Science diplomacy is coming to the fore as a formidable dimension of interstate power relations. As the challenges of the world increasingly transcend borders, so too have researchers and innovators forged international coalitions to resolve global pathologies. In doing so, new channels of influence and opportunity have opened up for states alongside the 'traditional' modes of foreign diplomacy. Understanding how these channels influence global socio-economic outcomes is thereby crucial for scholars interested in the still-ambiguous structure and processes of global governance. This article advances understanding of the domains of science diplomacy by drawing attention to the 'political intercostalities' of state actors, scientific communities and other transnational actors within the new architectures of global governance. Here we trace the growing array of informal international associations alongside transgovernmental policy networks and 'global public-policy partnerships' that deal with highly specialised and technical matters of international policy and how they are drawn into science diplomacy. This article thus presents a research agenda for a particular mode of 'impact' in politics and international studies.

The Translational Science Benefits Model: A New Framework for Assessing the Health and Societal Benefits of Clinical and Translational Sciences

We report the development of the Translational Science Benefits Model (TSBM), a framework designed to support institutional assessment of clinical and translational research outcomes to measure clinical and community health impacts beyond bibliometric measures. The TSBM includes 30 specific and potentially measurable indicators that reflect benefits that accrue from clinical and translational science research such as products, system characteristics, or activities. Development of the TSBM was based on literature review, a modified Delphi method, and in‐house expert panel feedback. Three case studies illustrate the feasibility and face validity of the TSBM for identification of clinical and community health impacts that result from translational science activities. Future plans for the TSBM include further pilot testing and a resource library that will be freely available for evaluators, translational scientists, and academic institutions who wish to implement the TSBM framework in their own evaluation efforts.

Beyond Academia - Interrogating Research Impact in the Research Excellence Framework

Big changes to the way in which research funding is allocated to UK universities were brought about in the Research Excellence Framework (REF), overseen by the Higher Education Funding Council, England. Replacing the earlier Research Assessment Exercise, the purpose of the REF was to assess the quality and reach of research in UK universities–and allocate funding accordingly. For the first time, this included an assessment of research ‘impact’, accounting for 20% of the funding allocation. In this article we use a text mining technique to investigate the interpretations of impact put forward via impact case studies in the REF process. We find that institutions have developed a diverse interpretation of impact, ranging from commercial applications to public and cultural engagement activities. These interpretations of impact vary from discipline to discipline and between institutions, with more broad-based institutions depicting a greater variety of impacts. Comparing the interpretations with the score given by REF, we found no evidence of one particular interpretation being more highly rewarded than another. Importantly, we also found a positive correlation between impact score and [overall research] quality score, suggesting that impact is not being achieved at the expense of research excellence.

Taking health geography out of the academy: Measuring academic impact

In recent years the academic landscape has been shifting and significantly affected by the introduction of an 'impact agenda'. Academics are increasingly expected to demonstrate their broader engagement with the world and evidence related outcomes. Whilst different countries are at various stages along this impact journey, the UK is the first country to link impact to funding outcomes; here impact now accounts for 20% of an academic unit of assessment's Research Excellence Framework (REF) result. This concept of 'research impact' implies that our work can effect change through one or more identifiable events in a direct, preferably linear and certainly measurable manner. In this paper, focusing on impact in social science, and policy-related impact in particular, we argue that such a cause and effect model is inappropriate. Furthermore that impact is not immediate or indeed linear within social science research. Drawing on recent work on alcohol and tobacco environments in Scotland we present a case study of impact, reflect on the process and respond to the challenges of moving beyond 'business as usual' public participation towards the measurement of outcomes. In doing so we critique the way in which 'impact' is currently measured and suggest a move towards an enlightenment model with greater recognition of process.

Measuring the Impact of Research: Lessons from the UK's Research Excellence Framework 2014

Impactful academic research plays a stellar role in society, pressing to ask the question of how one measures the impact created by different areas of academic research. Measuring the societal, cultural, economic and scientific impact of research is currently the priority of the National Science Foundation, European Commission and several research funding agencies. The recently concluded United Kingdom's national research quality exercise, the Research Excellence Framework (REF) 2014, which piloted impact assessment as part of the overall evaluation offers a lens to view how impact of research in different disciplines can be measured. Overall research quality was assessed through quality of outputs, 'impact' and research environment. We performed two studies using the REF 2014 as a case study. The first study on 363 Impact Case Studies (ICSs) submitted in 5 research areas (UoAs) reveals that, in general, the impact scores were constructed upon a combination of factors i.e. quantity of quartile-one (Q1) publications, quantity and value of grants/income, number of researchers stated in the ICSs, spin-offs created, discoveries/patents and presentation of esteem data, informing researchers/ academics of the factors to consider in order to achieve a better impact score in research impact assessments. However, there were differences among disciplines in terms of the role played by the factors in achieving their overall scores for the ICSs. The outcome of this study is thus a set of impact indicators, and their relationship with the overall score of impact of research in different disciplines as determined in REF2014, which would in the first instance provide some answers to impact measures that would be useful for researchers in different disciplines. The second study extracts the general themes of impact reported by universities by performing a word frequency analysis in all the ICSs submitted in the five chosen research areas, which were substantially varied owing to their fields.

Investigating the sustainability of careers in academic primary care: a UK survey

Background

The UK National Health Service (NHS) is undergoing institutional reorganisation due to the Health and Social Care Act-2012 with a continued restriction on funding within the NHS and clinically focused academic institutions. The UK Society for Academic Primary Care (SAPC) is examining the sustainability of academic primary care careers within this climate and preliminary qualitative work has highlighted individual and organisational barriers. This study seeks to quantify the current situation for academics within primary care.

Methods

A survey of academic primary care staff was undertaken. Fifty-three academic primary care departments were selected. Members were invited to complete a survey which contained questions about an individual’s career, clarity of career pathways, organisational culture, and general experience of working within the area. Data were analysed descriptively with cross-tabulations between survey responses and career position (early, mid-level, senior), disciplinary background (medical, scientist), and gender. Pearson chi-square test was used to determine likelihood that any observed difference between the sets arose by chance.

Results

Responses were received from 217 people. Career pathways were unclear for the majority of people (64%) and 43% of the workforce felt that the next step in their career was unclear. This was higher in women (52% vs. men 25%; χ²(3) = 14.76; p = 0.002) and higher in those in early career (50% vs. senior career, 25%) and mid-career(45%; vs. senior career; χ²(6) = 29.19, p < 0.001). The workforce appeared geographically static but unstable with only 50% of people having their contract renewed or extended. The majority of people (59%) have never been promoted by their institution. There were perceptions of gender equality even in the context of females being underrepresented in senior positions (19% vs. males 39%; χ²(3) = 8.43, p = 0.015). Despite these findings, the majority of the workforce reported positive organisational and cultural experiences.

Conclusions

Sustainability of a academic primary care career is undermined by unclear pathways and a lack of promotion. If the discipline is to thrive, there is a need to support early and mid-career individuals via greater transparency of career pathways. Despite these findings staff remained positive about their careers.

Electronic supplementary material

The online version of this article (doi:10.1186/s12875-014-0205-6) contains supplementary material, which is available to authorized users.

The development and application of audit criteria for assessing knowledge exchange plans in health research grant applications

Background

Research funders expect evidence of end user engagement and impact plans in research proposals. Drawing upon existing frameworks, we developed audit criteria to help researchers and their institutions assess the knowledge exchange plans of health research proposals.

Findings

Criteria clustered around five themes: problem definition; involvement of research users; public and patient engagement; dissemination and implementation; and planning, management and evaluation of knowledge exchange. We applied these to a sample of grant applications from one research institution in the United Kingdom to demonstrate feasibility.

Conclusion

Our criteria may be useful as a tool for researcher self-assessment and for research institutions to assess the quality of knowledge exchange plans and identify areas for systematic improvement.

Detail and the devil of on-farm parasite control under climate change

Levels and seasonal patterns of parasite challenge to livestock are likely to be affected by climate change, through direct effects on life cycle stages outside the definitive host and through alterations in management that affect exposure and susceptibility. Net effects and options for adapting to them will depend very strongly on details of the system under consideration. This short paper is not a comprehensive review of climate change effects on parasites, but rather seeks to identify key areas in which detail is important and arguably under-recognized in supporting farmer adaptation. I argue that useful predictions should take fuller account of system-specific properties that influence disease emergence, and not just the effects of climatic variables on parasite biology. At the same time, excessive complexity is ill-suited to useful farm-level decision support. Dealing effectively with the ‘devil of detail’ in this area will depend on finding the right balance, and will determine our success in applying science to climate change adaptation by farmers.

Towards a new paradigm in health research and practice? Collaborations for Leadership in Applied Health Research and Care

PURPOSE

Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) are a new UK initiative to promote collaboration between universities and healthcare organisations in carrying out and applying the findings of applied health research. But they face significant, institutionalised barriers to their success. This paper seeks to analyse these challenges and discuss prospects for overcoming them.

DESIGN/METHODOLOGY/APPROACH

The paper draws on in-depth qualitative interview data from the first round of an ongoing evaluation of one CLAHRC to understand the views of different stakeholders on its progress so far, challenges faced, and emergent solutions.

FINDINGS

The breadth of CLAHRCs' missions seems crucial to mobilise the diverse stakeholders needed to succeed, but also produces disagreement about what the prime goal of the Collaborations should be. A process of consensus building is necessary to instil a common vision among CLAHRC members, but deep-seated institutional divisions continue to orient them in divergent directions, which may need to be overcome through other means.

ORIGINALITY/VALUE

This analysis suggests some of the key means by which those involved in joint enterprises such as CLAHRCs can achieve consensus and action towards a current goal, and offers recommendations for those involved in their design, commissioning and performance management.

Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators

BACKGROUND

Funders of medical research the world over are increasingly seeking, in research assessment, to complement traditional output measures of scientific publications with more outcome-based indicators of societal and economic impact. In the United Kingdom, the Higher Education Funding Council for England (HEFCE) developed proposals for the Research Excellence Framework (REF) to allocate public research funding to higher education institutions, inter alia, on the basis of the social and economic impact of their research. In 2010, it conducted a pilot exercise to test these proposals and refine impact indicators and criteria.

METHODS

The impact indicators proposed in the 2010 REF impact pilot exercise are critically reviewed and appraised using insights from the relevant literature and empirical data collected for the University of Oxford's REF pilot submission in clinical medicine. The empirical data were gathered from existing administrative sources and an online administrative survey carried out by the university's Medical Sciences Division among 289 clinical medicine faculty members (48.1% response rate).

RESULTS

The feasibility and scope of measuring research impact in clinical medicine in a given university are assessed. Twenty impact indicators from seven categories proposed by HEFCE are presented; their strengths and limitations are discussed using insights from the relevant biomedical and research policy literature.

CONCLUSIONS

While the 2010 pilot exercise has confirmed that the majority of the proposed indicators have some validity, there are significant challenges in operationalising and measuring these indicators reliably, as well as in comparing evidence of research impact across different cases in a standardised manner. It is suggested that the public funding agencies, medical research charities, universities, and the wider medical research community work together to develop more robust methodologies for capturing and describing impact, including more valid and reliable impact indicators.