Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme

A multidimensional assessment of payback: Evaluating the impact of Breast Cancer Trials

AIM

This study evaluates the impact created by clinical trial investment and research undertaken by Breast Cancer Trials (BCT).

METHODS

The approach involved using the Payback Framework, and included a review of BCT archival information, public health data, a questionnaire sent to BCT members, individual interviews with key informants, a focus group with members of the organization's Consumer Advisory Panel, and case studies of select BCT trials. The evaluation assessed the evidence against the Payback Framework criteria: informing policy and product development, knowledge production, the research system, health and health sector benefits, and broader economic benefits.

RESULTS

Analysis using the Payback Framework revealed impact was created in each category and a range of positive outcomes were identified for various stakeholder groups. BCT is maximizing the impact of its research and contributing to a global pool of scientific knowledge by collaborating with over 100 institutions and 820 researchers, yet its benefits go beyond research contributions. Findings highlight strong financial returns from BCT's research program and that BCT has played an important role in enhancing the public perception of clinical trials by promoting participation in trials, educating and empowering participants, and communicating trial outcomes.

CONCLUSION

BCT's clinical trials have had a significant impact on the prevention, detection, treatment, and management of breast cancer. Measuring and reporting impact can be resource intensive but support BCT in remaining accountable to all those invested in the organization and in breast cancer clinical trials, evidencing the multiple dimensions of payback resulting from the organization's research.

A scoping review of the globally available tools for assessing health research partnership outcomes and impacts

BACKGROUND

Health research partnership approaches have grown in popularity over the past decade, but the systematic evaluation of their outcomes and impacts has not kept equal pace. Identifying partnership assessment tools and key partnership characteristics is needed to advance partnerships, partnership measurement, and the assessment of their outcomes and impacts through systematic study.

OBJECTIVE

To locate and identify globally available tools for assessing the outcomes and impacts of health research partnerships.

METHODS

We searched four electronic databases (Ovid MEDLINE, Embase, CINAHL + , PsychINFO) with an a priori strategy from inception to June 2021, without limits. We screened studies independently and in duplicate, keeping only those involving a health research partnership and the development, use and/or assessment of tools to evaluate partnership outcomes and impacts. Reviewer disagreements were resolved by consensus. Study, tool and partnership characteristics, and emerging research questions, gaps and key recommendations were synthesized using descriptive statistics and thematic analysis.

RESULTS

We screened 36 027 de-duplicated citations, reviewed 2784 papers in full text, and kept 166 studies and three companion reports. Most studies originated in North America and were published in English after 2015. Most of the 205 tools we identified were questionnaires and surveys targeting researchers, patients and public/community members. While tools were comprehensive and usable, most were designed for single use and lacked validity or reliability evidence. Challenges associated with the interchange and definition of terms (i.e., outcomes, impacts, tool type) were common and may obscure partnership measurement and comparison. Very few of the tools identified in this study overlapped with tools identified by other, similar reviews. Partnership tool development, refinement and evaluation, including tool measurement and optimization, are key areas for future tools-related research.

CONCLUSION

This large scoping review identified numerous, single-use tools that require further development and testing to improve their psychometric and scientific qualities. The review also confirmed that the health partnership research domain and its measurement tools are still nascent and actively evolving. Dedicated efforts and resources are required to better understand health research partnerships, partnership optimization and partnership measurement and evaluation using valid, reliable and practical tools that meet partners' needs.

The value and impact of health technology assessment: discussions and recommendations from the 2023 Health Technology Assessment International Global Policy Forum

Health technology assessment (HTA) programs inform decision making about the value and reimbursement of new and existing health technologies; however, they are under increasing pressure to demonstrate that they are a cost-effective use of finite healthcare resources themselves. The 2023 HTAi Global Policy Forum (GPF) discussed the value and impact of HTA, including how it is assessed and communicated, and how it could be enhanced in the future. This article summarizes the discussions held at the 2023 HTAi GPF, where the challenges and opportunities related to the value and impact of HTA were debated. Core themes and recommendations identified that defining the purpose of value and impact assessment is an essential first step prior to undertaking it, and that it can be done through the use and expansion of existing tools. Further work around aligning HTA programs with underlying societal values is needed to ensure the long-term value and impact of HTA. HTA could also have a role in assessing the efficiency of the wider health system by applying HTA methods or concepts to broader budgetary allocations and organizational aspects of health care. Stakeholders (particularly patients, industry, and clinicians but also payers, wider society, and the media) should ideally be actively engaged when undertaking the value and impact assessment of HTA. More concerted efforts in communicating the role and remit of HTA bodies would also help stakeholders to better understand the value and impact of HTA, which in turn could improve the implementation of HTA recommendations and application to future actions in the lifecycle of technologies.

Utilization of innovative medical technologies in German inpatient care: does evidence matter?

BACKGROUND

The reimbursement of new technologies in inpatient care is not always linked to a requirement for evidence-based evaluation of patient benefit. In Germany, every new technology approved for market was until recently eligible for reimbursement in inpatient care unless explicitly excluded. The aim of this work was (1) to investigate the type of evidence that was available at the time of introduction of 25 innovative technologies and how this evidence evolved over time, and (2) to explore the relationship between clinical evidence and utilization for these technologies in German inpatient care.

METHODS

This study combined different methods. A systematic search for evidence published between 2003 and 2017 was conducted in four bibliographic databases, clinical trial registries, resources for clinical guidelines, and health technology assessment-databases. Information was also collected on funding mechanisms and safety notices. Utilization was measured by hospital procedures captured in claims data. The body of evidence, funding and safety notices per technology were analyzed descriptively. The relationship between utilization and evidence was explored empirically using a multilevel regression analysis.

RESULTS

The number of included publications per technology ranges from two to 498. For all technologies, non-comparative studies form the bulk of the evidence. The number of randomized controlled clinical trials per technology ranges from zero to 19. Some technologies were utilized for several years without an adequate evidence base. A relationship between evidence and utilization could be shown for several but not all technologies.

CONCLUSIONS

This study reveals a mixed picture regarding the evidence available for new technologies, and the relationship between the development of evidence and the use of technologies over time. Although the influence of funding and safety notices requires further investigation, these results re-emphasize the need for strengthening market approval standards and HTA pathways as well as approaches such as coverage with evidence development.

The impact of the European Association of Endoscopic Surgery research grant scheme-a mixed qualitative quantitative methodology study protocol

Background

The European Association of Endoscopic Surgery (EAES) is a surgical society who promotes the development and expansion of minimally invasive surgery to surgeons and surgical trainees. It does so through its activities in education, training, and research. The EAES research committee aims to promote the highest quality clinical research in endoscopic and minimally invasive surgery. They have provided grant funding since 2009 in education, surgery, and basic science. Despite the success and longevity of the scheme, the academic and non-academic impact of the research funding scheme has not been evaluated.

Aims

The primary aim of this project is to assess the short, long term academic and real world impact of the EAES funding scheme. The secondary aims are to identify barriers and facilitators for achieving good impact.

Methods

This will be a mixed qualitative and quantitative study. Semi-structured interviews will be performed with previous grant recipients. The questions for the interviews will be selected after a consensus is achieved amongst the members of the steering committee of this project. The responses will be transcribed and thematic analysis will be applied. The results of the thematic analysis will be used to populate a questionnaire which will be disseminated to grant recipients. This study is kindly funded by the EAES.

Discussion

The first question this project is expected to answer is whether the EAES research funding scheme had a significant positive impact on research output, career progression but also non-academic output such as change in clinical guidelines, healthcare quality and cost-effectiveness improvement. This project however is also expected to identify facilitators and barriers to successful completion of projects and to achieving high impact. This will inform EAES and the rest of the surgical and academic communities as to how clinicians would like to be supported when conducting research. There should also be a positive and decisive change towards removing factors that hinder the timely and successful completion of projects.

Impact assessment of the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery

BACKGROUND

Research impact is an emerging measure of research achievement alongside traditional academic outputs such as publications. We present the results of applying the Framework to Assess the Impact from Translational health research (FAIT) to the Centre for Research Excellence (CRE) in Stroke Rehabilitation and Brain Recovery (CRE-Stroke, 2014-2019) and report on the feasibility and lessons from the application of FAIT to a CRE rather than a discrete research project.

METHODS

Data were gathered via online surveys, in-depth interviews, document analysis and review of relevant websites/databases to report on the three major FAIT methods: the modified Payback Framework, an assessment of costs against monetized consequences, and a narrative account of the impact generated from CRE-Stroke activities. FAIT was applied during the last 4 years of CRE-Stroke operation.

RESULTS

With an economic investment of AU$ 3.9 million over 5 years, CRE-Stroke delivered a return on investment that included AU$ 18.8 million in leveraged grants, fellowships and consultancies. Collectively, CRE-Stroke members produced 354 publications that were accessed 470,000 times and cited over 7220 times. CRE-Stroke supported 26 PhDs, 39 postdocs and seven novice clinician researchers. There were 59 capacity-building events benefiting 744 individuals including policy-makers and consumers. CRE-Stroke created research infrastructure (including a research register of stroke survivors and a brain biobank), and its global leadership produced international consensus recommendations to influence the stroke research landscape worldwide. Members contributed to the Australian Living Stroke Guidelines: four researchers' outputs were directly referenced. Based only on the consequences that could be monetized, CRE-Stroke returned AU$ 4.82 for every dollar invested in the CRE.

CONCLUSION

This case example in the developing field of impact assessment illustrates how researchers can use evidence to demonstrate and report the impact of and returns on research investment. The prospective application of FAIT by a dedicated research impact team demonstrated impact in broad categories of knowledge-gain, capacity-building, new infrastructure, input to policy and economic benefits. The methods can be used by other research teams to provide comprehensive evidence to governments and other research funders about what has been generated from their research investment but requires dedicated resources to complete.

Describing the state of a research network: A mixed methods approach to network evaluation

Diabetes Action Canada Strategy for Patient-Oriented Research (SPOR) Network in Chronic Disease was formed in 2016 and is funded primarily through the Canadian Institutes of Health Research (CIHR). We propose a novel mixed-methods approach to a network evaluation integrating the State of Network Evaluation framework and the Canadian Academy of Health Sciences (CAHS) preferred framework and indicators. We measure key network themes of connectivity, health and results, and impact and return on investment associated with health research networks. Our methods consist of a longitudinal cross-sectional network survey of members and social network analysis to examine Network Connectivity and assess the frequency of interactions, the topics discussed during them, and how networking effectively facilitates interactions and collaboration among members. Network Health will be evaluated through semistructured interviews, a membership survey inquiring about satisfaction and experience with the Network, and a review of documentary sources related to funding and infrastructure to evaluate Network Sustainability. Finally, we will examine Network Results and Impact using the CAHS preferred framework and indicators to measure returns on investment in health research across the five domains of the CAHS framework, which include: advancing knowledge, capacity building, informing decision making, health impact, and economic and social impact. Indicators will be assessed with various methods, including bibliometric analyses, review of relevant documentary sources (annual reports), member activities informing health and research policy, and Patient Partner involvement. The Network Evaluation will provide members and stakeholders with information for planning, improvements, and funding future Network endeavors.

What do we want to get out of this? a critical interpretive synthesis of the value of process evaluations, with a practical planning framework

BACKGROUND

Process evaluations aim to understand how complex interventions bring about outcomes by examining intervention mechanisms, implementation, and context. While much attention has been paid to the methodology of process evaluations in health research, the value of process evaluations has received less critical attention. We aimed to unpack how value is conceptualised in process evaluations by identifying and critically analysing 1) how process evaluations may create value and 2) what kind of value they may create.

METHODS

We systematically searched for and identified published literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues. We conducted a critical interpretive synthesis and developed a practical planning framework.

RESULTS

We identified and included 147 literature items. From these we determined three ways in which process evaluations may create value or negative consequences: 1) through the socio-technical processes of 'doing' the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge. We identified 15 value themes. We also found that value varies according to the characteristics of individual process evaluations, and is subjective and context dependent.

CONCLUSION

The concept of value in process evaluations is complex and multi-faceted. Stakeholders in different contexts may have very different expectations of process evaluations and the value that can and should be obtained from them. We propose a planning framework to support an open and transparent process to plan and create value from process evaluations and negotiate trade-offs. This will support the development of joint solutions and, ultimately, generate more value from process evaluations to all.

Researchers’ perspective of real-world impact from UK public health research: A qualitative study

Research funded by the National Institute for Health Research Public Health Research Programme is being undertaken in a complex system which brings opportunities and challenges for researchers to maximise the impact of their research. This study seeks to better understand the facilitators, challenges and barriers to research impact and knowledge mobilisation from the perspective of UK public health researchers. A qualitative study using semi-structured interviews, informed by the Payback Framework, with public health researchers who held a research award with the National Institute for Health Research Public Health Research programme up to March 2017 was conducted. Following a thematic analysis, three strongly interlinked themes were extracted from the data and three key factors were highlighted as important for facilitating knowledge mobilisation and impact in UK public health research: (1) Public health researcher’s perception of the purpose of the research (2) Approaches to undertaking Knowledge mobilisation activities (3) The complex nature of public health research in the wider research context. These have been reflected onto the Payback framework. Public health researchers can maximise the likelihood for impact by being aware of the context in which they are undertaking research, using different methods, and employing several strategies to take advantage of opportunities. There is a need to support researchers with knowledge mobilisation activities and for funders to identify their expectations of the impact resulting from research. Our findings have relevance to public health researchers and funders interested in increasing the benefit that research brings to society.

Research can be integrated into public health policy-making: global lessons for and from Spanish economic evaluations

WHO promotes the use of research in policy-making to drive improvements in health, including in achieving Sustainable Development Goals such as tobacco control. The European Union’s new €95 billion Horizon Europe research framework programme parallels these aims, and also includes commitments to fund economic evaluations. However, researchers often express frustration at the perceived lack of attention to scientific evidence during policy-making. For example, some researchers claim that evidence regarding the return on investment from optimal implementation of evidence-based policies is frequently overlooked. An increasingly large body of literature acknowledges inevitable barriers to research use, but also analyses facilitators encouraging such use. This opinion piece describes how some research is integrated into policy-making. It highlights two recent reviews. One examines impact assessments of 36 multi-project research programmes and identifies three characteristics of projects more likely to influence policy-making. These include a focus on healthcare system needs, engagement of stakeholders, and research conducted for organizations supported by structures to receive and use evidence. The second review suggests that such characteristics are likely to occur as part of a comprehensive national health research system strategy, especially one integrated into the healthcare system. We also describe two policy-informing economic evaluations conducted in Spain. These examined the most cost-effective package of evidence-based tobacco control interventions and the cost-effectiveness of different strategies to increase screening coverage for cervical cancer. Both projects focused on issues of healthcare concern and involved considerable stakeholder engagement. The Spanish examples reinforce some lessons from the global literature and, therefore, could help demonstrate to authorities in Spain the value of developing comprehensive health research systems, possibly following the interfaces and receptor model. The aim of this would be to integrate needs assessment and stakeholder engagement with structures spanning the research and health systems. In such structures, economic evaluation evidence could be collated, analysed by experts in relation to healthcare needs, and fed into both policy-making as appropriate, and future research calls. The increasingly large local and global evidence base on research utilization could inform detailed implementation of this approach once accepted as politically desirable. Given the COVID-19 pandemic, increasing the cost-effectiveness of healthcare systems and return on investment of public health interventions becomes even more important.

Collective health research assessment: developing a tool to measure the impact of multistakeholder research initiatives

Background

The need to more collaboratively measure the impact of health research and to do so from multidimensional perspectives has been acknowledged. A scorecard was developed as part of the Collective Research Impact Framework (CRIF), to engage stakeholders in the assessment of the impacts of health research and innovations. The purpose of this study was to describe the developmental process of the MULTI-ACT Master Scorecard (MSC) and how it can be used as a workable tool for collectively assessing future responsible research and innovation measures.

Methods

An extensive review of the health research impact literature and of multistakeholder initiatives resulted in a database of 1556 impact indicators. The MSC was then cocreated by engaging key stakeholders and conducting semi-structured interviews of experts in the field.

Results

The MSC consists of five accountability dimensions: excellence, efficacy, economic, social and patient-reported outcomes. The tool contains 125 potential indicators, classified into 53 impact measurement aspects that are considered the most relevant topics for multistakeholder research and innovation initiatives when assessing their impact on the basis of their mission and their stakeholders’ interests. The scorecard allows the strategic management of multistakeholder research initiatives to demonstrate their impact on people and society. The value of the tool is that it is comprehensive, customizable and easy to use.

Conclusions

The MSC is an example of how the views of society can be taken into account when research impacts are assessed in a more sustainable and balanced way. The engagement of patients and other stakeholders is an integral part of the CRIF, facilitating collaborative decision-making in the design of policies and research agendas. In policy making, the collective approach allows the evaluation perspective to be extended to the needs of society and towards responsible research and innovation. Multidimensionality makes research and innovations more responsive to systemic challenges, and developing more equitable and sustainable health services.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-022-00856-9.

Looking for evidence of research impact and use: A qualitative study of an Australian research-policy system

Current assessments of research impact have been criticized for capturing what can be easily counted not what actually counts. To empirically examine this issue, we approached measuring research impact from two directions, tracing forwards from research and backwards from policy, within a defined research-policy system (childhood obesity prevention research and policy in New South Wales, Australia from 2000 to 2015). The forward tracing research impact assessment component traced a sample of 148 local research projects forward to examine their policy impacts. Of the projects considered, 16% had an impact on local policy and for a further 19%, decision-makers were aware of the research, but there was no evidence it influenced policy decisions. The backward tracing component of the study included an analysis of research use across three policy initiatives. It provided a more nuanced understanding of the relative influence of research on policy. Both direct uses of specific research and indirect uses of research incorporated as broader bodies of knowledge were evident. Measuring research impact from both directions captured the diverse ways that research was used in decision-making. Our findings illustrate complexities in the assessment process and in real-life policymaking trajectories. They highlight the role that timing of assessment plays in perception of impacts and difficulties attributing longer-term impacts to specific research. This study supports the use of models where politics and complex system dynamics shape knowledge and its influence on decision-making, rather than research being the primary driver for policy change.

Fundamental challenges in assessing the impact of research infrastructure

Clinical research infrastructure is one of the unsung heroes of the scientific response to the current COVID-19 pandemic. The extensive, long-term funding into research support structures, skilled people, and technology allowed the United Kingdom research response to move off the starting blocks at pace by utilizing pre-existing platforms. The increasing focus from funders on evaluating the outcomes and impact of research infrastructure investment requires both a reframing and progression of the current models in order to address the contribution of the underlying support infrastructure. The majority of current evaluation/outcome models focus on a "pipeline" approach using a methodology which follows the traditional research funding route with the addition of quantitative metrics. These models fail to embrace the complexity caused by the interplay of previous investment, the coalescing of project outputs from different funders, the underlying infrastructure investment, and the parallel development across different parts of the system. Research infrastructure is the underpinning foundation of a project-driven research system and requires long-term, sustained funding and capital investment to maintain scientific and technological expertise. Therefore, the short-term focus on quantitative metrics that are easy to collect and interpret and that can be assessed in a roughly 5-year funding cycle needs to be addressed. The significant level of investment in research infrastructure necessitates investment to develop bespoke methodologies that develop fit-for-purpose, longer-term/continual approach(es) to evaluation. Real-world research should reflect real-world evaluation and allow for the accrual of a narrative of value indicators that build a picture of the contribution of infrastructure to research outcomes. The linear approach is not fit for purpose, the research endeavour is a complex, twisted road, and the evaluation approach needs to embrace this complexity through the development of realist approaches and the rapidly evolving data ecosystem. This paper sets out methodological challenges and considers the need to develop bespoke methodological approaches to allow a richer assessment of impact, contribution, attribution, and evaluation of research infrastructure. This paper is the beginning of a conversation that invites the community to "take up the mantle" and tackle the complexity of real-world research translation and evaluation.

Exploring research impact models: A systematic scoping review

This article explores the models and frameworks developed on “research impact’. We aim to provide a comprehensive overview of related literature through scoping study method. The present research investigates the nature, objectives, approaches, and other main attributes of the research impact models. It examines to analyze and classify models based on their characteristics. Forty-seven studies and 10 reviews published between 1996 and 2020 were included in the analysis. The majority of models were developed for the impact assessment and evaluation purposes. We identified three approaches in the models, namely outcome-based, process-based, and those utilized both of them, among which the outcome-based approach was the most frequently used by impact models and evaluation was considered as the main objective of this group. The process-based ones were mainly adapted from the W.K. Kellogg Foundation logic model and were potentially eligible for impact improvement. We highlighted the scope of processes and other specific features for the recent models. Given the benefits of the process-based approach in enhancing and accelerating the research impact, it is important to consider such approach in the development of impact models. Effective interaction between researchers and stakeholders, knowledge translation, and evidence synthesis are the other possible driving forces contributing to achieve and improve impact.

Assessing the performance of health technology assessment (HTA) agencies: developing a multi-country, multi-stakeholder, and multi-dimensional framework to explore mechanisms of impact

Background

Health technology assessment (HTA) agencies have an important role to play in managing the rising demands on health systems. However, creating and running such agencies potentially diverts resources from frontline services. A large number of studies address the question of ‘what is the impact of HTA?’. Several points of heterogeneity in this literature include: purpose of the study, definition of HTA, definition of impact, and scope and rigour of evaluations. Our study seeks to address several limitations in this literature. This study aims to explore the mechanisms of impact of an HTA agency. In doing so, we consider HTA as an institution rather than a knowledge product to build an impact evaluation framework from an international, multi-stakeholder and multi-dimensional perspective.

Methods

We conducted 9 key informant interviews with experts from the international HTA community. We addressed several questions, informed by existing frameworks of impact within the literature, to understand their perspectives on the mechanisms of impact of an HTA agency. We analyse data using logic modelling and impact mapping, as tools to understand and visualise mechanisms of change.

Findings

Our impact mapping highlights several distinct, but not necessarily mutually exclusive, mechanisms through which the overall impact of an HTA agency is achieved. These are: the effective conduct of HTA studies; effective use of HTA in agenda-setting and policy formulation processes; effective engagement and external communications; good institutional reputation and fit within the healthcare and policy-making system; effective use of HTA as a tool for the negotiation of health technology prices; and the effective implementation of policy change regarding health technologies. We also identify indicators of these effects.

Conclusions

Our findings and resulting evaluation framework complement and add to existing literature by offering a new perspective on the mechanisms by which HTA agencies generate impact. This new perspective considers HTA as an institution rather than a knowledge product, is international, multi-dimensional, and includes multi-stakeholder views. We hope the analysis will be useful to countries interested in managing HTA performance.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12962-021-00290-8.

Primary Care Research - Influencing and Implementing Into Policy

This editorial describes how research in primary health care can be used to influence policy. It draws on previous literature to give an example from the UK of how research in one part of primary care, the health-visiting service, has endeavoured to use evidence to influence policy and practice. The editorial considers frameworks for policy implementation such as Bardach's eight phase approach and concepts that can inform policy implementation such as Lipsky's Street-Level Bureaucrat approach.

Documenting HIV research-utilization activities, outputs and outcomes: examples and lessons learned from Project SOAR

The importance of using research findings to inform policy and program decisions is well recognized, but the literature on measuring research utilization activities is scarce. As funding to support some areas of research wanes or remains stagnant, the need to document the value of investing in research by its’ effect on improved programs and policies becomes increasingly necessary. We present the experience of Project SOAR, a six-year USAID-funded project focusing on HIV/AIDS-related implementation research, to demonstrate measurement of research utilization. We follow the project’s research-utilization logic model, including inputs, activities, outputs, and outcomes. We present tools the project developed and examples from project studies and discuss what works, remaining challenges and how to overcome them, and lessons learned. We then make recommendations for incorporating research-utilization activities and measurement in implementation-research studies.

Evaluating cancer research impact: lessons and examples from existing reviews on approaches to research impact assessment

BACKGROUND

Performing cancer research relies on substantial financial investment, and contributions in time and effort from patients. It is therefore important that this research has real life impacts which are properly evaluated. The optimal approach to cancer research impact evaluation is not clear. The aim of this study was to undertake a systematic review of review articles that describe approaches to impact assessment, and to identify examples of cancer research impact evaluation within these reviews.

METHODS

In total, 11 publication databases and the grey literature were searched to identify review articles addressing the topic of approaches to research impact assessment. Information was extracted on methods for data collection and analysis, impact categories and frameworks used for the purposes of evaluation. Empirical examples of impact assessments of cancer research were identified from these literature reviews. Approaches used in these examples were appraised, with a reflection on which methods would be suited to cancer research  impact evaluation going forward.

RESULTS

In total, 40 literature reviews were identified. Important methods to collect and analyse data for impact assessments were surveys, interviews and documentary analysis. Key categories of impact spanning the reviews were summarised, and a list of frameworks commonly used for impact assessment was generated. The Payback Framework was most often described. Fourteen examples of impact evaluation for cancer research were identified. They ranged from those assessing the impact of a national, charity-funded portfolio of cancer research to the clinical practice impact of a single trial. A set of recommendations for approaching cancer research impact assessment was generated.

CONCLUSIONS

Impact evaluation can demonstrate if and why conducting cancer research  is worthwhile. Using a mixed methods, multi-category assessment organised within a framework, will provide a robust evaluation, but the ability to perform this type of assessment may be constrained by time and resources. Whichever approach is used, easily measured, but inappropriate metrics should be avoided. Going forward, dissemination of the results of cancer research impact assessments will allow the cancer research community to learn how to conduct these evaluations.

A research impact model for work and health

Research organizations, governments and funding agencies are increasingly interested in the impact of research beyond academia. While a growing literature describes research impacts in healthcare and health services, little has focused on occupational health and safety research. This article describes a research impact model that has been in use for over a decade. The model was developed to track and describe the impact of research conducted by a mid-sized institute that focuses on work and health. Model development was informed by existing models, with the goal of contextualizing the institute's case studies describing three types of research impact: evidence of the diffusion of research; evidence of research informing decision-making; and evidence of societal impact. A logic model describes research actions and outcomes, as well as key audiences and knowledge transfer approaches. A unique element is its indication of the level of difficulty in determining types of impact. The model compares well with current research impact models developed or used in healthcare and health services research, and it has been useful in guiding a mid-sized research organization's process for tracking and describing the impact of its research. It may be useful to other small and mid-sized research organizations that focus on workplace health and safety.

Tracing Long-Term Outcomes of Basic Research Using Citation Networks

In recent years, the science of science policy has been facilitated by the greater availability of and access to digital data associated with the science, technology, and innovation enterprise. Historically, most of the studies from which such data are derived have been econometric or "scientometric" in nature, focusing on the development of quantitative data, models, and metrics of the scientific process as well as outputs and outcomes. Broader definitions of research impact, however, necessitate the use of qualitative case-study methods. For many years, U.S. federal science agencies such as the National Institutes of Health have demonstrated the impact of the research they support through tracing studies that document critical events in the development of successful technologies. A significant disadvantage and barrier of such studies is the labor-intensive nature of a case study approach. Currently, however, the same data infrastructures that have been developed to support scientometrics may also facilitate historical tracing studies. In this paper, we describe one approach we used to discover long-term, downstream outcomes of research supported in the late 1970's and early 1980's by the National Institute of General Medical Sciences, a component of the National Institutes of Health.

Research and Policy Impact of Trials Published by the UK National Institute of Health Research (2006-2015)

OBJECTIVES

Health technology assessment aims to inform and support healthcare decision making, and trials are part of that process. The purpose of this study was to measure the impact of a sample of trials in a meaningful but robust fashion.

METHODS

All randomized controlled trials funded and published by the UK National Institute of Health Research in the Health Technology Assessment journal series and other peer-reviewed journals were identified for 2006 to 2015. Citation analysis was performed for all trials, and quantitative content analysis was undertaken on a purposive sample to determine whether impact could be categorized as "instrumental" (ie, having a clear influence on key research and policy publications).

RESULTS

The search identified 133 relevant trials. The citation rate per trial was 102.97. Of the 133 trials, 129 (98%) were cited in 1 or more systematic reviews or meta-analyses (mean per trial = 7.18, range = 0-44). Where they were cited, the trials were used in some form of synthesis 63% of the time. Ninety-one of the 133 (68%) trials were found to be cited in 1 or more guidance or policy document (mean per trial = 2.75, range = 0-26) and had an instrumental influence 41% of the time. The publication of these trials' results in journals other than the Health Technology Assessment journal appears to enhance the discoverability of the trial data. Altmetric.com proved to be very useful in identifying unique policy and guidance documents.

CONCLUSION

These trials have impressive citation rates, and a sizeable proportion are certainly being used in key publications in a genuinely instrumental manner.

Impact of surgical intervention trials on healthcare: A systematic review of assessment methods, healthcare outcomes, and determinants

BACKGROUND

Frameworks used in research impact evaluation studies vary widely and it remains unclear which methods are most appropriate for evaluating research impact in the field of surgical research. Therefore, we aimed to identify and review the methods used to assess the impact of surgical intervention trials on healthcare and to identify determinants for surgical impact.

METHODS

We searched journal databases up to March 10, 2020 for papers assessing the impact of surgical effectiveness trials on healthcare. Two researchers independently screened the papers for eligibility and performed a Risk of Bias assessment. Characteristics of both impact papers and trial papers were summarized. Univariate analyses were performed to identify determinants for finding research impact, which was defined as a change in healthcare practice.

RESULTS

Sixty-one impact assessments were performed in 37 included impact papers. Some surgical trial papers were evaluated in more than one impact paper, which provides a total of 38 evaluated trial papers. Most impact papers were published after 2010 (n = 29). Medical records (n = 10), administrative databases (n = 22), and physician's opinion through surveys (n = 5) were used for data collection. Those data were analyzed purely descriptively (n = 3), comparing data before and after publication (n = 29), or through time series analyses (n = 5). Significant healthcare impact was observed 49 times and more often in more recent publications. Having impact was positively associated with using medical records or administrative databases (ref.: surveys), a longer timeframe for impact evaluation and more months between the publication of the trial paper and the impact paper, data collection in North America (ref.: Europe), no economic evaluation of the intervention, finding no significant difference in surgical outcomes, and suggesting de-implementation in the original trial paper.

CONCLUSIONS AND IMPLICATIONS

Research impact evaluation receives growing interest, but still a small number of impact papers per year was identified. The analysis showed that characteristics of both surgical trial papers and impact papers were associated with finding research impact. We advise to collect data from either medical records or administrative databases, with an evaluation time frame of at least 4 years since trial publication.

Mechanisms and pathways to impact in public health research: a preliminary analysis of research funded by the National Institute for Health Research (NIHR)

Background

The mechanisms and pathways to impacts from public health research in the UK have not been widely studied. Through the lens of one funder (NIHR), our aims are to map the diversity of public health research, in terms of funding mechanisms, disciplinary contributions, and public health impacts, identify examples of impacts, and pathways to impact that existing reporting mechanisms may not otherwise have captured, and provide illustrations of how public health researchers perceive the generation of non-academic impact from their work.

Methods

A total of 1386 projects were identified as ‘public health research’ by the NIHR and listed in the NIHR Public Health Overview database (2000–2016). From these, a subset of 857 projects were matched as potentially having begun reporting impacts via an external data-gathering platform (Researchfish). Data on the 857 projects were analyzed quantitatively, and nine projects were selected to investigate further through semi-structured interviews with principal investigators. Two workshops took place to validate emerging and final findings and facilitate analysis.

Results

In addition to the NIHR School for Public Health Research and the NIHR Public Health Research Programme, 89% of projects contained in the NIHR Public Health Overview portfolio as ‘public health research’ are funded via other NIHR research programmes, suggesting significant diversity in disciplines contributing to public health research and outcomes. The pathways to impact observed in our in-depth case studies include contributing to debates on what constitutes appropriate evidence for national policy change, acknowledging local ‘unintended’ impacts, building trusted relationships with stakeholders across health and non-health sectors and actors, collaborating with local authorities, and using non-academic dissemination channels.

Conclusions

Public health as a discipline contributes substantially to impact beyond academia. To support the diversity of these impacts, we need to recognise localized smaller-scale impacts, and the difference in types of evidence required for community and local authority-based impacts. This will also require building capacity and resources to enable impact to take place from public health research. Finally, support is required for engagement with local authorities and working with non-health sectors that contribute to health outcomes.

How do organisations implement research impact assessment (RIA) principles and good practice? A narrative review and exploratory study of four international research funding and administrative organisations

Background

Public research funding agencies and research organisations are increasingly accountable for the wider impacts of the research they support. While research impact assessment (RIA) frameworks and tools exist, little is known and shared of how these organisations implement RIA activities in practice.

Methods

We conducted a review of academic literature to search for research organisations’ published experiences of RIAs. We followed this with semi-structured interviews from a convenience sample (n = 7) of representatives of four research organisations deploying strategies to support and assess research impact.

Results

We found only five studies reporting empirical evidence on how research organisations put RIA principles into practice. From our interviews, we observed a disconnect between published RIA frameworks and tools, and the realities of organisational practices, which tended not to be reported.

We observed varying maturity and readiness with respect to organisations’ structural set ups for conducting RIAs, particularly relating to leadership, skills for evaluation and automating RIA data collection. Key processes for RIA included efforts to engage researcher communities to articulate and plan for impact, using a diversity of methods, frameworks and indicators, and supporting a learning approach. We observed outcomes of RIAs as having supported a dialogue to orient research to impact, underpinned shared learning from analyses of research, and provided evidence of the value of research in different domains and to different audiences.

Conclusions

Putting RIA principles and frameworks into practice is still in early stages for research organisations. We recommend that organisations (1) get set up by considering upfront the resources, time and leadership required to embed impact strategies throughout the organisation and wider research ‘ecosystem’, and develop methodical approaches to assessing impact; (2) work together by engaging researcher communities and wider stakeholders as a core part of impact pathway planning and subsequent assessment; and (3) recognise the benefits that RIA can bring about as a means to improve mutual understanding of the research process between different actors with an interest in research.

Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?

Societal impact of research does not occur primarily as unexpected extraordinary incidents of particularly useful breakthroughs in science. It is more often a result of normal everyday interactions between organizations that need to create, exchange, and make use of new knowledge to further their goals. We use the distinctions between normal and extraordinary societal impact and between organizational- and individual-level activities and responsibilities to discuss how science–society relations can better be understood, evaluated, and improved by focusing on the organizations that typically interact in a specific domain of research.

A macroeconomic assessment of the impact of medical research expenditure: A case study of NIHR Biomedical Research Centres

Quantifying the value of investment in medical research can inform decision-making on the prioritisation of research programmes. Existing methodologies to estimate the rate of return of medical research are inappropriate for early-phase translational research due to censoring of health benefits and time lags. A strategy to improve the process of translational research for patient benefit has been initiated as part of the UK National Institute for Health Research (NIHR) investment in Biomedical Research Centres (BRCs) in England. By providing a platform for partnership between universities, NHS trusts and industry, successful BRCs should reduce time lags within translational research whilst also providing an impetus for local economic growth through industry collaboration. We present a novel contribution in the assessment of early-phase biomedical research by estimating the impact of the Oxford Biomedical Research Centre (OxBRC) on income and job creation following the initial NIHR investment. We adopt a macroeconomic assessment approach using Input-Output Analysis to estimate the value of medical research in terms of income and job creation during the early pathway towards translational biomedical research. Inter-industry linkages are assessed by building a model economy for the South East England region to estimate the return on investment of the OxBRC. The results from the input-output model estimate that the return on investment in biomedical research within the OxBRC is 46%. Each £1 invested in the OxBRC generates an additional £0.46 through income and job creation alone. Multiplicative employment effects following a marginal investment in the OxBRC of £98m during the period 2007-2017 result in an estimated additional 196 full time equivalent positions being created within the local economy on top of direct employment within OxBRC. Results from input-output analyses can be used to inform the prioritisation of biomedical research programmes when compared against national minimum thresholds of investment.

Assessing research impact: Australian Research Council criteria and the case of Family Wellbeing research

Researchers worldwide are increasingly reporting the societal impact of their research as part of national research productivity assessments. However, the challenges they encounter in developing their impact case studies against specified government assessment criteria and how pitfalls can be mitigated are not reported. This paper examines the key steps taken to develop an Aboriginal Family Wellbeing (FWB) empowerment research impact case study in the context of an Australian Research Council (ARC) pilot research impact assessment exercise and the challenges involved in applying the ARC criteria. The requirement that researchers demonstrate how their institutions support them to conduct impactful research has the potential to create supportive environments for researchers to be more responsive to the needs of users outside academia. However, the 15-year reference period for the associated research underpinning the reported impact and the focus on researcher's current institutional affiliation constitute potential constraints to demonstrating the true impact of research. For researchers working with Indigenous people, relationships that build over long periods of time, irrespective of university affiliation, are critical to conducting impactful research. A more open-ended time-frame, with no institutional restrictions for the 'associated research' provides the best opportunity to demonstrate the true benefits of research not only for Indigenous people but for Australian society more broadly.

The characteristics and reporting quality of research impact case studies: A systematic review

Despite the growing expectation that researchers report the impact of their research using a case study approach, systematic reviews of research impact have focused on frameworks, indicators, methods of data collection and assessment rather than impact case studies. Our aim is to provide an overview of the characteristics of published research impact case studies, including translation activities, and their reporting quality. We searched for peer-reviewed impact studies published between 2000 and 2018 using a case study approach and selected 25 suitable papers. We applied descriptive statistics to study characteristics, conducted thematic analysis of research translation activities and assessed reporting quality using the 10-point ISRIA statement. 24 papers reported intermediate impacts, such as advocacy, or the development of statements, tools, or technology. 4 reported on longer-term societal impacts, such as health outcomes and economic return on investment. 7 reported on translation activities. Papers scored well against the ISRIA statement on 5 domains of reporting quality. Weakest scores centred around identification of stakeholder needs and stakeholder involvement, and ethics and conflict of interest. We identified the need for more consistency in reporting through a case study approach, more systematic reporting of translation pathways and greater transparency concerning estimated costs and benefits of the research and its translation and impact assessment.

The impact generated by publicly and charity-funded research in the United Kingdom: a systematic literature review

Objective

To identify, synthesise and critically assess the empirical evidence of the impact generated by publicly and charity-funded health research in the United Kingdom.

Methods

We conducted a systematic literature review of the empirical evidence published in English in peer-reviewed journals between 2006 and 2017. Studies meeting the inclusion criteria were selected and their findings were analysed using the Payback Framework and categorised into five main dimensions, namely knowledge, benefits to future research and research use, benefits from informing policy and product development, health and health sector benefits, and broader economic benefits. We assessed the studies for risk of selection, reporting and funding bias.

Results

Thirteen studies met the inclusion criteria. The majority of the studies (10 out of 13) assessed impact at multiple domains including the main five key themes of the Payback Framework. All of them showed a positive impact of funded research on outcomes. Of those studies, one (8%), six (46%) and six (46%) presented a low, moderate and high risk of bias, respectively.

Conclusions

Empirical evidence on the impact of publicly and charity-funded research is still limited and subject to funding and selection bias. More work is needed to establish the causal effects of funded research on academic outcomes, policy, practice and the broader economy.

Electronic supplementary material

The online version of this article (10.1186/s12961-019-0425-2) contains supplementary material, which is available to authorized users.

Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA

The systematic use of evidence to inform healthcare decisions, particularly health technology assessment (HTA), has gained increased recognition. HTA has become a standard policy tool for informing decision makers who must manage the entry and use of pharmaceuticals, medical devices, and other technologies (including complex interventions) within health systems, for example, through reimbursement and pricing. Despite increasing attention to HTA activities, there has been no attempt to comprehensively synthesize good practices or emerging good practices to support population-based decision-making in recent years. After the identification of some good practices through the release of the ISPOR Guidelines Index in 2013, the ISPOR HTA Council identified a need to more thoroughly review existing guidance. The purpose of this effort was to create a basis for capacity building, education, and improved consistency in approaches to HTA-informed decision-making. Our findings suggest that although many good practices have been developed in areas of assessment and some other key aspects of defining HTA processes, there are also many areas where good practices are lacking. This includes good practices in defining the organizational aspects of HTA, the use of deliberative processes, and measuring the impact of HTA. The extent to which these good practices are used and applied by HTA bodies is beyond the scope of this report, but may be of interest to future researchers.

Embracing complexity and uncertainty to create impact: exploring the processes and transformative potential of co-produced research through development of a social impact model

The potential use, influence and impact of health research is seldom fully realised. This stubborn problem has caused burgeoning global interest in research aiming to address the implementation ‘gap’ and factors inhibiting the uptake of scientific evidence. Scholars and practitioners have questioned the nature of evidence used and required for healthcare, highlighting the complex ways in which knowledge is formed, shared and modified in practice and policy. This has led to rapid expansion, expertise and innovation in the field of knowledge mobilisation and funding for experimentation into the effectiveness of different knowledge mobilisation models. One approach gaining prominence involves stakeholders (e.g. researchers, practitioners, service users, policy-makers, managers and carers) in the co-production, and application, of knowledge for practice, policy and research (frequently termed integrated knowledge translation in Canada). Its popularity stems largely from its potential to address dilemmas inherent in the implementation of knowledge generated using more reductionist methods. However, despite increasing recognition, demands for co-produced research to illustrate its worth are becoming pressing while the means to do so remain challenging. This is due not only to the diversity of approaches to co-production and their application, but also to the ways through which different stakeholders conceptualise, measure, reward and use research. While research co-production can lead to demonstrable benefits such as policy or practice change, it may also have more diffuse and subtle impact on relationships, knowledge sharing, and in engendering culture shifts and research capacity-building. These relatively intangible outcomes are harder to measure and require new emphases and tools. This opinion paper uses six Canadian and United Kingdom case studies to explore the principles and practice of co-production and illustrate how it can influence interactions between research, policy and practice, and benefit diverse stakeholders. In doing so, we identify a continuum of co-production processes. We propose and illustrate the use of a new ‘social model of impact’ and framework to capture multi-layered and potentially transformative impacts of co-produced research. We make recommendations for future directions in research co-production and impact measurement.

Economic returns to medical research funding

Following the publication of the final paper in a planned series of four studies estimating the economic returns from biomedical and health research, we reflect on what we have learnt from these types of assessment.

Implementing a protocol for a research impact assessment of the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery

Background

There is growing recognition that the wider benefits of research (economic, social and health impacts) should be assessed and valued alongside traditional research performance metrics such as peer-reviewed papers. Translation of findings into policy and practice needs to accelerate and pathways to impact need to be better understood. This research protocol outlines a mixed methods study to apply the Framework to Assess the Impact from Translational health research (FAIT) to the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery (CRE-Stroke). FAIT is purpose-designed to encourage research translation and assess research impact but lacks validation.

Methods/Design

Phase 1 involves application of the FAIT-modified programme logic model to each CRE-Stroke research stream including identifying process, output and impact metrics, as well as end users of the research. A scoping review will inform potential impacts anticipated from CRE-Stroke. In Phase 2, audit and feedback on achievements against plans will track and encourage research translation. Logic models will be updated to account for changes in the research pathways over time. In Phase 3, three proven methods for measuring research impact – Payback, economic assessment and narratives – will be applied to each research stream and the data triangulated and reported in Phase 4. The feasibility of applying FAIT will also be assessed as part of Phase 3.

Discussion

Use of prospective, comprehensive research impact frameworks for large interdisciplinary programmes of research is rare. FAIT’s application to CRE-Stroke will provide opportunity for the impact of CRE-Stroke to be assessed and a range of impacts beyond standard academic achievements to be reliably reported. The feasibility of FAIT’s application will also be assessed and, if necessary, refined. The usefulness of FAIT for encouraging research translation will also be described and may prove useful for other programmes looking to implement a research impact framework.

Does citation matter? Research citation in policy documents as an indicator of research impact - an Australian obesity policy case-study

Background

Citation of research in policy documents has been suggested as an indicator of the potential longer-term impacts of research. We investigated the use of research citations in childhood obesity prevention policy documents from New South Wales (NSW), Australia, considering the feasibility and value of using research citation as a proxy measure of research impact.

Methods

We examined childhood obesity policy documents produced between 2000 and 2015, extracting childhood obesity-related references and coding these according to reference type, geographical origin and type of research. A content analysis of the policy documents examined where and how research was cited in the documents and the context of citation for individual research publications.

Results

Over a quarter (28%) of the policy documents (n = 86) were not publicly available, almost two-thirds (63%) contained references, half (47%) cited obesity-related research and over a third (41%) of those containing references used unorthodox referencing styles, making reference extraction laborious. No patterns, in terms of the types of documents more likely to cite research, were observed and the number of obesity research publications cited per document was highly variable. In total, 263 peer-reviewed and 94 non-peer-reviewed obesity research publications were cited. Research was most commonly cited to support a policy argument or choice of solution. However, it was not always possible to determine how or why individual publications were cited or whether the cited research itself had influenced the policy process. Content analysis identified circumstances where research was mentioned or considered, but not directly cited.

Conclusions

Citation of research in policy documents in this case did not always provide evidence that the cited research had influenced the policy process, only that it was accessible and relevant to the content of the policy document. Research citation across these public health policy documents varied greatly and is unlikely to be an accurate reflection of actual research use by the policy agencies involved. The links between citation and impact may be more easily drawn in specific policy areas or types of documents (e.g. clinical guidelines), where research appraisal feeds directly into policy recommendations.

Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Background

Measuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research.

Methods

Keyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework.

Results

We identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilised forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change.

Conclusions

The Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0310-4) contains supplementary material, which is available to authorized users.

Measuring research impact: a large cancer research funding programme in Australia

BACKGROUND

Measuring research impact is of critical interest to philanthropic and government funding agencies interested in ensuring that the research they fund is both scientifically excellent and has meaningful impact into health and other outcomes. The Beat Cancer Project (BCP) is a AUD $34 m cancer research funding scheme that commenced in 2011. It was initiated by an Australian charity (Cancer Council SA), and supported by the South Australian Government and the state's major universities.

METHODS

This study applied Buxton and Hanney's Payback Framework to assess research impact generated from the BCP after 3 years of funding. Data sources were an audit of peer-reviewed publications from January 2011 to September 2014 from Web of Knowledge and a self-report survey of investigators awarded BCP research funding during its first 3 years of implementation (2011-2013). Of the 104 surveys, 92 (88%) were completed.

RESULTS

The BCP performed well across all five categories of the Payback Framework. In terms of knowledge production, 1257 peer-reviewed publications were generated and the mean impact factor of publishing journals increased annually. There were many benefits to future research with 21 respondents (23%) reporting career advancement, and 110 higher degrees obtained or expected (including 84 PhDs). Overall, 52% of funded projects generated tools for future research. The funded research attracted substantial further income yielding a very high rate of leverage. For every AUD $1 that the cancer charity invested, the BCP gained an additional AUD $6.06. Five projects (5%) had informed policy and 5 (5%) informed product development, with an additional 31 (34%) and 35 (38%) projects, respectively, anticipating doing so. In terms of health and sector and broader economic benefits, 8 (9%) projects had influenced practice or behaviour of health staff and 32 (34%) would reportedly to do so in the future.

CONCLUSIONS

Research impact was a priority of charity and government funders and led to a deliberate funding strategy. Emphasising research impact while maintaining rigorous, competitive processes can achieve the joint objectives of excellence in research, yielding good research impact and a high rate of leverage for philanthropic and public investment, as indicated by these early results.

Estimating the returns to United Kingdom publicly funded musculoskeletal disease research in terms of net value of improved health outcomes

BACKGROUND

Building on an approach applied to cardiovascular and cancer research, we estimated the economic returns from United Kingdom public- and charitable-funded musculoskeletal disease (MSD) research that arise from the net value of the improved health outcomes in the United Kingdom.

METHODS

To calculate the economic returns from MSD-related research in the United Kingdom, we estimated (1) the public and charitable expenditure on MSD-related research in the United Kingdom between 1970 and 2013; (2) the net monetary benefit (NMB), derived from the health benefit in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of £25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1994 to 2013; (3) the proportion of NMB attributable to United Kingdom research; and (4) the elapsed time between research funding and health gain. The data collected from these four key elements were used to estimate the internal rate of return (IRR) from MSD-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using a one-way sensitivity analysis.

RESULTS

Expressed in 2013 prices, total expenditure on MSD-related research from 1970 to 2013 was £3.5 billion, and for the period used to estimate the rate of return, 1978-1997, was £1.4 billion. Over the period 1994-2013 the key interventions analysed produced 871,000 QALYs with a NMB of £16 billion, allowing for the net NHS costs resulting from them and valuing a QALY at £25,000. The proportion of benefit attributable to United Kingdom research was 30% and the elapsed time between funding and impact of MSD treatments was 16 years. Our best estimate of the IRR from MSD-related research was 7%, which is similar to the 9% for CVD and 10% for cancer research.

CONCLUSIONS

Our estimate of the IRR from the net health gain to public and charitable funding of MSD-related research in the United Kingdom is substantial, and justifies the research investments made between 1978 and 1997. We also demonstrated the applicability of the approach previously used in assessing the returns from cardiovascular and cancer research. Inevitably, with a study of this kind, there are a number of important assumptions and caveats that we highlight, and these can inform future research.

Encouraging translation and assessing impact of the Centre for Research Excellence in Integrated Quality Improvement: rationale and protocol for a research impact assessment

INTRODUCTION

There is growing recognition among health researchers and funders that the wider benefits of research such as economic, social and health impacts ought to be assessed and valued alongside academic outputs such as peer-reviewed papers. Research translation needs to increase and the pathways to impact ought to be more transparent. These processes are particularly pertinent to the Indigenous health sector given continued concerns that Indigenous communities are over-researched with little corresponding improvement in health outcomes. This paper describes the research protocol of a mixed methods study to apply FAIT (Framework to Assess the Impact from Translational health research) to the Centre for Research Excellence in Integrated Quality Improvement (CRE-IQI). FAIT will be applied to five selected CRE-IQI Flagship projects to encourage research translation and assess the wider impact of that research.

METHODS AND ANALYSIS

Phase I will develop a modified programme logic model for each Flagship project including identifying process, output and impact metrics so progress can be monitored. A scoping review will inform potential benefits. In phase II, programme logic models will be updated to account for changes in the research pathways over time. Audit and feedback will be used to encourage research translation and collect evidence of achievement of any process, output and interim impacts. In phase III, three proven methodologies for measuring research impact-Payback, economic assessment and narratives-will be applied. Data on the application of FAIT will be collected and analysed to inform and improve FAIT's performance.

ETHICS AND DISSEMINATION

This study is funded by a nationally competitive grant (ID 1078927) from the Australian National Health and Medical Research Council. Ethics approval was obtained from the University of Newcastle's Human Research Ethics Committee (ID: H-2017-0026). The results from the study will be presented in several peer-reviewed publications, through conference presentations and via social media.

Transferring research from a university to the United Kingdom National Health Service: the implications for impact

The aim of this article is to inform readers of the author’s reflections on the experience of transferring university-based research into the commercial sector, and of the processes and strategies employed when preparing for impact in so doing. Concepts for the transfer are illustrated by the author’s reflection on aspects that arose during the birthing and subsequent start-up of a university spin-off, Pathways2Wellbeing, a form of reflection-on-action. This is the vehicle for the adaption required to transfer research into the delivery of a specialised clinic in the United Kingdom National Health Service for people with medically unexplained, persistent, bodily symptoms such as fibromyalgia, chronic fatigue and chronic pain. It is hoped that the article will provide readers with an insight into how knowledge transfer can take place through engagement with stakeholders to create an exchange of knowledges to result in impact on health service policy for service users, despite the challenges, and the enablers that facilitated this process. The reflections on the process of knowledge transfer and the implications for impact are underpinned by relevant theory.

Achieving Research Impact Through Co-creation in Community-Based Health Services: Literature Review and Case Study

Policy Points:

Co‐creation—collaborative knowledge generation by academics working alongside other stakeholders—is an increasingly popular approach to aligning research and service development.

It has potential for “moving beyond the ivory towers” to deliver significant societal impact via dynamic, locally adaptive community‐academic partnerships.

Principles of successful co‐creation include a systems perspective, a creative approach to research focused on improving human experience, and careful attention to governance and process.

If these principles are not followed, co‐creation efforts may fail.

Context

Co‐creation—collaborative knowledge generation by academics working alongside other stakeholders—reflects a “Mode 2” relationship (knowledge production rather than knowledge translation) between universities and society. Co‐creation is widely believed to increase research impact.

Methods

We undertook a narrative review of different models of co‐creation relevant to community‐based health services. We contrasted their diverse disciplinary roots and highlighted their common philosophical assumptions, principles of success, and explanations for failures. We applied these to an empirical case study of a community‐based research‐service partnership led by the Centre of Research Excellence in Quality and Safety in Integrated Primary‐Secondary Care at the University of Queensland, Australia.

Findings

Co‐creation emerged independently in several fields, including business studies (“value co‐creation”), design science (“experience‐based co‐design”), computer science (“technology co‐design”), and community development (“participatory research”). These diverse models share some common features, which were also evident in the case study. Key success principles included (1) a systems perspective (assuming emergence, local adaptation, and nonlinearity); (2) the framing of research as a creative enterprise with human experience at its core; and (3) an emphasis on process (the framing of the program, the nature of relationships, and governance and facilitation arrangements, especially the style of leadership and how conflict is managed). In both the literature review and the case study, co‐creation “failures” could often be tracked back to abandoning (or never adopting) these principles. All co‐creation models made strong claims for significant and sustainable societal impacts as a result of the adaptive and developmental research process; these were illustrated in the case study.

Conclusions

Co‐creation models have high potential for societal impact but depend critically on key success principles. To capture the nonlinear chains of causation in the co‐creation pathway, impact metrics must reflect the dynamic nature and complex interdependencies of health research systems and address processes as well as outcomes.

The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews

BACKGROUND

We sought to analyse the impacts found, and the methods used, in a series of assessments of programmes and portfolios of health research consisting of multiple projects.

METHODS

We analysed a sample of 36 impact studies of multi-project research programmes, selected from a wider sample of impact studies included in two narrative systematic reviews published in 2007 and 2016. We included impact studies in which the individual projects in a programme had been assessed for wider impact, especially on policy or practice, and where findings had been described in such a way that allowed them to be collated and compared.

RESULTS

Included programmes were highly diverse in terms of location (11 different countries plus two multi-country ones), number of component projects (8 to 178), nature of the programme, research field, mode of funding, time between completion and impact assessment, methods used to assess impact, and level of impact identified. Thirty-one studies reported on policy impact, 17 on clinician behaviour or informing clinical practice, three on a combined category such as policy and clinician impact, and 12 on wider elements of impact (health gain, patient benefit, improved care or other benefits to the healthcare system). In those multi-programme projects that assessed the respective categories, the percentage of projects that reported some impact was policy 35% (range 5-100%), practice 32% (10-69%), combined category 64% (60-67%), and health gain/health services 27% (6-48%). Variations in levels of impact achieved partly reflected differences in the types of programme, levels of collaboration with users, and methods and timing of impact assessment. Most commonly, principal investigators were surveyed; some studies involved desk research and some interviews with investigators and/or stakeholders. Most studies used a conceptual framework such as the Payback Framework. One study attempted to assess the monetary value of a research programme's health gain.

CONCLUSION

The widespread impact reported for some multi-project programmes, including needs-led and collaborative ones, could potentially be used to promote further research funding. Moves towards greater standardisation of assessment methods could address existing inconsistencies and better inform strategic decisions about research investment; however, unresolved issues about such moves remain.

Research impact: a narrative review

Impact occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base. Its mechanisms are complex and reflect the multiple ways in which knowledge is generated and utilised. Much progress has been made in measuring both the outcomes of research and the processes and activities through which these are achieved, though the measurement of impact is not without its critics. We review the strengths and limitations of six established approaches (Payback, Research Impact Framework, Canadian Academy of Health Sciences, monetisation, societal impact assessment, UK Research Excellence Framework) plus recently developed and largely untested ones (including metrics and electronic databases). We conclude that (1) different approaches to impact assessment are appropriate in different circumstances; (2) the most robust and sophisticated approaches are labour-intensive and not always feasible or affordable; (3) whilst most metrics tend to capture direct and proximate impacts, more indirect and diffuse elements of the research-impact link can and should be measured; and (4) research on research impact is a rapidly developing field with new methodologies on the horizon.