1
|
Escudero-Vilaplana V, Guisado-Gil AB, Santos-Ramos B, Herranz A. Development and consensus of a dashboard model to evaluate research activity in Spanish Hospital Pharmacy Services. FARMACIA HOSPITALARIA 2024; 48 Suppl 1:S28-S34. [PMID: 39097364 DOI: 10.1016/j.farma.2023.12.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 11/30/2023] [Accepted: 12/04/2023] [Indexed: 08/05/2024] Open
Abstract
OBJECTIVE To develop by consensus a dashboard model to standardize and promote the evaluation of research activity in Spanish Hospital Pharmacy Services. METHODS The study was carried out in 5 phases following the modified Delphi methodology: constitution of the coordinating group, elaboration of a list of scenarios, selection of participating centers, evaluation of the list of scenarios, and analysis of the results. The coordinating group designed a questionnaire with 114 questions. General research questions and different scenarios (indicators) were included to form the dashboard. The Hospital Pharmacy Services with the highest number of publications were identified to participate in the Delphi consultation. Two rounds of consultations were conducted in which the "Need" and/or "Feasibility" of their measurement was evaluated for each of the scenarios, using a numerical scale from 1 (lowest score) to 9 (highest score). RESULTS Sixteen Hospital Pharmacy Services, belonging to 8 different autonomous communities, participated in the Delphi consultation. A total of 100% of them responded to all the questions in the 2 rounds of consultations. It was considered that the Hospital Pharmacy Services should have a research dashboard (need = 100%) with a basic structure and a common minimum set of data for all them (need = 87.5%). The consensus was reached on distinguishing research projects led by the Hospital Pharmacy Services from those led by other groups in which the Hospital Pharmacy Services collaborate (need = 87.5%), and a definition was approved on the leadership of these projects according to whether they are single-center or multicenter. A consensus was reached on 40 indicators to form the dashboard, which evaluates publications (13 indicators), human resources (12 indicators), research projects (9 indicators), doctoral theses (4 indicators), and patents and intellectual property registrations (2 indicators). CONCLUSIONS This is the first consensus dashboard developed to evaluate the research activity of the Hospital Pharmacy Services, which will help to analyze the productivity and impact of research systematically and continuously. In addition, it will allow comparison between them and will help to establish synergies and identify trends, patterns, and challenges.
Collapse
Affiliation(s)
- Vicente Escudero-Vilaplana
- Servicio de Farmacia, Hospital General Universitario Gregorio Marañón, Madrid, España; Instituto de Investigación Sanitaria Gregorio Marañón, Madrid, España
| | | | | | - Ana Herranz
- Servicio de Farmacia, Hospital General Universitario Gregorio Marañón, Madrid, España; Instituto de Investigación Sanitaria Gregorio Marañón, Madrid, España
| |
Collapse
|
2
|
Escudero-Vilaplana V, Guisado-Gil AB, Santos-Ramos B, Herranz A. [Translated article] Development and consensus of a dashboard model to evaluate research activity in Spanish Hospital Pharmacy Services. FARMACIA HOSPITALARIA 2024; 48 Suppl 1:TS28-TS34. [PMID: 39097373 DOI: 10.1016/j.farma.2024.03.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 11/30/2023] [Accepted: 12/04/2023] [Indexed: 08/05/2024] Open
Abstract
OBJECTIVE To develop by consensus a dashboard model to standardise and promote the evaluation of research activity in Spanish Hospital Pharmacy Services. METHODS The study was carried out in 5 phases following the modified Delphi methodology: constitution of the coordinating group, elaboration of a list of scenarios, selection of participating centres, evaluation of the list of scenarios, and analysis of the results. The coordinating group designed a questionnaire with 114 questions. General research questions and different scenarios (indicators) were included to form the dashboard. The Hospital Pharmacy Services with the highest number of publications were identified to participate in the Delphi consultation. Two rounds of consultations were conducted in which the "Need" and/or "Feasibility" of their measurement was evaluated for each of the scenarios, using a numerical scale from 1 (lowest score) to 9 (highest score). RESULTS Sixteen Hospital Pharmacy Services, belonging to 8 different Autonomous Communities, participated in the Delphi consultation. A total of 100% of them responded to all the questions in the 2 rounds of consultations. It was considered that the Hospital Pharmacy Services should have a research dashboard (Need=100%) with a basic structure and a common minimum set of data for all them (Need=87.5%). The consensus was reached on distinguishing research projects led by the Hospital Pharmacy Services from those led by other groups in which the Hospital Pharmacy Services collaborate (Need=87.5%), and a definition was approved on the leadership of these projects according to whether they are single-centre or multicentre. A consensus was reached on 40 indicators to form the dashboard, which evaluates publications (13 indicators), human resources (12 indicators), research projects (9 indicators), doctoral theses (4 indicators), and patents and intellectual property registrations (2 indicators). CONCLUSIONS This is the first consensus dashboard developed to evaluate the research activity of the Hospital Pharmacy Services, which will help to analyse the productivity and impact of research systematically and continuously. In addition, it will allow comparison between them and will help to establish synergies and identify trends, patterns, and challenges.
Collapse
Affiliation(s)
- Vicente Escudero-Vilaplana
- Servicio de Farmacia, Hospital General Universitario Gregorio Marañón, Madrid, Spain; Instituto de Investigación Sanitaria Gregorio Marañón, Madrid, Spain
| | | | | | - Ana Herranz
- Servicio de Farmacia, Hospital General Universitario Gregorio Marañón, Madrid, Spain; Instituto de Investigación Sanitaria Gregorio Marañón, Madrid, Spain
| |
Collapse
|
3
|
Knight R, Luke B, Meuter R, Peters K, Chell K. A multidimensional assessment of payback: Evaluating the impact of Breast Cancer Trials. Asia Pac J Clin Oncol 2024; 20:372-378. [PMID: 38174644 DOI: 10.1111/ajco.14043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2023] [Revised: 11/29/2023] [Accepted: 12/17/2023] [Indexed: 01/05/2024]
Abstract
AIM This study evaluates the impact created by clinical trial investment and research undertaken by Breast Cancer Trials (BCT). METHODS The approach involved using the Payback Framework, and included a review of BCT archival information, public health data, a questionnaire sent to BCT members, individual interviews with key informants, a focus group with members of the organization's Consumer Advisory Panel, and case studies of select BCT trials. The evaluation assessed the evidence against the Payback Framework criteria: informing policy and product development, knowledge production, the research system, health and health sector benefits, and broader economic benefits. RESULTS Analysis using the Payback Framework revealed impact was created in each category and a range of positive outcomes were identified for various stakeholder groups. BCT is maximizing the impact of its research and contributing to a global pool of scientific knowledge by collaborating with over 100 institutions and 820 researchers, yet its benefits go beyond research contributions. Findings highlight strong financial returns from BCT's research program and that BCT has played an important role in enhancing the public perception of clinical trials by promoting participation in trials, educating and empowering participants, and communicating trial outcomes. CONCLUSION BCT's clinical trials have had a significant impact on the prevention, detection, treatment, and management of breast cancer. Measuring and reporting impact can be resource intensive but support BCT in remaining accountable to all those invested in the organization and in breast cancer clinical trials, evidencing the multiple dimensions of payback resulting from the organization's research.
Collapse
Affiliation(s)
- Ruth Knight
- Faculty of Business and Law, The Australian Centre for Philanthropy and Nonprofit Studies, Queensland University of Technology, Brisbane, Australia
| | - Belinda Luke
- Faculty of Business and Law, School of Accountancy, Queensland University of Technology, Brisbane, Australia
| | - Renata Meuter
- Faculty of Health, School of Psychology and Counselling, Queensland University of Technology, Brisbane, Australia
| | - Kaitlin Peters
- Faculty of Business and Law, The Australian Centre for Philanthropy and Nonprofit Studies, Queensland University of Technology, Brisbane, Australia
| | - Kathleen Chell
- Faculty of Business and Law, The Australian Centre for Philanthropy and Nonprofit Studies, Queensland University of Technology, Brisbane, Australia
| |
Collapse
|
4
|
Mrklas KJ, Boyd JM, Shergill S, Merali S, Khan M, Moser C, Nowell L, Goertzen A, Swain L, Pfadenhauer LM, Sibley KM, Vis-Dunbar M, Hill MD, Raffin-Bouchal S, Tonelli M, Graham ID. A scoping review of the globally available tools for assessing health research partnership outcomes and impacts. Health Res Policy Syst 2023; 21:139. [PMID: 38129871 PMCID: PMC10740226 DOI: 10.1186/s12961-023-00958-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2022] [Accepted: 01/03/2023] [Indexed: 12/23/2023] Open
Abstract
BACKGROUND Health research partnership approaches have grown in popularity over the past decade, but the systematic evaluation of their outcomes and impacts has not kept equal pace. Identifying partnership assessment tools and key partnership characteristics is needed to advance partnerships, partnership measurement, and the assessment of their outcomes and impacts through systematic study. OBJECTIVE To locate and identify globally available tools for assessing the outcomes and impacts of health research partnerships. METHODS We searched four electronic databases (Ovid MEDLINE, Embase, CINAHL + , PsychINFO) with an a priori strategy from inception to June 2021, without limits. We screened studies independently and in duplicate, keeping only those involving a health research partnership and the development, use and/or assessment of tools to evaluate partnership outcomes and impacts. Reviewer disagreements were resolved by consensus. Study, tool and partnership characteristics, and emerging research questions, gaps and key recommendations were synthesized using descriptive statistics and thematic analysis. RESULTS We screened 36 027 de-duplicated citations, reviewed 2784 papers in full text, and kept 166 studies and three companion reports. Most studies originated in North America and were published in English after 2015. Most of the 205 tools we identified were questionnaires and surveys targeting researchers, patients and public/community members. While tools were comprehensive and usable, most were designed for single use and lacked validity or reliability evidence. Challenges associated with the interchange and definition of terms (i.e., outcomes, impacts, tool type) were common and may obscure partnership measurement and comparison. Very few of the tools identified in this study overlapped with tools identified by other, similar reviews. Partnership tool development, refinement and evaluation, including tool measurement and optimization, are key areas for future tools-related research. CONCLUSION This large scoping review identified numerous, single-use tools that require further development and testing to improve their psychometric and scientific qualities. The review also confirmed that the health partnership research domain and its measurement tools are still nascent and actively evolving. Dedicated efforts and resources are required to better understand health research partnerships, partnership optimization and partnership measurement and evaluation using valid, reliable and practical tools that meet partners' needs.
Collapse
Affiliation(s)
- Kelly J Mrklas
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3D10-3280 Hospital Drive NW, Calgary, AB, T2N 4Z6, Canada.
- Strategic Clinical Networks™, Provincial Clinical Excellence, Alberta Health Services, Calgary, AB, Canada.
| | - Jamie M Boyd
- Knowledge Translation Program, St Michael's Hospital, Li Ka Shing Knowledge Institute, Unity Health Toronto, Toronto, ON, Canada
| | - Sumair Shergill
- Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | - Sera Merali
- Faculty of Kinesiology, University of Calgary, Calgary, AB, Canada
| | - Masood Khan
- Department of Community Health Sciences, University of Manitoba, Winnipeg, MB, Canada
| | - Cheryl Moser
- Department of Community Health Sciences, University of Manitoba, Winnipeg, MB, Canada
| | - Lorelli Nowell
- Faculty of Nursing, University of Calgary, Calgary, AB, Canada
| | - Amelia Goertzen
- Faculty of Science, University of Alberta, Edmonton, AB, Canada
| | - Liam Swain
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3D10-3280 Hospital Drive NW, Calgary, AB, T2N 4Z6, Canada
| | - Lisa M Pfadenhauer
- Institute for Medical Information Processing, and Epidemiology-IBE, Ludwig-Maximilians Universität Munich, Munich, Germany
- Pettenkofer School of Public Health, Munich, Germany
| | - Kathryn M Sibley
- Department of Community Health Sciences, University of Manitoba, Winnipeg, MB, Canada
- George & Fay Yee Centre for Healthcare Innovation, University of Manitoba, Winnipeg, MB, Canada
| | | | - Michael D Hill
- Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3D10-3280 Hospital Drive NW, Calgary, AB, T2N 4Z6, Canada
- Departments of Clinical Neurosciences, Medicine and Radiology, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
| | | | - Marcello Tonelli
- Department of Medicine, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada
- Office of the Vice-President (Research), University of Calgary, Calgary, AB, Canada
| | - Ian D Graham
- Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada
- Schools of Epidemiology and Public Health and Nursing, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
5
|
Trowman R, Migliore A, Ollendorf DA. The value and impact of health technology assessment: discussions and recommendations from the 2023 Health Technology Assessment International Global Policy Forum. Int J Technol Assess Health Care 2023; 39:e75. [PMID: 38130164 DOI: 10.1017/s0266462323002763] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2023]
Abstract
Health technology assessment (HTA) programs inform decision making about the value and reimbursement of new and existing health technologies; however, they are under increasing pressure to demonstrate that they are a cost-effective use of finite healthcare resources themselves. The 2023 HTAi Global Policy Forum (GPF) discussed the value and impact of HTA, including how it is assessed and communicated, and how it could be enhanced in the future. This article summarizes the discussions held at the 2023 HTAi GPF, where the challenges and opportunities related to the value and impact of HTA were debated. Core themes and recommendations identified that defining the purpose of value and impact assessment is an essential first step prior to undertaking it, and that it can be done through the use and expansion of existing tools. Further work around aligning HTA programs with underlying societal values is needed to ensure the long-term value and impact of HTA. HTA could also have a role in assessing the efficiency of the wider health system by applying HTA methods or concepts to broader budgetary allocations and organizational aspects of health care. Stakeholders (particularly patients, industry, and clinicians but also payers, wider society, and the media) should ideally be actively engaged when undertaking the value and impact assessment of HTA. More concerted efforts in communicating the role and remit of HTA bodies would also help stakeholders to better understand the value and impact of HTA, which in turn could improve the implementation of HTA recommendations and application to future actions in the lifecycle of technologies.
Collapse
Affiliation(s)
| | - Antonio Migliore
- Health Technology Assessment International (HTAi), Edmonton, AB, Canada
| | - Daniel A Ollendorf
- Tufts Medical Center, Center for the Evaluation of Value and Risk in Health, Boston, MA, USA
| |
Collapse
|
6
|
Eckhardt H, Felgner S, Dreger M, Fuchs S, Ermann H, Rödiger H, Rombey T, Busse R, Henschke C, Panteli D. Utilization of innovative medical technologies in German inpatient care: does evidence matter? Health Res Policy Syst 2023; 21:100. [PMID: 37784100 PMCID: PMC10546629 DOI: 10.1186/s12961-023-01047-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2021] [Accepted: 08/26/2023] [Indexed: 10/04/2023] Open
Abstract
BACKGROUND The reimbursement of new technologies in inpatient care is not always linked to a requirement for evidence-based evaluation of patient benefit. In Germany, every new technology approved for market was until recently eligible for reimbursement in inpatient care unless explicitly excluded. The aim of this work was (1) to investigate the type of evidence that was available at the time of introduction of 25 innovative technologies and how this evidence evolved over time, and (2) to explore the relationship between clinical evidence and utilization for these technologies in German inpatient care. METHODS This study combined different methods. A systematic search for evidence published between 2003 and 2017 was conducted in four bibliographic databases, clinical trial registries, resources for clinical guidelines, and health technology assessment-databases. Information was also collected on funding mechanisms and safety notices. Utilization was measured by hospital procedures captured in claims data. The body of evidence, funding and safety notices per technology were analyzed descriptively. The relationship between utilization and evidence was explored empirically using a multilevel regression analysis. RESULTS The number of included publications per technology ranges from two to 498. For all technologies, non-comparative studies form the bulk of the evidence. The number of randomized controlled clinical trials per technology ranges from zero to 19. Some technologies were utilized for several years without an adequate evidence base. A relationship between evidence and utilization could be shown for several but not all technologies. CONCLUSIONS This study reveals a mixed picture regarding the evidence available for new technologies, and the relationship between the development of evidence and the use of technologies over time. Although the influence of funding and safety notices requires further investigation, these results re-emphasize the need for strengthening market approval standards and HTA pathways as well as approaches such as coverage with evidence development.
Collapse
Affiliation(s)
- Helene Eckhardt
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany.
| | - Susanne Felgner
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
- Berlin Centre for Health Economics Research (BerlinHECOR), Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| | - Marie Dreger
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
- Berlin Centre for Health Economics Research (BerlinHECOR), Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| | - Sabine Fuchs
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| | - Hanna Ermann
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| | - Hendrikje Rödiger
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| | - Tanja Rombey
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| | - Reinhard Busse
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
- Berlin Centre for Health Economics Research (BerlinHECOR), Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| | - Cornelia Henschke
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
- Berlin Centre for Health Economics Research (BerlinHECOR), Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| | - Dimitra Panteli
- Department of Health Care Management, Technische Universität Berlin, Straße Des 17. Juni 135, 10623, Berlin, Germany
| |
Collapse
|
7
|
McClean A, Huo B, Kwan JY, Long J, Walshaw J, Mesri M, Francis N, Arulampalam TH, Chetter I, Yiasemidou M. The impact of the European Association of Endoscopic Surgery research grant scheme-a mixed qualitative quantitative methodology study protocol. Front Surg 2023; 10:1197103. [PMID: 37405059 PMCID: PMC10315820 DOI: 10.3389/fsurg.2023.1197103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2023] [Accepted: 05/31/2023] [Indexed: 07/06/2023] Open
Abstract
Background The European Association of Endoscopic Surgery (EAES) is a surgical society who promotes the development and expansion of minimally invasive surgery to surgeons and surgical trainees. It does so through its activities in education, training, and research. The EAES research committee aims to promote the highest quality clinical research in endoscopic and minimally invasive surgery. They have provided grant funding since 2009 in education, surgery, and basic science. Despite the success and longevity of the scheme, the academic and non-academic impact of the research funding scheme has not been evaluated. Aims The primary aim of this project is to assess the short, long term academic and real world impact of the EAES funding scheme. The secondary aims are to identify barriers and facilitators for achieving good impact. Methods This will be a mixed qualitative and quantitative study. Semi-structured interviews will be performed with previous grant recipients. The questions for the interviews will be selected after a consensus is achieved amongst the members of the steering committee of this project. The responses will be transcribed and thematic analysis will be applied. The results of the thematic analysis will be used to populate a questionnaire which will be disseminated to grant recipients. This study is kindly funded by the EAES. Discussion The first question this project is expected to answer is whether the EAES research funding scheme had a significant positive impact on research output, career progression but also non-academic output such as change in clinical guidelines, healthcare quality and cost-effectiveness improvement. This project however is also expected to identify facilitators and barriers to successful completion of projects and to achieving high impact. This will inform EAES and the rest of the surgical and academic communities as to how clinicians would like to be supported when conducting research. There should also be a positive and decisive change towards removing factors that hinder the timely and successful completion of projects.
Collapse
Affiliation(s)
- Adam McClean
- Clinical Fellow General Surgery, Bradford Teaching Hospitals, Bradford, United Kingdom
| | - Bright Huo
- Faculty of Medicine, Dalhousie University, Halifax, NS, Canada
| | - Jing Yi Kwan
- NIHR Academic Clinical Fellow, Vascular Surgery, Leeds Teaching Hospitals, Leeds, United Kingdom
| | - Judith Long
- Research Manager, Academic Vascular Surgical Unit, Hull University Teaching Hospitals NHS Trust, Hull, United Kingdom
| | - Josephine Walshaw
- Department of Health Research, University of York, York, United kingdom
| | - Mina Mesri
- Department of Medicine, Hull York MedicalSchool, York, United kingdom
| | - Nader Francis
- Consultant Colorectal Surgeon, Department of Surgery, Yeovil District Hospital NHS Foundation Trust, Honorary Professor of Surgery UCL, Yeovil, United Kingdom
| | - Tan H. Arulampalam
- Consultant Colorectal Surgeon, ICENI Centre, Colchester Hospital University NHS Foundation Trust, Colchester, Essex, United Kingdom
| | - Ian Chetter
- Professor of Surgery, Hull University Teaching Hospital, Hull, United Kingdom
| | - Marina Yiasemidou
- Department of Medicine, NIHR Academic Clinical Lecturer in General Surgery, Hull York Medical School, Hull, United Kingdom
| |
Collapse
|
8
|
Ramanathan S, Lynch E, Bernhardt J, Nilsson M, Cadilhac DA, Carey L, Middleton S, Chamberlain J, Walker FR, Reeves P, Searles A. Impact assessment of the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery. Health Res Policy Syst 2023; 21:30. [PMID: 37127659 PMCID: PMC10152619 DOI: 10.1186/s12961-023-00974-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2022] [Accepted: 03/08/2023] [Indexed: 05/03/2023] Open
Abstract
BACKGROUND Research impact is an emerging measure of research achievement alongside traditional academic outputs such as publications. We present the results of applying the Framework to Assess the Impact from Translational health research (FAIT) to the Centre for Research Excellence (CRE) in Stroke Rehabilitation and Brain Recovery (CRE-Stroke, 2014-2019) and report on the feasibility and lessons from the application of FAIT to a CRE rather than a discrete research project. METHODS Data were gathered via online surveys, in-depth interviews, document analysis and review of relevant websites/databases to report on the three major FAIT methods: the modified Payback Framework, an assessment of costs against monetized consequences, and a narrative account of the impact generated from CRE-Stroke activities. FAIT was applied during the last 4 years of CRE-Stroke operation. RESULTS With an economic investment of AU$ 3.9 million over 5 years, CRE-Stroke delivered a return on investment that included AU$ 18.8 million in leveraged grants, fellowships and consultancies. Collectively, CRE-Stroke members produced 354 publications that were accessed 470,000 times and cited over 7220 times. CRE-Stroke supported 26 PhDs, 39 postdocs and seven novice clinician researchers. There were 59 capacity-building events benefiting 744 individuals including policy-makers and consumers. CRE-Stroke created research infrastructure (including a research register of stroke survivors and a brain biobank), and its global leadership produced international consensus recommendations to influence the stroke research landscape worldwide. Members contributed to the Australian Living Stroke Guidelines: four researchers' outputs were directly referenced. Based only on the consequences that could be monetized, CRE-Stroke returned AU$ 4.82 for every dollar invested in the CRE. CONCLUSION This case example in the developing field of impact assessment illustrates how researchers can use evidence to demonstrate and report the impact of and returns on research investment. The prospective application of FAIT by a dedicated research impact team demonstrated impact in broad categories of knowledge-gain, capacity-building, new infrastructure, input to policy and economic benefits. The methods can be used by other research teams to provide comprehensive evidence to governments and other research funders about what has been generated from their research investment but requires dedicated resources to complete.
Collapse
Affiliation(s)
- Shanthi Ramanathan
- Health Research Economics, Hunter Medical Research Institute, Locked Bag 100, New Lambton Heights, NSW, 2305, Australia.
- School of Medicine and Public Health, College of Health, Medicine and Wellbeing, University of Newcastle, Callaghan, NSW, Australia.
| | - Elizabeth Lynch
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Parkville, VIC, Australia
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
- College of Nursing and Health Sciences, Flinders University, Bedford Park, SA, Australia
| | - Julie Bernhardt
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Parkville, VIC, Australia
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
| | - Michael Nilsson
- Health Research Economics, Hunter Medical Research Institute, Locked Bag 100, New Lambton Heights, NSW, 2305, Australia
- School of Medicine and Public Health, College of Health, Medicine and Wellbeing, University of Newcastle, Callaghan, NSW, Australia
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
| | - Dominique A Cadilhac
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Parkville, VIC, Australia
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
- Stroke and Ageing Research, School of Clinical Sciences at Monash Health, Monash University, Clayton, VIC, Australia
| | - Leeanne Carey
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Parkville, VIC, Australia
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
- Occupational Therapy, School of Allied Health, Human Services and Sport, LaTrobe University, Bundoora, VIC, Australia
| | - Sandy Middleton
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
- Nursing Research Institute, St Vincent's Health Australia (Sydney) and Australian Catholic University, Darlinghurst, Sydney, Australia
| | - Jan Chamberlain
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Parkville, VIC, Australia
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
| | - Frederick Rohan Walker
- School of Medicine and Public Health, College of Health, Medicine and Wellbeing, University of Newcastle, Callaghan, NSW, Australia
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
| | - Penny Reeves
- Health Research Economics, Hunter Medical Research Institute, Locked Bag 100, New Lambton Heights, NSW, 2305, Australia
- School of Medicine and Public Health, College of Health, Medicine and Wellbeing, University of Newcastle, Callaghan, NSW, Australia
| | - Andrew Searles
- Health Research Economics, Hunter Medical Research Institute, Locked Bag 100, New Lambton Heights, NSW, 2305, Australia
- School of Medicine and Public Health, College of Health, Medicine and Wellbeing, University of Newcastle, Callaghan, NSW, Australia
| |
Collapse
|
9
|
Bowen JM, Ouimet M, Lawarée J, Bielecki J, Rhéaume A, Greenberg C, Rac VE. Describing the state of a research network: A mixed methods approach to network evaluation. RESEARCH EVALUATION 2023; 32:188-199. [PMID: 37799115 PMCID: PMC10550251 DOI: 10.1093/reseval/rvac034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2021] [Revised: 08/03/2021] [Accepted: 09/05/2021] [Indexed: 10/07/2023]
Abstract
Diabetes Action Canada Strategy for Patient-Oriented Research (SPOR) Network in Chronic Disease was formed in 2016 and is funded primarily through the Canadian Institutes of Health Research (CIHR). We propose a novel mixed-methods approach to a network evaluation integrating the State of Network Evaluation framework and the Canadian Academy of Health Sciences (CAHS) preferred framework and indicators. We measure key network themes of connectivity, health and results, and impact and return on investment associated with health research networks. Our methods consist of a longitudinal cross-sectional network survey of members and social network analysis to examine Network Connectivity and assess the frequency of interactions, the topics discussed during them, and how networking effectively facilitates interactions and collaboration among members. Network Health will be evaluated through semistructured interviews, a membership survey inquiring about satisfaction and experience with the Network, and a review of documentary sources related to funding and infrastructure to evaluate Network Sustainability. Finally, we will examine Network Results and Impact using the CAHS preferred framework and indicators to measure returns on investment in health research across the five domains of the CAHS framework, which include: advancing knowledge, capacity building, informing decision making, health impact, and economic and social impact. Indicators will be assessed with various methods, including bibliometric analyses, review of relevant documentary sources (annual reports), member activities informing health and research policy, and Patient Partner involvement. The Network Evaluation will provide members and stakeholders with information for planning, improvements, and funding future Network endeavors.
Collapse
Affiliation(s)
- James M Bowen
- Program for Health System and Technology Evaluation, Ted Rogers Centre for Heart Research at Peter Munk Cardiac Centre, Toronto General Hospital Research Institute (TGHRI), Toronto, ON M5G 2C4, Canada
- THETA Collaborative, University Health Network, 10th Floor Eaton North, 200 Elizabeth Street, Toronto, ON M5G 2C4, Canada
- Health Technology Assessment and Network Analytics, Diabetes Action Canada, Toronto, ON M5G 2C4,Canada
- Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON M5T 3M6,Canada
- Department of Health Research Methods, Evidence and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
| | - Mathieu Ouimet
- Health Technology Assessment and Network Analytics, Diabetes Action Canada, Toronto, ON M5G 2C4,Canada
- Département de Science Politique, Faculté des Science Social, Université Laval, Québec, QC G1V 0A6, Canada
| | - Justin Lawarée
- Département de Science Politique, Faculté des Science Social, Université Laval, Québec, QC G1V 0A6, Canada
| | - Joanna Bielecki
- Program for Health System and Technology Evaluation, Ted Rogers Centre for Heart Research at Peter Munk Cardiac Centre, Toronto General Hospital Research Institute (TGHRI), Toronto, ON M5G 2C4, Canada
- THETA Collaborative, University Health Network, 10th Floor Eaton North, 200 Elizabeth Street, Toronto, ON M5G 2C4, Canada
| | - Ashley Rhéaume
- Département de Science Politique, Faculté des Science Social, Université Laval, Québec, QC G1V 0A6, Canada
| | - Caylee Greenberg
- Program for Health System and Technology Evaluation, Ted Rogers Centre for Heart Research at Peter Munk Cardiac Centre, Toronto General Hospital Research Institute (TGHRI), Toronto, ON M5G 2C4, Canada
- THETA Collaborative, University Health Network, 10th Floor Eaton North, 200 Elizabeth Street, Toronto, ON M5G 2C4, Canada
| | - Valeria E Rac
- Program for Health System and Technology Evaluation, Ted Rogers Centre for Heart Research at Peter Munk Cardiac Centre, Toronto General Hospital Research Institute (TGHRI), Toronto, ON M5G 2C4, Canada
- THETA Collaborative, University Health Network, 10th Floor Eaton North, 200 Elizabeth Street, Toronto, ON M5G 2C4, Canada
- Health Technology Assessment and Network Analytics, Diabetes Action Canada, Toronto, ON M5G 2C4,Canada
- Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON M5T 3M6,Canada
| |
Collapse
|
10
|
French C, Dowrick A, Fudge N, Pinnock H, Taylor SJC. What do we want to get out of this? a critical interpretive synthesis of the value of process evaluations, with a practical planning framework. BMC Med Res Methodol 2022; 22:302. [PMID: 36434520 PMCID: PMC9700891 DOI: 10.1186/s12874-022-01767-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Accepted: 10/21/2022] [Indexed: 11/26/2022] Open
Abstract
BACKGROUND Process evaluations aim to understand how complex interventions bring about outcomes by examining intervention mechanisms, implementation, and context. While much attention has been paid to the methodology of process evaluations in health research, the value of process evaluations has received less critical attention. We aimed to unpack how value is conceptualised in process evaluations by identifying and critically analysing 1) how process evaluations may create value and 2) what kind of value they may create. METHODS We systematically searched for and identified published literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues. We conducted a critical interpretive synthesis and developed a practical planning framework. RESULTS We identified and included 147 literature items. From these we determined three ways in which process evaluations may create value or negative consequences: 1) through the socio-technical processes of 'doing' the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge. We identified 15 value themes. We also found that value varies according to the characteristics of individual process evaluations, and is subjective and context dependent. CONCLUSION The concept of value in process evaluations is complex and multi-faceted. Stakeholders in different contexts may have very different expectations of process evaluations and the value that can and should be obtained from them. We propose a planning framework to support an open and transparent process to plan and create value from process evaluations and negotiate trade-offs. This will support the development of joint solutions and, ultimately, generate more value from process evaluations to all.
Collapse
Affiliation(s)
- Caroline French
- grid.4868.20000 0001 2171 1133Wolfson Institute of Population Health, Barts and The London School of Medicine and Dentistry, Queen Mary University of London, Yvonne Carter Building, 58 Turner Street, London, E1 2AB UK
| | - Anna Dowrick
- grid.4991.50000 0004 1936 8948Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Primary Care Building, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GC UK
| | - Nina Fudge
- grid.4868.20000 0001 2171 1133Wolfson Institute of Population Health, Barts and The London School of Medicine and Dentistry, Queen Mary University of London, Yvonne Carter Building, 58 Turner Street, London, E1 2AB UK
| | - Hilary Pinnock
- grid.4305.20000 0004 1936 7988Usher Institute, The University of Edinburgh, Doorway 3, Medical School, Teviot Place, Edinburgh, EH8 9AG UK
| | - Stephanie J. C. Taylor
- grid.4868.20000 0001 2171 1133Wolfson Institute of Population Health, Barts and The London School of Medicine and Dentistry, Queen Mary University of London, Yvonne Carter Building, 58 Turner Street, London, E1 2AB UK
| |
Collapse
|
11
|
Researchers’ perspective of real-world impact from UK public health research: A qualitative study. PLoS One 2022; 17:e0268675. [PMID: 35759461 PMCID: PMC9236254 DOI: 10.1371/journal.pone.0268675] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2021] [Accepted: 05/04/2022] [Indexed: 11/19/2022] Open
Abstract
Research funded by the National Institute for Health Research Public Health Research Programme is being undertaken in a complex system which brings opportunities and challenges for researchers to maximise the impact of their research. This study seeks to better understand the facilitators, challenges and barriers to research impact and knowledge mobilisation from the perspective of UK public health researchers. A qualitative study using semi-structured interviews, informed by the Payback Framework, with public health researchers who held a research award with the National Institute for Health Research Public Health Research programme up to March 2017 was conducted. Following a thematic analysis, three strongly interlinked themes were extracted from the data and three key factors were highlighted as important for facilitating knowledge mobilisation and impact in UK public health research: (1) Public health researcher’s perception of the purpose of the research (2) Approaches to undertaking Knowledge mobilisation activities (3) The complex nature of public health research in the wider research context. These have been reflected onto the Payback framework. Public health researchers can maximise the likelihood for impact by being aware of the context in which they are undertaking research, using different methods, and employing several strategies to take advantage of opportunities. There is a need to support researchers with knowledge mobilisation activities and for funders to identify their expectations of the impact resulting from research. Our findings have relevance to public health researchers and funders interested in increasing the benefit that research brings to society.
Collapse
|
12
|
Trapero-Bertran M, Pokhrel S, Hanney S. Research can be integrated into public health policy-making: global lessons for and from Spanish economic evaluations. Health Res Policy Syst 2022; 20:67. [PMID: 35717247 PMCID: PMC9206096 DOI: 10.1186/s12961-022-00875-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2021] [Accepted: 05/27/2022] [Indexed: 11/26/2022] Open
Abstract
WHO promotes the use of research in policy-making to drive improvements in health, including in achieving Sustainable Development Goals such as tobacco control. The European Union’s new €95 billion Horizon Europe research framework programme parallels these aims, and also includes commitments to fund economic evaluations. However, researchers often express frustration at the perceived lack of attention to scientific evidence during policy-making. For example, some researchers claim that evidence regarding the return on investment from optimal implementation of evidence-based policies is frequently overlooked. An increasingly large body of literature acknowledges inevitable barriers to research use, but also analyses facilitators encouraging such use. This opinion piece describes how some research is integrated into policy-making. It highlights two recent reviews. One examines impact assessments of 36 multi-project research programmes and identifies three characteristics of projects more likely to influence policy-making. These include a focus on healthcare system needs, engagement of stakeholders, and research conducted for organizations supported by structures to receive and use evidence. The second review suggests that such characteristics are likely to occur as part of a comprehensive national health research system strategy, especially one integrated into the healthcare system. We also describe two policy-informing economic evaluations conducted in Spain. These examined the most cost-effective package of evidence-based tobacco control interventions and the cost-effectiveness of different strategies to increase screening coverage for cervical cancer. Both projects focused on issues of healthcare concern and involved considerable stakeholder engagement. The Spanish examples reinforce some lessons from the global literature and, therefore, could help demonstrate to authorities in Spain the value of developing comprehensive health research systems, possibly following the interfaces and receptor model. The aim of this would be to integrate needs assessment and stakeholder engagement with structures spanning the research and health systems. In such structures, economic evaluation evidence could be collated, analysed by experts in relation to healthcare needs, and fed into both policy-making as appropriate, and future research calls. The increasingly large local and global evidence base on research utilization could inform detailed implementation of this approach once accepted as politically desirable. Given the COVID-19 pandemic, increasing the cost-effectiveness of healthcare systems and return on investment of public health interventions becomes even more important.
Collapse
Affiliation(s)
- Marta Trapero-Bertran
- Basic Sciences Department, Patients Institute, Universitat Internacional de Catalunya, Barcelona, Spain
| | - Subhash Pokhrel
- Health Economics Research Group, Department of Health Sciences, Brunel University London, London, UK.
| | - Stephen Hanney
- Health Economics Research Group, Department of Health Sciences, Brunel University London, London, UK
| |
Collapse
|
13
|
Kork AA, Antonini C, García-Torea N, Luque-Vílchez M, Costa E, Senn J, Larrinaga C, Bertorello D, Brichetto G, Zaratin P, Andreaus M. Collective health research assessment: developing a tool to measure the impact of multistakeholder research initiatives. Health Res Policy Syst 2022; 20:49. [PMID: 35501895 PMCID: PMC9063051 DOI: 10.1186/s12961-022-00856-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2021] [Accepted: 04/12/2022] [Indexed: 11/27/2022] Open
Abstract
Background The need to more collaboratively measure the impact of health research and to do so from multidimensional perspectives has been acknowledged. A scorecard was developed as part of the Collective Research Impact Framework (CRIF), to engage stakeholders in the assessment of the impacts of health research and innovations. The purpose of this study was to describe the developmental process of the MULTI-ACT Master Scorecard (MSC) and how it can be used as a workable tool for collectively assessing future responsible research and innovation measures. Methods An extensive review of the health research impact literature and of multistakeholder initiatives resulted in a database of 1556 impact indicators. The MSC was then cocreated by engaging key stakeholders and conducting semi-structured interviews of experts in the field. Results The MSC consists of five accountability dimensions: excellence, efficacy, economic, social and patient-reported outcomes. The tool contains 125 potential indicators, classified into 53 impact measurement aspects that are considered the most relevant topics for multistakeholder research and innovation initiatives when assessing their impact on the basis of their mission and their stakeholders’ interests. The scorecard allows the strategic management of multistakeholder research initiatives to demonstrate their impact on people and society. The value of the tool is that it is comprehensive, customizable and easy to use. Conclusions The MSC is an example of how the views of society can be taken into account when research impacts are assessed in a more sustainable and balanced way. The engagement of patients and other stakeholders is an integral part of the CRIF, facilitating collaborative decision-making in the design of policies and research agendas. In policy making, the collective approach allows the evaluation perspective to be extended to the needs of society and towards responsible research and innovation. Multidimensionality makes research and innovations more responsive to systemic challenges, and developing more equitable and sustainable health services. Supplementary Information The online version contains supplementary material available at 10.1186/s12961-022-00856-9.
Collapse
Affiliation(s)
- Anna-Aurora Kork
- Faculty of Social Sciences, Tampere University, Tampere, Finland.
| | - Carla Antonini
- Department of Accounting, Universidad Autónoma de Madrid, Madrid, Spain
| | - Nicolás García-Torea
- Department of Economy and Business Administration, Universidad de Burgos, Burgos, Spain
| | - Mercedes Luque-Vílchez
- Department of Agriculture Economy, Finance and Accounting, Universidad de Córdoba, Córdoba, Spain.,European Financial Reporting Advisory Group (EFRAG), Brussels, Belgium
| | - Ericka Costa
- Department of Economic and Management, University of Trento, Trento, Italy
| | | | - Carlos Larrinaga
- Department of Economy and Business Administration, Universidad de Burgos, Burgos, Spain
| | | | | | - Paola Zaratin
- FISM-Italian Multiple Sclerosis Society Foundation, Genoa, Italy
| | - Michele Andreaus
- Department of Economic and Management, University of Trento, Trento, Italy
| |
Collapse
|
14
|
Newson RS, Rychetnik L, King L, Milat AJ, Bauman AE. Looking for evidence of research impact and use: A qualitative study of an Australian research-policy system. RESEARCH EVALUATION 2021. [DOI: 10.1093/reseval/rvab017] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022]
Abstract
Abstract
Current assessments of research impact have been criticized for capturing what can be easily counted not what actually counts. To empirically examine this issue, we approached measuring research impact from two directions, tracing forwards from research and backwards from policy, within a defined research-policy system (childhood obesity prevention research and policy in New South Wales, Australia from 2000 to 2015). The forward tracing research impact assessment component traced a sample of 148 local research projects forward to examine their policy impacts. Of the projects considered, 16% had an impact on local policy and for a further 19%, decision-makers were aware of the research, but there was no evidence it influenced policy decisions. The backward tracing component of the study included an analysis of research use across three policy initiatives. It provided a more nuanced understanding of the relative influence of research on policy. Both direct uses of specific research and indirect uses of research incorporated as broader bodies of knowledge were evident. Measuring research impact from both directions captured the diverse ways that research was used in decision-making. Our findings illustrate complexities in the assessment process and in real-life policymaking trajectories. They highlight the role that timing of assessment plays in perception of impacts and difficulties attributing longer-term impacts to specific research. This study supports the use of models where politics and complex system dynamics shape knowledge and its influence on decision-making, rather than research being the primary driver for policy change.
Collapse
Affiliation(s)
- Robyn S Newson
- School of Public Health, University of Sydney, Edward Ford Building (A27) Fisher Road, Sydney, NSW, 2006, Australia
| | - Lucie Rychetnik
- School of Public Health, University of Sydney, Edward Ford Building (A27) Fisher Road, Sydney, NSW, 2006, Australia
- The Australian Prevention Partnership Centre, Sax Institute, Glebe, Level 3, 30C Wentworth Street, NSW 2037, Australia
| | - Lesley King
- School of Public Health, University of Sydney, Edward Ford Building (A27) Fisher Road, Sydney, NSW, 2006, Australia
| | - Andrew J Milat
- School of Public Health, University of Sydney, Edward Ford Building (A27) Fisher Road, Sydney, NSW, 2006, Australia
| | - Adrian E Bauman
- School of Public Health, University of Sydney, Edward Ford Building (A27) Fisher Road, Sydney, NSW, 2006, Australia
| |
Collapse
|
15
|
Zakaria S, Grant J, Luff J. Fundamental challenges in assessing the impact of research infrastructure. Health Res Policy Syst 2021; 19:119. [PMID: 34407849 PMCID: PMC8371591 DOI: 10.1186/s12961-021-00769-z] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2021] [Accepted: 08/03/2021] [Indexed: 12/15/2022] Open
Abstract
Clinical research infrastructure is one of the unsung heroes of the scientific response to the current COVID-19 pandemic. The extensive, long-term funding into research support structures, skilled people, and technology allowed the United Kingdom research response to move off the starting blocks at pace by utilizing pre-existing platforms. The increasing focus from funders on evaluating the outcomes and impact of research infrastructure investment requires both a reframing and progression of the current models in order to address the contribution of the underlying support infrastructure. The majority of current evaluation/outcome models focus on a "pipeline" approach using a methodology which follows the traditional research funding route with the addition of quantitative metrics. These models fail to embrace the complexity caused by the interplay of previous investment, the coalescing of project outputs from different funders, the underlying infrastructure investment, and the parallel development across different parts of the system. Research infrastructure is the underpinning foundation of a project-driven research system and requires long-term, sustained funding and capital investment to maintain scientific and technological expertise. Therefore, the short-term focus on quantitative metrics that are easy to collect and interpret and that can be assessed in a roughly 5-year funding cycle needs to be addressed. The significant level of investment in research infrastructure necessitates investment to develop bespoke methodologies that develop fit-for-purpose, longer-term/continual approach(es) to evaluation. Real-world research should reflect real-world evaluation and allow for the accrual of a narrative of value indicators that build a picture of the contribution of infrastructure to research outcomes. The linear approach is not fit for purpose, the research endeavour is a complex, twisted road, and the evaluation approach needs to embrace this complexity through the development of realist approaches and the rapidly evolving data ecosystem. This paper sets out methodological challenges and considers the need to develop bespoke methodological approaches to allow a richer assessment of impact, contribution, attribution, and evaluation of research infrastructure. This paper is the beginning of a conversation that invites the community to "take up the mantle" and tackle the complexity of real-world research translation and evaluation.
Collapse
Affiliation(s)
- Sana Zakaria
- Central Commissioning Facility, National Institute of Health Research, 15 Church Street, TW1 3NL, Twickenham, United Kingdom.
| | - Jonathan Grant
- Policy Institute, King's College London, SE1 8WA, London, United Kingdom
| | - Jane Luff
- Central Commissioning Facility, National Institute of Health Research, 15 Church Street, TW1 3NL, Twickenham, United Kingdom
| |
Collapse
|
16
|
Razmgir M, Panahi S, Ghalichi L, Mousavi SAJ, Sedghi S. Exploring research impact models: A systematic scoping review. RESEARCH EVALUATION 2021. [DOI: 10.1093/reseval/rvab009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
Abstract
This article explores the models and frameworks developed on “research impact’. We aim to provide a comprehensive overview of related literature through scoping study method. The present research investigates the nature, objectives, approaches, and other main attributes of the research impact models. It examines to analyze and classify models based on their characteristics. Forty-seven studies and 10 reviews published between 1996 and 2020 were included in the analysis. The majority of models were developed for the impact assessment and evaluation purposes. We identified three approaches in the models, namely outcome-based, process-based, and those utilized both of them, among which the outcome-based approach was the most frequently used by impact models and evaluation was considered as the main objective of this group. The process-based ones were mainly adapted from the W.K. Kellogg Foundation logic model and were potentially eligible for impact improvement. We highlighted the scope of processes and other specific features for the recent models. Given the benefits of the process-based approach in enhancing and accelerating the research impact, it is important to consider such approach in the development of impact models. Effective interaction between researchers and stakeholders, knowledge translation, and evidence synthesis are the other possible driving forces contributing to achieve and improve impact.
Collapse
Affiliation(s)
- Maryam Razmgir
- Department of Medical library and Information Science, School of Health Management and Information Sciences, Iran University of Medical Sciences, PO Box 14665-354, Tehran, Iran
| | - Sirous Panahi
- Department of Medical library and Information Science, School of Health Management and Information Sciences, Iran University of Medical Sciences, PO Box 14665-354, Tehran, Iran
- Health Management and Economics Research Center, Iran University of Medical Sciences, PO Box 14665-354, Tehran, Iran
| | - Leila Ghalichi
- Mental Health Research Center, Psychosocial Health Research Institute, Iran University of Medical Sciences, PO Box 14665-354, Tehran, Iran
| | - Seyed Ali Javad Mousavi
- Department of Pulmonology, School of Medicine, Iran University of Medical Sciences, PO Box 14665-354, Tehran, Iran
| | - Shahram Sedghi
- Department of Medical library and Information Science, School of Health Management and Information Sciences, Iran University of Medical Sciences, PO Box 14665-354, Tehran, Iran
- Health Management and Economics Research Center, Iran University of Medical Sciences, PO Box 14665-354, Tehran, Iran
| |
Collapse
|
17
|
Millar R, Morton A, Bufali MV, Engels S, Dabak SV, Isaranuwatchai W, Chalkidou K, Teerawattananon Y. Assessing the performance of health technology assessment (HTA) agencies: developing a multi-country, multi-stakeholder, and multi-dimensional framework to explore mechanisms of impact. COST EFFECTIVENESS AND RESOURCE ALLOCATION 2021; 19:37. [PMID: 34215282 PMCID: PMC8252304 DOI: 10.1186/s12962-021-00290-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2020] [Accepted: 06/07/2021] [Indexed: 11/10/2022] Open
Abstract
Background Health technology assessment (HTA) agencies have an important role to play in managing the rising demands on health systems. However, creating and running such agencies potentially diverts resources from frontline services. A large number of studies address the question of ‘what is the impact of HTA?’. Several points of heterogeneity in this literature include: purpose of the study, definition of HTA, definition of impact, and scope and rigour of evaluations. Our study seeks to address several limitations in this literature. This study aims to explore the mechanisms of impact of an HTA agency. In doing so, we consider HTA as an institution rather than a knowledge product to build an impact evaluation framework from an international, multi-stakeholder and multi-dimensional perspective. Methods We conducted 9 key informant interviews with experts from the international HTA community. We addressed several questions, informed by existing frameworks of impact within the literature, to understand their perspectives on the mechanisms of impact of an HTA agency. We analyse data using logic modelling and impact mapping, as tools to understand and visualise mechanisms of change. Findings Our impact mapping highlights several distinct, but not necessarily mutually exclusive, mechanisms through which the overall impact of an HTA agency is achieved. These are: the effective conduct of HTA studies; effective use of HTA in agenda-setting and policy formulation processes; effective engagement and external communications; good institutional reputation and fit within the healthcare and policy-making system; effective use of HTA as a tool for the negotiation of health technology prices; and the effective implementation of policy change regarding health technologies. We also identify indicators of these effects. Conclusions Our findings and resulting evaluation framework complement and add to existing literature by offering a new perspective on the mechanisms by which HTA agencies generate impact. This new perspective considers HTA as an institution rather than a knowledge product, is international, multi-dimensional, and includes multi-stakeholder views. We hope the analysis will be useful to countries interested in managing HTA performance. Supplementary Information The online version contains supplementary material available at 10.1186/s12962-021-00290-8.
Collapse
Affiliation(s)
- Robyn Millar
- Department of Management Science, University of Strathclyde Business School, Sir William Duncan Building (Level 7), 130 Rottenrow, Glasgow, G4 0GE, UK.
| | - Alec Morton
- Department of Management Science, University of Strathclyde Business School, Sir William Duncan Building (Level 7), 130 Rottenrow, Glasgow, G4 0GE, UK
| | - Maria Vittoria Bufali
- Department of Management Science, University of Strathclyde Business School, Sir William Duncan Building (Level 7), 130 Rottenrow, Glasgow, G4 0GE, UK
| | - Sven Engels
- Health Intervention and Technology Assessment Program, Ministry of Public Health, Nonthaburi, Thailand
| | | | - Wanrudee Isaranuwatchai
- Health Intervention and Technology Assessment Program, Ministry of Public Health, Nonthaburi, Thailand.,Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada
| | - Kalipso Chalkidou
- Global Health at Imperial College London, London, UK.,Director of Global Health Policy and Senior Fellow at the Centre of Global Development, London, UK
| | - Yot Teerawattananon
- Health Intervention and Technology Assessment Program, Ministry of Public Health, Nonthaburi, Thailand.,Saw Swee Hock School of Public Health, National University of Singapore, Singapore, Singapore
| |
Collapse
|
18
|
Primary Care Research - Influencing and Implementing Into Policy. Zdr Varst 2021; 60:138-144. [PMID: 34249159 PMCID: PMC8256767 DOI: 10.2478/sjph-2021-0020] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Accepted: 01/29/2021] [Indexed: 11/20/2022] Open
Abstract
This editorial describes how research in primary health care can be used to influence policy. It draws on previous literature to give an example from the UK of how research in one part of primary care, the health-visiting service, has endeavoured to use evidence to influence policy and practice. The editorial considers frameworks for policy implementation such as Bardach's eight phase approach and concepts that can inform policy implementation such as Lipsky's Street-Level Bureaucrat approach.
Collapse
|
19
|
Kalibala S, Sinai I, Nutley T. Documenting HIV research-utilization activities, outputs and outcomes: examples and lessons learned from Project SOAR. ACTA ACUST UNITED AC 2021; 79:99. [PMID: 34130745 PMCID: PMC8204507 DOI: 10.1186/s13690-021-00628-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2021] [Accepted: 06/01/2021] [Indexed: 11/10/2022]
Abstract
The importance of using research findings to inform policy and program decisions is well recognized, but the literature on measuring research utilization activities is scarce. As funding to support some areas of research wanes or remains stagnant, the need to document the value of investing in research by its’ effect on improved programs and policies becomes increasingly necessary. We present the experience of Project SOAR, a six-year USAID-funded project focusing on HIV/AIDS-related implementation research, to demonstrate measurement of research utilization. We follow the project’s research-utilization logic model, including inputs, activities, outputs, and outcomes. We present tools the project developed and examples from project studies and discuss what works, remaining challenges and how to overcome them, and lessons learned. We then make recommendations for incorporating research-utilization activities and measurement in implementation-research studies.
Collapse
Affiliation(s)
- Samuel Kalibala
- Project SOAR, 4301 Connecticut Ave., NW, Suite 280, Palladium, Washington, DC, USA.
| | - Irit Sinai
- Data Informatics and Analytical Solutions, Palladium, Washington, DC, USA
| | - Tara Nutley
- Data Informatics and Analytical Solutions, Palladium, Chapel Hill, North Carolina, USA
| |
Collapse
|
20
|
Hanna CR, Boyd KA, Jones RJ. Evaluating cancer research impact: lessons and examples from existing reviews on approaches to research impact assessment. Health Res Policy Syst 2021; 19:36. [PMID: 33706777 PMCID: PMC7953786 DOI: 10.1186/s12961-020-00658-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2020] [Accepted: 11/09/2020] [Indexed: 12/15/2022] Open
Abstract
BACKGROUND Performing cancer research relies on substantial financial investment, and contributions in time and effort from patients. It is therefore important that this research has real life impacts which are properly evaluated. The optimal approach to cancer research impact evaluation is not clear. The aim of this study was to undertake a systematic review of review articles that describe approaches to impact assessment, and to identify examples of cancer research impact evaluation within these reviews. METHODS In total, 11 publication databases and the grey literature were searched to identify review articles addressing the topic of approaches to research impact assessment. Information was extracted on methods for data collection and analysis, impact categories and frameworks used for the purposes of evaluation. Empirical examples of impact assessments of cancer research were identified from these literature reviews. Approaches used in these examples were appraised, with a reflection on which methods would be suited to cancer research impact evaluation going forward. RESULTS In total, 40 literature reviews were identified. Important methods to collect and analyse data for impact assessments were surveys, interviews and documentary analysis. Key categories of impact spanning the reviews were summarised, and a list of frameworks commonly used for impact assessment was generated. The Payback Framework was most often described. Fourteen examples of impact evaluation for cancer research were identified. They ranged from those assessing the impact of a national, charity-funded portfolio of cancer research to the clinical practice impact of a single trial. A set of recommendations for approaching cancer research impact assessment was generated. CONCLUSIONS Impact evaluation can demonstrate if and why conducting cancer research is worthwhile. Using a mixed methods, multi-category assessment organised within a framework, will provide a robust evaluation, but the ability to perform this type of assessment may be constrained by time and resources. Whichever approach is used, easily measured, but inappropriate metrics should be avoided. Going forward, dissemination of the results of cancer research impact assessments will allow the cancer research community to learn how to conduct these evaluations.
Collapse
Affiliation(s)
- Catherine R. Hanna
- CRUK Clinical Trials Unit, Institute of Cancer Sciences, University of Glasgow, Glasgow, United Kingdom
| | - Kathleen A. Boyd
- Health Economics and Health Technology Assessment, Institute of Health and Wellbeing, University of Glasgow, Glasgow, United Kingdom
| | - Robert J. Jones
- CRUK Clinical Trials Unit, Institute of Cancer Sciences, University of Glasgow, Glasgow, United Kingdom
| |
Collapse
|
21
|
Van Eerd D, Moser C, Saunders R. A research impact model for work and health. Am J Ind Med 2021; 64:3-12. [PMID: 33231875 PMCID: PMC7756356 DOI: 10.1002/ajim.23201] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2020] [Revised: 09/29/2020] [Accepted: 10/27/2020] [Indexed: 12/29/2022]
Abstract
Research organizations, governments and funding agencies are increasingly interested in the impact of research beyond academia. While a growing literature describes research impacts in healthcare and health services, little has focused on occupational health and safety research. This article describes a research impact model that has been in use for over a decade. The model was developed to track and describe the impact of research conducted by a mid-sized institute that focuses on work and health. Model development was informed by existing models, with the goal of contextualizing the institute's case studies describing three types of research impact: evidence of the diffusion of research; evidence of research informing decision-making; and evidence of societal impact. A logic model describes research actions and outcomes, as well as key audiences and knowledge transfer approaches. A unique element is its indication of the level of difficulty in determining types of impact. The model compares well with current research impact models developed or used in healthcare and health services research, and it has been useful in guiding a mid-sized research organization's process for tracking and describing the impact of its research. It may be useful to other small and mid-sized research organizations that focus on workplace health and safety.
Collapse
Affiliation(s)
| | - Cindy Moser
- Institute for Work and Health Toronto Ontario Canada
| | - Ron Saunders
- Institute for Work and Health Toronto Ontario Canada
| |
Collapse
|
22
|
Büttner F, Ardern CL, Blazey P, Dastouri S, McKay HA, Moher D, Khan KM. Counting publications and citations is not just irrelevant: it is an incentive that subverts the impact of clinical research. Br J Sports Med 2020; 55:647-648. [PMID: 33361277 PMCID: PMC8208942 DOI: 10.1136/bjsports-2020-103146] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/13/2020] [Indexed: 11/09/2022]
Affiliation(s)
- Fionn Büttner
- School of Public Health, Physiotherapy and Sports Science, University College Dublin, Dublin, Ireland
| | - Clare L Ardern
- Division of Physiotherapy, Karolinska Institute, Stockholm, Sweden.,Sport & Exercise Medicine Research Centre, La Trobe University, Melbourne, Victoria, Australia
| | - Paul Blazey
- Centre for Hip Health and Mobility, The University of British Columbia, Vancouver, British Columbia, Canada
| | - Serenna Dastouri
- Institute of Musculoskeletal Health and Arthritis, Canadian Institutes of Health Research, Vancouver, Ontario, Canada
| | - Heather A McKay
- Deparment of Orthopaedics, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
| | - David Moher
- Faculty of Medicine, University of Ottawa School of Epidemiology and Public Health, Ottawa, Ontario, Canada.,Centre for Journalology and Canadian EQUATOR Centre, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
| | - Karim M Khan
- Centre for Hip Health and Mobility, The University of British Columbia, Vancouver, British Columbia, Canada.,Department of Family Practice and School of Kinesiology, The University of British Columbia, Vancouver, British Columbia, Canada
| |
Collapse
|
23
|
Onken J, Miklos AC, Aragon R. Tracing Long-Term Outcomes of Basic Research Using Citation Networks. Front Res Metr Anal 2020; 5:5. [PMID: 33870043 PMCID: PMC8028394 DOI: 10.3389/frma.2020.00005] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2020] [Accepted: 08/03/2020] [Indexed: 12/22/2022] Open
Abstract
In recent years, the science of science policy has been facilitated by the greater availability of and access to digital data associated with the science, technology, and innovation enterprise. Historically, most of the studies from which such data are derived have been econometric or "scientometric" in nature, focusing on the development of quantitative data, models, and metrics of the scientific process as well as outputs and outcomes. Broader definitions of research impact, however, necessitate the use of qualitative case-study methods. For many years, U.S. federal science agencies such as the National Institutes of Health have demonstrated the impact of the research they support through tracing studies that document critical events in the development of successful technologies. A significant disadvantage and barrier of such studies is the labor-intensive nature of a case study approach. Currently, however, the same data infrastructures that have been developed to support scientometrics may also facilitate historical tracing studies. In this paper, we describe one approach we used to discover long-term, downstream outcomes of research supported in the late 1970's and early 1980's by the National Institute of General Medical Sciences, a component of the National Institutes of Health.
Collapse
Affiliation(s)
- James Onken
- Research Enterprise Analytics, LLC, Rockville, MD, United States
| | - Andrew C Miklos
- Division of Data Integration, Modeling, and Analytics, National Institute of General Medical Sciences, Bethesda, MD, United States
| | - Richard Aragon
- Division of Data Integration, Modeling, and Analytics, National Institute of General Medical Sciences, Bethesda, MD, United States
| |
Collapse
|
24
|
Carroll C, Tattersall A. Research and Policy Impact of Trials Published by the UK National Institute of Health Research (2006-2015). VALUE IN HEALTH : THE JOURNAL OF THE INTERNATIONAL SOCIETY FOR PHARMACOECONOMICS AND OUTCOMES RESEARCH 2020; 23:727-733. [PMID: 32540230 DOI: 10.1016/j.jval.2020.01.012] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/21/2019] [Revised: 01/17/2020] [Accepted: 01/25/2020] [Indexed: 06/11/2023]
Abstract
OBJECTIVES Health technology assessment aims to inform and support healthcare decision making, and trials are part of that process. The purpose of this study was to measure the impact of a sample of trials in a meaningful but robust fashion. METHODS All randomized controlled trials funded and published by the UK National Institute of Health Research in the Health Technology Assessment journal series and other peer-reviewed journals were identified for 2006 to 2015. Citation analysis was performed for all trials, and quantitative content analysis was undertaken on a purposive sample to determine whether impact could be categorized as "instrumental" (ie, having a clear influence on key research and policy publications). RESULTS The search identified 133 relevant trials. The citation rate per trial was 102.97. Of the 133 trials, 129 (98%) were cited in 1 or more systematic reviews or meta-analyses (mean per trial = 7.18, range = 0-44). Where they were cited, the trials were used in some form of synthesis 63% of the time. Ninety-one of the 133 (68%) trials were found to be cited in 1 or more guidance or policy document (mean per trial = 2.75, range = 0-26) and had an instrumental influence 41% of the time. The publication of these trials' results in journals other than the Health Technology Assessment journal appears to enhance the discoverability of the trial data. Altmetric.com proved to be very useful in identifying unique policy and guidance documents. CONCLUSION These trials have impressive citation rates, and a sizeable proportion are certainly being used in key publications in a genuinely instrumental manner.
Collapse
Affiliation(s)
- Christopher Carroll
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, England, UK.
| | - Andy Tattersall
- School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, England, UK
| |
Collapse
|
25
|
van Munster JJCM, Zamanipoor Najafabadi AH, de Boer NP, Peul WC, van den Hout WB, van Benthem PPG. Impact of surgical intervention trials on healthcare: A systematic review of assessment methods, healthcare outcomes, and determinants. PLoS One 2020; 15:e0233318. [PMID: 32442235 PMCID: PMC7244162 DOI: 10.1371/journal.pone.0233318] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2020] [Accepted: 05/01/2020] [Indexed: 12/16/2022] Open
Abstract
BACKGROUND Frameworks used in research impact evaluation studies vary widely and it remains unclear which methods are most appropriate for evaluating research impact in the field of surgical research. Therefore, we aimed to identify and review the methods used to assess the impact of surgical intervention trials on healthcare and to identify determinants for surgical impact. METHODS We searched journal databases up to March 10, 2020 for papers assessing the impact of surgical effectiveness trials on healthcare. Two researchers independently screened the papers for eligibility and performed a Risk of Bias assessment. Characteristics of both impact papers and trial papers were summarized. Univariate analyses were performed to identify determinants for finding research impact, which was defined as a change in healthcare practice. RESULTS Sixty-one impact assessments were performed in 37 included impact papers. Some surgical trial papers were evaluated in more than one impact paper, which provides a total of 38 evaluated trial papers. Most impact papers were published after 2010 (n = 29). Medical records (n = 10), administrative databases (n = 22), and physician's opinion through surveys (n = 5) were used for data collection. Those data were analyzed purely descriptively (n = 3), comparing data before and after publication (n = 29), or through time series analyses (n = 5). Significant healthcare impact was observed 49 times and more often in more recent publications. Having impact was positively associated with using medical records or administrative databases (ref.: surveys), a longer timeframe for impact evaluation and more months between the publication of the trial paper and the impact paper, data collection in North America (ref.: Europe), no economic evaluation of the intervention, finding no significant difference in surgical outcomes, and suggesting de-implementation in the original trial paper. CONCLUSIONS AND IMPLICATIONS Research impact evaluation receives growing interest, but still a small number of impact papers per year was identified. The analysis showed that characteristics of both surgical trial papers and impact papers were associated with finding research impact. We advise to collect data from either medical records or administrative databases, with an evaluation time frame of at least 4 years since trial publication.
Collapse
Affiliation(s)
- Juliëtte J. C. M. van Munster
- Department of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical Center (LUMC), Leiden University, Leiden, the Netherlands
- Leiden University Neurosurgical Center Holland (UNCH), LUMC and The Hague Medical Center (HMC), Leiden, the Netherlands
| | - Amir H. Zamanipoor Najafabadi
- Leiden University Neurosurgical Center Holland (UNCH), LUMC and The Hague Medical Center (HMC), Leiden, the Netherlands
| | - Nick P. de Boer
- Department of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical Center (LUMC), Leiden University, Leiden, the Netherlands
| | - Wilco C. Peul
- Leiden University Neurosurgical Center Holland (UNCH), LUMC and The Hague Medical Center (HMC), Leiden, the Netherlands
| | - Wilbert B. van den Hout
- Department of Biomedical Data Science–Medical Decision Making, Leiden University Medical Center, Leiden University, Leiden, the Netherlands
| | - Peter Paul G. van Benthem
- Department of Otorhinolaryngology and Head and Neck Surgery, Leiden University Medical Center (LUMC), Leiden University, Leiden, the Netherlands
| |
Collapse
|
26
|
Boulding H, Kamenetzky A, Ghiga I, Ioppolo B, Herrera F, Parks S, Manville C, Guthrie S, Hinrichs-Krapels S. Mechanisms and pathways to impact in public health research: a preliminary analysis of research funded by the National Institute for Health Research (NIHR). BMC Med Res Methodol 2020; 20:34. [PMID: 32075580 PMCID: PMC7031933 DOI: 10.1186/s12874-020-0905-7] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2018] [Accepted: 01/20/2020] [Indexed: 11/23/2022] Open
Abstract
Background The mechanisms and pathways to impacts from public health research in the UK have not been widely studied. Through the lens of one funder (NIHR), our aims are to map the diversity of public health research, in terms of funding mechanisms, disciplinary contributions, and public health impacts, identify examples of impacts, and pathways to impact that existing reporting mechanisms may not otherwise have captured, and provide illustrations of how public health researchers perceive the generation of non-academic impact from their work. Methods A total of 1386 projects were identified as ‘public health research’ by the NIHR and listed in the NIHR Public Health Overview database (2000–2016). From these, a subset of 857 projects were matched as potentially having begun reporting impacts via an external data-gathering platform (Researchfish). Data on the 857 projects were analyzed quantitatively, and nine projects were selected to investigate further through semi-structured interviews with principal investigators. Two workshops took place to validate emerging and final findings and facilitate analysis. Results In addition to the NIHR School for Public Health Research and the NIHR Public Health Research Programme, 89% of projects contained in the NIHR Public Health Overview portfolio as ‘public health research’ are funded via other NIHR research programmes, suggesting significant diversity in disciplines contributing to public health research and outcomes. The pathways to impact observed in our in-depth case studies include contributing to debates on what constitutes appropriate evidence for national policy change, acknowledging local ‘unintended’ impacts, building trusted relationships with stakeholders across health and non-health sectors and actors, collaborating with local authorities, and using non-academic dissemination channels. Conclusions Public health as a discipline contributes substantially to impact beyond academia. To support the diversity of these impacts, we need to recognise localized smaller-scale impacts, and the difference in types of evidence required for community and local authority-based impacts. This will also require building capacity and resources to enable impact to take place from public health research. Finally, support is required for engagement with local authorities and working with non-health sectors that contribute to health outcomes.
Collapse
Affiliation(s)
- Harriet Boulding
- The Policy Institute, King's College London, 22 Kingsway, London, WC2B 6LE, UK
| | - Adam Kamenetzky
- The Policy Institute, King's College London, 22 Kingsway, London, WC2B 6LE, UK
| | - Ioana Ghiga
- RAND Europe, Westbrook Centre, Cambridge, CB4 1YG, UK
| | - Becky Ioppolo
- RAND Europe, Westbrook Centre, Cambridge, CB4 1YG, UK
| | | | - Sarah Parks
- RAND Europe, Westbrook Centre, Cambridge, CB4 1YG, UK
| | | | - Susan Guthrie
- RAND Europe, Westbrook Centre, Cambridge, CB4 1YG, UK
| | | |
Collapse
|
27
|
Kamenetzky A, Hinrichs-Krapels S. How do organisations implement research impact assessment (RIA) principles and good practice? A narrative review and exploratory study of four international research funding and administrative organisations. Health Res Policy Syst 2020; 18:6. [PMID: 31959198 PMCID: PMC6971910 DOI: 10.1186/s12961-019-0515-1] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2019] [Accepted: 12/09/2019] [Indexed: 12/14/2022] Open
Abstract
Background Public research funding agencies and research organisations are increasingly accountable for the wider impacts of the research they support. While research impact assessment (RIA) frameworks and tools exist, little is known and shared of how these organisations implement RIA activities in practice. Methods We conducted a review of academic literature to search for research organisations’ published experiences of RIAs. We followed this with semi-structured interviews from a convenience sample (n = 7) of representatives of four research organisations deploying strategies to support and assess research impact. Results We found only five studies reporting empirical evidence on how research organisations put RIA principles into practice. From our interviews, we observed a disconnect between published RIA frameworks and tools, and the realities of organisational practices, which tended not to be reported. We observed varying maturity and readiness with respect to organisations’ structural set ups for conducting RIAs, particularly relating to leadership, skills for evaluation and automating RIA data collection. Key processes for RIA included efforts to engage researcher communities to articulate and plan for impact, using a diversity of methods, frameworks and indicators, and supporting a learning approach. We observed outcomes of RIAs as having supported a dialogue to orient research to impact, underpinned shared learning from analyses of research, and provided evidence of the value of research in different domains and to different audiences. Conclusions Putting RIA principles and frameworks into practice is still in early stages for research organisations. We recommend that organisations (1) get set up by considering upfront the resources, time and leadership required to embed impact strategies throughout the organisation and wider research ‘ecosystem’, and develop methodical approaches to assessing impact; (2) work together by engaging researcher communities and wider stakeholders as a core part of impact pathway planning and subsequent assessment; and (3) recognise the benefits that RIA can bring about as a means to improve mutual understanding of the research process between different actors with an interest in research.
Collapse
Affiliation(s)
- Adam Kamenetzky
- National Institute for Health Research Central Commissioning Facility, Twickenham, TW1 3NL, United Kingdom. .,Policy Institute at King's College London, Strand Campus, London, WC2B 6LE, United Kingdom.
| | - Saba Hinrichs-Krapels
- Policy Institute at King's College London, Strand Campus, London, WC2B 6LE, United Kingdom.,King's Global Health Institute, King's College London, Denmark Hill, London, SE5 9RJ, United Kingdom
| |
Collapse
|
28
|
Sivertsen G, Meijer I. Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society? RESEARCH EVALUATION 2019. [DOI: 10.1093/reseval/rvz032] [Citation(s) in RCA: 27] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
Abstract
Societal impact of research does not occur primarily as unexpected extraordinary incidents of particularly useful breakthroughs in science. It is more often a result of normal everyday interactions between organizations that need to create, exchange, and make use of new knowledge to further their goals. We use the distinctions between normal and extraordinary societal impact and between organizational- and individual-level activities and responsibilities to discuss how science–society relations can better be understood, evaluated, and improved by focusing on the organizations that typically interact in a specific domain of research.
Collapse
Affiliation(s)
- Gunnar Sivertsen
- Nordic Institute for Studies in Innovation, Research and Education (NIFU), Tøyen, Oslo N-0608, Norway
| | - Ingeborg Meijer
- Centre for Science and Technology Studies (CWTS), Leiden University, Leiden AX 2300, The Netherlands
| |
Collapse
|
29
|
Braithwaite RS. Evidence-based medicine: clinicians are taught to say it but not taught to think it. BMJ Evid Based Med 2019; 24:165-167. [PMID: 30275103 PMCID: PMC6837243 DOI: 10.1136/bmjebm-2018-110970] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Download PDF] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 08/27/2018] [Indexed: 11/13/2022]
|
30
|
Smith JBE, Channon K, Kiparoglou V, Forbes JF, Gray AM. A macroeconomic assessment of the impact of medical research expenditure: A case study of NIHR Biomedical Research Centres. PLoS One 2019; 14:e0214361. [PMID: 30970015 PMCID: PMC6457483 DOI: 10.1371/journal.pone.0214361] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2018] [Accepted: 03/12/2019] [Indexed: 11/18/2022] Open
Abstract
Quantifying the value of investment in medical research can inform decision-making on the prioritisation of research programmes. Existing methodologies to estimate the rate of return of medical research are inappropriate for early-phase translational research due to censoring of health benefits and time lags. A strategy to improve the process of translational research for patient benefit has been initiated as part of the UK National Institute for Health Research (NIHR) investment in Biomedical Research Centres (BRCs) in England. By providing a platform for partnership between universities, NHS trusts and industry, successful BRCs should reduce time lags within translational research whilst also providing an impetus for local economic growth through industry collaboration. We present a novel contribution in the assessment of early-phase biomedical research by estimating the impact of the Oxford Biomedical Research Centre (OxBRC) on income and job creation following the initial NIHR investment. We adopt a macroeconomic assessment approach using Input-Output Analysis to estimate the value of medical research in terms of income and job creation during the early pathway towards translational biomedical research. Inter-industry linkages are assessed by building a model economy for the South East England region to estimate the return on investment of the OxBRC. The results from the input-output model estimate that the return on investment in biomedical research within the OxBRC is 46%. Each £1 invested in the OxBRC generates an additional £0.46 through income and job creation alone. Multiplicative employment effects following a marginal investment in the OxBRC of £98m during the period 2007-2017 result in an estimated additional 196 full time equivalent positions being created within the local economy on top of direct employment within OxBRC. Results from input-output analyses can be used to inform the prioritisation of biomedical research programmes when compared against national minimum thresholds of investment.
Collapse
Affiliation(s)
- Joel B. E. Smith
- Health Economics Research Centre, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, John Radcliffe Hospital, Oxford, United Kingdom
| | - Keith Channon
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, John Radcliffe Hospital, Oxford, United Kingdom
- British Heart Foundation Centre of Research Excellence, Division of Cardiovascular Medicine, University of Oxford, Oxford, United Kingdom
- Wellcome Trust Centre for Human Genetics, University of Oxford, Oxford, United Kingdom
| | - Vasiliki Kiparoglou
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, John Radcliffe Hospital, Oxford, United Kingdom
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom
| | - John F. Forbes
- Graduate Entry Medical School, University of Limerick, Limerick, Ireland
| | - Alastair M. Gray
- Health Economics Research Centre, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom
- NIHR Oxford Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, John Radcliffe Hospital, Oxford, United Kingdom
| |
Collapse
|
31
|
Tsey K, Onnis LA, Whiteside M, McCalman J, Williams M, Heyeres M, Lui SMC, Klieve H, Cadet-James Y, Baird L, Brown C, Watkin Lui F, Grainger D, Gabriel Z, Millgate N, Cheniart B, Hunter T, Liu HB, Yinghong Y, Yan L, Lovett R, Chong A, Kinchin I. Assessing research impact: Australian Research Council criteria and the case of Family Wellbeing research. EVALUATION AND PROGRAM PLANNING 2019; 73:176-186. [PMID: 30665124 DOI: 10.1016/j.evalprogplan.2019.01.004] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/19/2018] [Revised: 12/11/2018] [Accepted: 01/11/2019] [Indexed: 06/09/2023]
Abstract
Researchers worldwide are increasingly reporting the societal impact of their research as part of national research productivity assessments. However, the challenges they encounter in developing their impact case studies against specified government assessment criteria and how pitfalls can be mitigated are not reported. This paper examines the key steps taken to develop an Aboriginal Family Wellbeing (FWB) empowerment research impact case study in the context of an Australian Research Council (ARC) pilot research impact assessment exercise and the challenges involved in applying the ARC criteria. The requirement that researchers demonstrate how their institutions support them to conduct impactful research has the potential to create supportive environments for researchers to be more responsive to the needs of users outside academia. However, the 15-year reference period for the associated research underpinning the reported impact and the focus on researcher's current institutional affiliation constitute potential constraints to demonstrating the true impact of research. For researchers working with Indigenous people, relationships that build over long periods of time, irrespective of university affiliation, are critical to conducting impactful research. A more open-ended time-frame, with no institutional restrictions for the 'associated research' provides the best opportunity to demonstrate the true benefits of research not only for Indigenous people but for Australian society more broadly.
Collapse
Affiliation(s)
- Komla Tsey
- The Cairns Institute and College of Arts Society and Education, James Cook University, Cairns, Australia; The Cairns Institute, James Cook University, PO Box 6811, Cairns, QLD, 4870, Australia.
| | - Leigh-Ann Onnis
- Indigenous Education and Research Centre and The Cairns Institute, James Cook University, Cairns, Australia.
| | - Mary Whiteside
- School of Allied Health, La Trobe University, Bundoora, Australia.
| | - Janya McCalman
- School of Health, Medicine and Applied Sciences, Central Queensland University, Cairns, Australia.
| | - Megan Williams
- Graduate School of Health, University of Technology Sydney, Sydney, Australia.
| | - Marion Heyeres
- The Cairns Institute and College of Arts Society and Education, James Cook University, Cairns, Australia.
| | - Siu Man Carrie Lui
- The Cairns Institute and College of Arts Society and Education, James Cook University, Cairns, Australia.
| | - Helen Klieve
- School of Education and Professional Studies, Griffith University, Mt Gravatt, Australia.
| | - Yvonne Cadet-James
- Indigenous Education and Research Centre, James Cook University, Townsville, Australia.
| | - Leslie Baird
- Indigenous Education and Research Centre and The Cairns Institute, James Cook University, Cairns, Australia.
| | - Catherine Brown
- Indigenous Education and Research Centre and The Cairns Institute, James Cook University, Cairns, Australia.
| | - Felecia Watkin Lui
- Indigenous Education and Research Centre and The Cairns Institute, James Cook University, Cairns, Australia.
| | - Daniel Grainger
- College of Business, Law and Governance, James Cook University, Townsville, Australia.
| | - Zona Gabriel
- Central Coast Primary Care, New South Wales, Australia.
| | | | - Ben Cheniart
- Central Coast Primary Care, New South Wales, Australia.
| | - Tahalani Hunter
- Indigenous Education and Research Centre and The Cairns Institute, James Cook University, Cairns, Australia.
| | - Hong-Bo Liu
- College of Business, Law and Governance, James Cook University, Townsville, Australia.
| | - Yang Yinghong
- College of Economics and Management, Shenyang University of Chemical Technology, Shenyang, Liaoning Province, China.
| | - Li Yan
- College of Economics and Management, Shenyang University of Chemical Technology, Shenyang, Liaoning Province, China.
| | - Ray Lovett
- Aboriginal and Torres Strait Islander Health, National Centre for Epidemiology and Population Health, Australian National University, Canberra, Australia.
| | - Alwin Chong
- Sansom Institute for Health Research Division of Health Sciences, University of South Australia, Adelaide, Australia.
| | - Irina Kinchin
- Centre for Indigenous Health Equity Research / School of Health, Medical and Applied Sciences Central Queensland University.
| |
Collapse
|
32
|
Heyeres M, Tsey K, Yang Y, Yan L, Jiang H. The characteristics and reporting quality of research impact case studies: A systematic review. EVALUATION AND PROGRAM PLANNING 2019; 73:10-23. [PMID: 30453183 DOI: 10.1016/j.evalprogplan.2018.11.002] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/22/2018] [Revised: 10/22/2018] [Accepted: 11/06/2018] [Indexed: 06/09/2023]
Abstract
Despite the growing expectation that researchers report the impact of their research using a case study approach, systematic reviews of research impact have focused on frameworks, indicators, methods of data collection and assessment rather than impact case studies. Our aim is to provide an overview of the characteristics of published research impact case studies, including translation activities, and their reporting quality. We searched for peer-reviewed impact studies published between 2000 and 2018 using a case study approach and selected 25 suitable papers. We applied descriptive statistics to study characteristics, conducted thematic analysis of research translation activities and assessed reporting quality using the 10-point ISRIA statement. 24 papers reported intermediate impacts, such as advocacy, or the development of statements, tools, or technology. 4 reported on longer-term societal impacts, such as health outcomes and economic return on investment. 7 reported on translation activities. Papers scored well against the ISRIA statement on 5 domains of reporting quality. Weakest scores centred around identification of stakeholder needs and stakeholder involvement, and ethics and conflict of interest. We identified the need for more consistency in reporting through a case study approach, more systematic reporting of translation pathways and greater transparency concerning estimated costs and benefits of the research and its translation and impact assessment.
Collapse
Affiliation(s)
- Marion Heyeres
- The Cairns Institute & College of Arts Society and Education, James Cook University, Cairns, QLD, Australia
| | - Komla Tsey
- The Cairns Institute & College of Arts Society and Education, James Cook University, Cairns, QLD, Australia.
| | - Yinghong Yang
- Department of International Trade, College of Economics and Management, Shenyang University of Chemical Technology, Shenyang, China
| | - Li Yan
- Department of International Trade, College of Economics and Management, Shenyang University of Chemical Technology, Shenyang, China
| | - Hua Jiang
- School of Higher Education, Dalian University of Technology, Dalian, China
| |
Collapse
|
33
|
Gomes D, Stavropoulou C. The impact generated by publicly and charity-funded research in the United Kingdom: a systematic literature review. Health Res Policy Syst 2019; 17:22. [PMID: 30819185 PMCID: PMC6394081 DOI: 10.1186/s12961-019-0425-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2018] [Accepted: 02/11/2019] [Indexed: 12/11/2022] Open
Abstract
Objective To identify, synthesise and critically assess the empirical evidence of the impact generated by publicly and charity-funded health research in the United Kingdom. Methods We conducted a systematic literature review of the empirical evidence published in English in peer-reviewed journals between 2006 and 2017. Studies meeting the inclusion criteria were selected and their findings were analysed using the Payback Framework and categorised into five main dimensions, namely knowledge, benefits to future research and research use, benefits from informing policy and product development, health and health sector benefits, and broader economic benefits. We assessed the studies for risk of selection, reporting and funding bias. Results Thirteen studies met the inclusion criteria. The majority of the studies (10 out of 13) assessed impact at multiple domains including the main five key themes of the Payback Framework. All of them showed a positive impact of funded research on outcomes. Of those studies, one (8%), six (46%) and six (46%) presented a low, moderate and high risk of bias, respectively. Conclusions Empirical evidence on the impact of publicly and charity-funded research is still limited and subject to funding and selection bias. More work is needed to establish the causal effects of funded research on academic outcomes, policy, practice and the broader economy. Electronic supplementary material The online version of this article (10.1186/s12961-019-0425-2) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Daniela Gomes
- School of Health Sciences, City, University of London, Northampton Square, EC1V 0HB, London, UK
| | - Charitini Stavropoulou
- School of Health Sciences, City, University of London, Northampton Square, EC1V 0HB, London, UK.
| |
Collapse
|
34
|
Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). J Informetr 2019. [DOI: 10.1016/j.joi.2019.01.008] [Citation(s) in RCA: 51] [Impact Index Per Article: 10.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
|
35
|
Kristensen FB, Husereau D, Huić M, Drummond M, Berger ML, Bond K, Augustovski F, Booth A, Bridges JFP, Grimshaw J, IJzerman MJ, Jonsson E, Ollendorf DA, Rüther A, Siebert U, Sharma J, Wailoo A. Identifying the Need for Good Practices in Health Technology Assessment: Summary of the ISPOR HTA Council Working Group Report on Good Practices in HTA. VALUE IN HEALTH : THE JOURNAL OF THE INTERNATIONAL SOCIETY FOR PHARMACOECONOMICS AND OUTCOMES RESEARCH 2019; 22:13-20. [PMID: 30661627 DOI: 10.1016/j.jval.2018.08.010] [Citation(s) in RCA: 67] [Impact Index Per Article: 13.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/12/2018] [Revised: 08/10/2018] [Accepted: 08/14/2018] [Indexed: 05/11/2023]
Abstract
The systematic use of evidence to inform healthcare decisions, particularly health technology assessment (HTA), has gained increased recognition. HTA has become a standard policy tool for informing decision makers who must manage the entry and use of pharmaceuticals, medical devices, and other technologies (including complex interventions) within health systems, for example, through reimbursement and pricing. Despite increasing attention to HTA activities, there has been no attempt to comprehensively synthesize good practices or emerging good practices to support population-based decision-making in recent years. After the identification of some good practices through the release of the ISPOR Guidelines Index in 2013, the ISPOR HTA Council identified a need to more thoroughly review existing guidance. The purpose of this effort was to create a basis for capacity building, education, and improved consistency in approaches to HTA-informed decision-making. Our findings suggest that although many good practices have been developed in areas of assessment and some other key aspects of defining HTA processes, there are also many areas where good practices are lacking. This includes good practices in defining the organizational aspects of HTA, the use of deliberative processes, and measuring the impact of HTA. The extent to which these good practices are used and applied by HTA bodies is beyond the scope of this report, but may be of interest to future researchers.
Collapse
Affiliation(s)
| | - Don Husereau
- School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Ottawa, ON, Canada; Department of Public Health, Health Services Research and Health Technology Assessment, UMIT - University of Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria.
| | - Mirjana Huić
- Agency for Quality and Accreditation in Health Care and Social Welfare, Zagreb, Croatia
| | | | | | - Kenneth Bond
- Patient Engagement, Ethics and International Affairs, Canadian Agency for Drugs and Technologies in Health (CADTH), Ottawa, ON, Canada
| | - Federico Augustovski
- Economic Evaluations and HTA Department, Institute for Clinical Effectiveness and Health Policy (IECS), Buenos Aires, Argentina
| | - Andrew Booth
- ScHARR, The University of Sheffield, Sheffield, UK
| | - John F P Bridges
- Department of Biomedical Informatics, The Ohio State University College of Medicine, Columbus, OH, USA
| | - Jeremy Grimshaw
- Cochrane Canada and Professor of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Maarten J IJzerman
- School of Population and Global Health, University of Melbourne, Melbourne, Australia; Department of Health Technology & Services Research, University of Twente, Enschede, The Netherlands
| | - Egon Jonsson
- Institute of Health Economics, Edmonton, AB, Canada
| | - Daniel A Ollendorf
- Center for the Evaluation of Value and Risk in Health (CEVR), Tufts University, Boston, MA, USA
| | - Alric Rüther
- International Affairs, Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany
| | - Uwe Siebert
- Department of Public Health, Health Services Research and Health Technology Assessment, UMIT - University of Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria; Division of Health Technology Assessment, ONCOTYROL - Center for Personalized Cancer Medicine, Innsbruck, Austria; Institute for Technology Assessment, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, and Department of Health Policy and Management, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Jitendar Sharma
- AP MedTech Zone & Advisor (Health), Department of Health & Family Welfare, Andhra Pradesh, India
| | - Allan Wailoo
- ScHARR, The University of Sheffield, Sheffield, UK; NICE Decision Support Unit, Sheffield, UK
| |
Collapse
|
36
|
Beckett K, Farr M, Kothari A, Wye L, le May A. Embracing complexity and uncertainty to create impact: exploring the processes and transformative potential of co-produced research through development of a social impact model. Health Res Policy Syst 2018; 16:118. [PMID: 30537975 PMCID: PMC6288891 DOI: 10.1186/s12961-018-0375-0] [Citation(s) in RCA: 59] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2018] [Accepted: 10/02/2018] [Indexed: 11/17/2022] Open
Abstract
The potential use, influence and impact of health research is seldom fully realised. This stubborn problem has caused burgeoning global interest in research aiming to address the implementation ‘gap’ and factors inhibiting the uptake of scientific evidence. Scholars and practitioners have questioned the nature of evidence used and required for healthcare, highlighting the complex ways in which knowledge is formed, shared and modified in practice and policy. This has led to rapid expansion, expertise and innovation in the field of knowledge mobilisation and funding for experimentation into the effectiveness of different knowledge mobilisation models. One approach gaining prominence involves stakeholders (e.g. researchers, practitioners, service users, policy-makers, managers and carers) in the co-production, and application, of knowledge for practice, policy and research (frequently termed integrated knowledge translation in Canada). Its popularity stems largely from its potential to address dilemmas inherent in the implementation of knowledge generated using more reductionist methods. However, despite increasing recognition, demands for co-produced research to illustrate its worth are becoming pressing while the means to do so remain challenging. This is due not only to the diversity of approaches to co-production and their application, but also to the ways through which different stakeholders conceptualise, measure, reward and use research. While research co-production can lead to demonstrable benefits such as policy or practice change, it may also have more diffuse and subtle impact on relationships, knowledge sharing, and in engendering culture shifts and research capacity-building. These relatively intangible outcomes are harder to measure and require new emphases and tools. This opinion paper uses six Canadian and United Kingdom case studies to explore the principles and practice of co-production and illustrate how it can influence interactions between research, policy and practice, and benefit diverse stakeholders. In doing so, we identify a continuum of co-production processes. We propose and illustrate the use of a new ‘social model of impact’ and framework to capture multi-layered and potentially transformative impacts of co-produced research. We make recommendations for future directions in research co-production and impact measurement.
Collapse
Affiliation(s)
- Kate Beckett
- The University of The West of England, Centre for Child & Adolescent Health, Oakfield House, Oakfield Grove, Bristol, BS8 2BN, United Kingdom.
| | - Michelle Farr
- NIHR CLAHRC West, University Hospitals Bristol NHS Foundation Trust, 9th Floor, Whitefriars, Lewins Mead, Bristol, BS1 2NT, United Kingdom.,Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, United Kingdom
| | - Anita Kothari
- School of Health Studies, University of Western Ontario, Health Sciences Building Room 222, London, ON, N6A 5B9, Canada
| | - Lesley Wye
- Centre for Academic Primary Care, Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, United Kingdom
| | - Andrée le May
- Faculty of Health Sciences, University of Southampton, University Road, Southampton, SO17 1BJ, United Kingdom
| |
Collapse
|
37
|
Abstract
Following the publication of the final paper in a planned series of four studies estimating the economic returns from biomedical and health research, we reflect on what we have learnt from these types of assessment.
Collapse
Affiliation(s)
- Jonathan Grant
- Policy Institute at King’s, King’s College London, London, UK
| | - Martin J Buxton
- Health Economic Research Group, Brunel University London, Uxbridge, UK
| |
Collapse
|
38
|
Ramanathan S, Reeves P, Deeming S, Bernhardt J, Nilsson M, Cadilhac DA, Walker FR, Carey L, Middleton S, Lynch E, Searles A. Implementing a protocol for a research impact assessment of the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery. Health Res Policy Syst 2018; 16:71. [PMID: 30068358 PMCID: PMC6090618 DOI: 10.1186/s12961-018-0349-2] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2017] [Accepted: 07/13/2018] [Indexed: 01/22/2023] Open
Abstract
Background There is growing recognition that the wider benefits of research (economic, social and health impacts) should be assessed and valued alongside traditional research performance metrics such as peer-reviewed papers. Translation of findings into policy and practice needs to accelerate and pathways to impact need to be better understood. This research protocol outlines a mixed methods study to apply the Framework to Assess the Impact from Translational health research (FAIT) to the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery (CRE-Stroke). FAIT is purpose-designed to encourage research translation and assess research impact but lacks validation. Methods/Design Phase 1 involves application of the FAIT-modified programme logic model to each CRE-Stroke research stream including identifying process, output and impact metrics, as well as end users of the research. A scoping review will inform potential impacts anticipated from CRE-Stroke. In Phase 2, audit and feedback on achievements against plans will track and encourage research translation. Logic models will be updated to account for changes in the research pathways over time. In Phase 3, three proven methods for measuring research impact – Payback, economic assessment and narratives – will be applied to each research stream and the data triangulated and reported in Phase 4. The feasibility of applying FAIT will also be assessed as part of Phase 3. Discussion Use of prospective, comprehensive research impact frameworks for large interdisciplinary programmes of research is rare. FAIT’s application to CRE-Stroke will provide opportunity for the impact of CRE-Stroke to be assessed and a range of impacts beyond standard academic achievements to be reliably reported. The feasibility of FAIT’s application will also be assessed and, if necessary, refined. The usefulness of FAIT for encouraging research translation will also be described and may prove useful for other programmes looking to implement a research impact framework.
Collapse
Affiliation(s)
- Shanthi Ramanathan
- Health Research Economics, Hunter Medical Research Insitute, Locked Bag 1, New Lambton Heights, NSW, 2305, Australia. .,School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Callaghan, NSW, Australia.
| | - Penny Reeves
- Health Research Economics, Hunter Medical Research Insitute, Locked Bag 1, New Lambton Heights, NSW, 2305, Australia.,School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Callaghan, NSW, Australia
| | - Simon Deeming
- Health Research Economics, Hunter Medical Research Insitute, Locked Bag 1, New Lambton Heights, NSW, 2305, Australia.,School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Callaghan, NSW, Australia
| | - Julie Bernhardt
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Melbourne, VIC, Australia.,NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
| | - Michael Nilsson
- Health Research Economics, Hunter Medical Research Insitute, Locked Bag 1, New Lambton Heights, NSW, 2305, Australia.,School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Callaghan, NSW, Australia.,NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
| | - Dominique A Cadilhac
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Melbourne, VIC, Australia.,NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia.,Stroke and Ageing Research, School of Clinical Sciences at Monash Health, Monash University, Clayton, VIC, Australia
| | - Frederick Rohan Walker
- School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Callaghan, NSW, Australia.,NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia
| | - Leeanne Carey
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Melbourne, VIC, Australia.,NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia.,School of Allied Health, College of Science, Health and Engineering, Department of Community and Clinical Allied Health, LaTrobe University, Bundoora, VIC, Australia
| | - Sandy Middleton
- NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia.,Nursing Research Institute, St Vincent's Health Australia (Sydney) and Australian Catholic University, Darlinghurst, NSW, Australia
| | - Elizabeth Lynch
- The Florey Institute of Neuroscience and Mental Health, Melbourne University, Melbourne, VIC, Australia.,NHMRC Centre of Research Excellence in Stroke Rehabilitation and Brain Recovery, Heidelberg, VIC, Australia.,Adelaide Nursing School, Faculty of Health and Medical Sciences, University of Adelaide, North Terrace, SA, Australia
| | - Andrew Searles
- Health Research Economics, Hunter Medical Research Insitute, Locked Bag 1, New Lambton Heights, NSW, 2305, Australia.,School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Callaghan, NSW, Australia
| |
Collapse
|
39
|
Newson R, Rychetnik L, King L, Milat A, Bauman A. Does citation matter? Research citation in policy documents as an indicator of research impact - an Australian obesity policy case-study. Health Res Policy Syst 2018; 16:55. [PMID: 29950167 PMCID: PMC6022713 DOI: 10.1186/s12961-018-0326-9] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2017] [Accepted: 05/09/2018] [Indexed: 11/16/2022] Open
Abstract
Background Citation of research in policy documents has been suggested as an indicator of the potential longer-term impacts of research. We investigated the use of research citations in childhood obesity prevention policy documents from New South Wales (NSW), Australia, considering the feasibility and value of using research citation as a proxy measure of research impact. Methods We examined childhood obesity policy documents produced between 2000 and 2015, extracting childhood obesity-related references and coding these according to reference type, geographical origin and type of research. A content analysis of the policy documents examined where and how research was cited in the documents and the context of citation for individual research publications. Results Over a quarter (28%) of the policy documents (n = 86) were not publicly available, almost two-thirds (63%) contained references, half (47%) cited obesity-related research and over a third (41%) of those containing references used unorthodox referencing styles, making reference extraction laborious. No patterns, in terms of the types of documents more likely to cite research, were observed and the number of obesity research publications cited per document was highly variable. In total, 263 peer-reviewed and 94 non-peer-reviewed obesity research publications were cited. Research was most commonly cited to support a policy argument or choice of solution. However, it was not always possible to determine how or why individual publications were cited or whether the cited research itself had influenced the policy process. Content analysis identified circumstances where research was mentioned or considered, but not directly cited. Conclusions Citation of research in policy documents in this case did not always provide evidence that the cited research had influenced the policy process, only that it was accessible and relevant to the content of the policy document. Research citation across these public health policy documents varied greatly and is unlikely to be an accurate reflection of actual research use by the policy agencies involved. The links between citation and impact may be more easily drawn in specific policy areas or types of documents (e.g. clinical guidelines), where research appraisal feeds directly into policy recommendations.
Collapse
Affiliation(s)
- Robyn Newson
- Sydney School of Public Health, Charles Perkins Centre D17, Level 6 Hub, The University of Sydney, Sydney, NSW, 2006, Australia.
| | - Lucie Rychetnik
- School of Medicine Sydney, University of Notre Dame Australia, 160 Oxford St, Darlinghurst, Australia
| | - Lesley King
- Sydney School of Public Health, Charles Perkins Centre D17, Level 6 Hub, The University of Sydney, Sydney, NSW, 2006, Australia
| | - Andrew Milat
- Sydney School of Public Health, Charles Perkins Centre D17, Level 6 Hub, The University of Sydney, Sydney, NSW, 2006, Australia
| | - Adrian Bauman
- Sydney School of Public Health, Charles Perkins Centre D17, Level 6 Hub, The University of Sydney, Sydney, NSW, 2006, Australia
| |
Collapse
|
40
|
Newson R, King L, Rychetnik L, Milat A, Bauman A. Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research. Health Res Policy Syst 2018; 16:54. [PMID: 29940961 PMCID: PMC6019310 DOI: 10.1186/s12961-018-0310-4] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2017] [Accepted: 04/02/2018] [Indexed: 11/26/2022] Open
Abstract
Background Measuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research. Methods Keyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework. Results We identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilised forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change. Conclusions The Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence. Electronic supplementary material The online version of this article (10.1186/s12961-018-0310-4) contains supplementary material, which is available to authorized users.
Collapse
Affiliation(s)
- Robyn Newson
- Sydney School of Public Health, The University of Sydney, Charles Perkins Centre D17, Level 6 Hub, Sydney, NSW, 2006, Australia.
| | - Lesley King
- Sydney School of Public Health, The University of Sydney, Charles Perkins Centre D17, Level 6 Hub, Sydney, NSW, 2006, Australia
| | - Lucie Rychetnik
- School of Medicine Sydney, University of Notre Dame Australia, 160 Oxford St, Darlinghurst, 2010, Australia
| | - Andrew Milat
- Sydney School of Public Health, The University of Sydney, Charles Perkins Centre D17, Level 6 Hub, Sydney, NSW, 2006, Australia
| | - Adrian Bauman
- Sydney School of Public Health, The University of Sydney, Charles Perkins Centre D17, Level 6 Hub, Sydney, NSW, 2006, Australia
| |
Collapse
|
41
|
Bowden JA, Sargent N, Wesselingh S, Size L, Donovan C, Miller CL. Measuring research impact: a large cancer research funding programme in Australia. Health Res Policy Syst 2018; 16:39. [PMID: 29743088 PMCID: PMC5944042 DOI: 10.1186/s12961-018-0311-3] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2017] [Accepted: 04/06/2018] [Indexed: 11/23/2022] Open
Abstract
BACKGROUND Measuring research impact is of critical interest to philanthropic and government funding agencies interested in ensuring that the research they fund is both scientifically excellent and has meaningful impact into health and other outcomes. The Beat Cancer Project (BCP) is a AUD $34 m cancer research funding scheme that commenced in 2011. It was initiated by an Australian charity (Cancer Council SA), and supported by the South Australian Government and the state's major universities. METHODS This study applied Buxton and Hanney's Payback Framework to assess research impact generated from the BCP after 3 years of funding. Data sources were an audit of peer-reviewed publications from January 2011 to September 2014 from Web of Knowledge and a self-report survey of investigators awarded BCP research funding during its first 3 years of implementation (2011-2013). Of the 104 surveys, 92 (88%) were completed. RESULTS The BCP performed well across all five categories of the Payback Framework. In terms of knowledge production, 1257 peer-reviewed publications were generated and the mean impact factor of publishing journals increased annually. There were many benefits to future research with 21 respondents (23%) reporting career advancement, and 110 higher degrees obtained or expected (including 84 PhDs). Overall, 52% of funded projects generated tools for future research. The funded research attracted substantial further income yielding a very high rate of leverage. For every AUD $1 that the cancer charity invested, the BCP gained an additional AUD $6.06. Five projects (5%) had informed policy and 5 (5%) informed product development, with an additional 31 (34%) and 35 (38%) projects, respectively, anticipating doing so. In terms of health and sector and broader economic benefits, 8 (9%) projects had influenced practice or behaviour of health staff and 32 (34%) would reportedly to do so in the future. CONCLUSIONS Research impact was a priority of charity and government funders and led to a deliberate funding strategy. Emphasising research impact while maintaining rigorous, competitive processes can achieve the joint objectives of excellence in research, yielding good research impact and a high rate of leverage for philanthropic and public investment, as indicated by these early results.
Collapse
Affiliation(s)
| | - Nicole Sargent
- South Australian Health and Medical Research Institute, Adelaide, Australia
| | - Steve Wesselingh
- South Australian Health and Medical Research Institute, Adelaide, Australia
| | - Lincoln Size
- Cancer Council South Australia, Adelaide, Australia
| | - Claire Donovan
- Division of Health Sciences, Brunel University London, Uxbridge, UK
| | - Caroline L. Miller
- South Australian Health and Medical Research Institute, Adelaide, Australia
- University of Adelaide, School of Public Health, Adelaide, Australia
| |
Collapse
|
42
|
Glover M, Montague E, Pollitt A, Guthrie S, Hanney S, Buxton M, Grant J. Estimating the returns to United Kingdom publicly funded musculoskeletal disease research in terms of net value of improved health outcomes. Health Res Policy Syst 2018; 16:1. [PMID: 29316935 PMCID: PMC5761203 DOI: 10.1186/s12961-017-0276-7] [Citation(s) in RCA: 52] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2017] [Accepted: 12/12/2017] [Indexed: 01/05/2023] Open
Abstract
BACKGROUND Building on an approach applied to cardiovascular and cancer research, we estimated the economic returns from United Kingdom public- and charitable-funded musculoskeletal disease (MSD) research that arise from the net value of the improved health outcomes in the United Kingdom. METHODS To calculate the economic returns from MSD-related research in the United Kingdom, we estimated (1) the public and charitable expenditure on MSD-related research in the United Kingdom between 1970 and 2013; (2) the net monetary benefit (NMB), derived from the health benefit in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of £25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1994 to 2013; (3) the proportion of NMB attributable to United Kingdom research; and (4) the elapsed time between research funding and health gain. The data collected from these four key elements were used to estimate the internal rate of return (IRR) from MSD-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using a one-way sensitivity analysis. RESULTS Expressed in 2013 prices, total expenditure on MSD-related research from 1970 to 2013 was £3.5 billion, and for the period used to estimate the rate of return, 1978-1997, was £1.4 billion. Over the period 1994-2013 the key interventions analysed produced 871,000 QALYs with a NMB of £16 billion, allowing for the net NHS costs resulting from them and valuing a QALY at £25,000. The proportion of benefit attributable to United Kingdom research was 30% and the elapsed time between funding and impact of MSD treatments was 16 years. Our best estimate of the IRR from MSD-related research was 7%, which is similar to the 9% for CVD and 10% for cancer research. CONCLUSIONS Our estimate of the IRR from the net health gain to public and charitable funding of MSD-related research in the United Kingdom is substantial, and justifies the research investments made between 1978 and 1997. We also demonstrated the applicability of the approach previously used in assessing the returns from cardiovascular and cancer research. Inevitably, with a study of this kind, there are a number of important assumptions and caveats that we highlight, and these can inform future research.
Collapse
Affiliation(s)
- Matthew Glover
- Health Economics Research Group, Brunel University London, Uxbridge, United Kingdom
| | - Erin Montague
- Policy Institute at King’s, King’s College London, Virginia Woolf Building, 22 Kingsway, London, WC2B 6LE United Kingdom
| | - Alexandra Pollitt
- Policy Institute at King’s, King’s College London, Virginia Woolf Building, 22 Kingsway, London, WC2B 6LE United Kingdom
| | | | - Stephen Hanney
- Health Economics Research Group, Brunel University London, Uxbridge, United Kingdom
| | - Martin Buxton
- Health Economics Research Group, Brunel University London, Uxbridge, United Kingdom
| | - Jonathan Grant
- Policy Institute at King’s, King’s College London, Virginia Woolf Building, 22 Kingsway, London, WC2B 6LE United Kingdom
| |
Collapse
|
43
|
Ramanathan S, Reeves P, Deeming S, Bailie RS, Bailie J, Bainbridge R, Cunningham F, Doran C, McPhail Bell K, Searles A. Encouraging translation and assessing impact of the Centre for Research Excellence in Integrated Quality Improvement: rationale and protocol for a research impact assessment. BMJ Open 2017; 7:e018572. [PMID: 29208619 PMCID: PMC5719272 DOI: 10.1136/bmjopen-2017-018572] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/23/2023] Open
Abstract
INTRODUCTION There is growing recognition among health researchers and funders that the wider benefits of research such as economic, social and health impacts ought to be assessed and valued alongside academic outputs such as peer-reviewed papers. Research translation needs to increase and the pathways to impact ought to be more transparent. These processes are particularly pertinent to the Indigenous health sector given continued concerns that Indigenous communities are over-researched with little corresponding improvement in health outcomes. This paper describes the research protocol of a mixed methods study to apply FAIT (Framework to Assess the Impact from Translational health research) to the Centre for Research Excellence in Integrated Quality Improvement (CRE-IQI). FAIT will be applied to five selected CRE-IQI Flagship projects to encourage research translation and assess the wider impact of that research. METHODS AND ANALYSIS Phase I will develop a modified programme logic model for each Flagship project including identifying process, output and impact metrics so progress can be monitored. A scoping review will inform potential benefits. In phase II, programme logic models will be updated to account for changes in the research pathways over time. Audit and feedback will be used to encourage research translation and collect evidence of achievement of any process, output and interim impacts. In phase III, three proven methodologies for measuring research impact-Payback, economic assessment and narratives-will be applied. Data on the application of FAIT will be collected and analysed to inform and improve FAIT's performance. ETHICS AND DISSEMINATION This study is funded by a nationally competitive grant (ID 1078927) from the Australian National Health and Medical Research Council. Ethics approval was obtained from the University of Newcastle's Human Research Ethics Committee (ID: H-2017-0026). The results from the study will be presented in several peer-reviewed publications, through conference presentations and via social media.
Collapse
Affiliation(s)
- Shanthi Ramanathan
- Health Research Economics, Hunter Medical Research Institute, Newcastle, New South Wales, Australia
- School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Newcastle, New South Wales, Australia
| | - Penny Reeves
- Health Research Economics, Hunter Medical Research Institute, Newcastle, New South Wales, Australia
- School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Newcastle, New South Wales, Australia
| | - Simon Deeming
- Health Research Economics, Hunter Medical Research Institute, Newcastle, New South Wales, Australia
- School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Newcastle, New South Wales, Australia
| | - Ross Stewart Bailie
- The University of Sydney, University Centre for Rural Health, Lismore, New South Wales, Australia
| | - Jodie Bailie
- The University of Sydney, University Centre for Rural Health, Lismore, New South Wales, Australia
| | - Roxanne Bainbridge
- Centre for Indigenous Health Equity Research, School of Health Medical and Applied Sciences, Central Queensland University, Brisbane, Queensland, Australia
- The Cairns Institute, James Cook University, Cairns, Queensland, Australia
| | - Frances Cunningham
- Menzies School of Health Research, Charles Darwin University, Brisbane, Queensland, Australia
| | - Christopher Doran
- Centre for Indigenous Health Equity Research, School of Health Medical and Applied Sciences, Central Queensland University, Brisbane, Queensland, Australia
| | - Karen McPhail Bell
- The University of Sydney, University Centre for Rural Health, Lismore, New South Wales, Australia
| | - Andrew Searles
- Health Research Economics, Hunter Medical Research Institute, Newcastle, New South Wales, Australia
- School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, Newcastle, New South Wales, Australia
| |
Collapse
|
44
|
Bertele' V, Gerardi C, Garattini S. Helping ethics committees separate the wheat from the chaff. Eur J Intern Med 2017. [PMID: 28629693 DOI: 10.1016/j.ejim.2017.06.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
Affiliation(s)
- Vittorio Bertele'
- IRCCS-Istituto di Ricerche Farmacologiche Mario Negri, Milan, Italy.
| | - Chiara Gerardi
- IRCCS-Istituto di Ricerche Farmacologiche Mario Negri, Milan, Italy
| | - Silvio Garattini
- IRCCS-Istituto di Ricerche Farmacologiche Mario Negri, Milan, Italy
| |
Collapse
|
45
|
Payne H. Transferring research from a university to the United Kingdom National Health Service: the implications for impact. Health Res Policy Syst 2017. [PMID: 28623933 PMCID: PMC5473996 DOI: 10.1186/s12961-017-0219-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
The aim of this article is to inform readers of the author’s reflections on the experience of transferring university-based research into the commercial sector, and of the processes and strategies employed when preparing for impact in so doing. Concepts for the transfer are illustrated by the author’s reflection on aspects that arose during the birthing and subsequent start-up of a university spin-off, Pathways2Wellbeing, a form of reflection-on-action. This is the vehicle for the adaption required to transfer research into the delivery of a specialised clinic in the United Kingdom National Health Service for people with medically unexplained, persistent, bodily symptoms such as fibromyalgia, chronic fatigue and chronic pain. It is hoped that the article will provide readers with an insight into how knowledge transfer can take place through engagement with stakeholders to create an exchange of knowledges to result in impact on health service policy for service users, despite the challenges, and the enablers that facilitated this process. The reflections on the process of knowledge transfer and the implications for impact are underpinned by relevant theory.
Collapse
Affiliation(s)
- Helen Payne
- University of Hertfordshire, School of Education, De Havilland Campus, Hatfield Business Park, Hatfield, Herts, AL10 0NZ, United Kingdom.
| |
Collapse
|
46
|
Greenhalgh T, Jackson C, Shaw S, Janamian T. Achieving Research Impact Through Co-creation in Community-Based Health Services: Literature Review and Case Study. Milbank Q 2017; 94:392-429. [PMID: 27265562 PMCID: PMC4911728 DOI: 10.1111/1468-0009.12197] [Citation(s) in RCA: 469] [Impact Index Per Article: 67.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022] Open
Abstract
Policy Points: Co‐creation—collaborative knowledge generation by academics working
alongside other stakeholders—is an increasingly popular approach to
aligning research and service development. It has potential for “moving beyond the ivory towers” to deliver
significant societal impact via dynamic, locally adaptive
community‐academic partnerships. Principles of successful co‐creation include a systems perspective,
a creative approach to research focused on improving human
experience, and careful attention to governance and process. If these principles are not followed, co‐creation efforts may
fail.
Context Co‐creation—collaborative knowledge generation by academics working
alongside other stakeholders—reflects a “Mode 2” relationship (knowledge
production rather than knowledge translation) between universities and society.
Co‐creation is widely believed to increase research impact. Methods We undertook a narrative review of different models of co‐creation
relevant to community‐based health services. We contrasted their diverse
disciplinary roots and highlighted their common philosophical assumptions,
principles of success, and explanations for failures. We applied these to an
empirical case study of a community‐based research‐service partnership led by the
Centre of Research Excellence in Quality and Safety in Integrated
Primary‐Secondary Care at the University of Queensland, Australia. Findings Co‐creation emerged independently in several fields, including
business studies (“value co‐creation”), design science (“experience‐based
co‐design”), computer science (“technology co‐design”), and community development
(“participatory research”). These diverse models share some common features, which
were also evident in the case study. Key success principles included (1) a systems
perspective (assuming emergence, local adaptation, and nonlinearity); (2) the
framing of research as a creative enterprise with human experience at its core;
and (3) an emphasis on process (the framing of the program, the nature of
relationships, and governance and facilitation arrangements, especially the style
of leadership and how conflict is managed). In both the literature review and the
case study, co‐creation “failures” could often be tracked back to abandoning (or
never adopting) these principles. All co‐creation models made strong claims for
significant and sustainable societal impacts as a result of the adaptive and
developmental research process; these were illustrated in the case study. Conclusions Co‐creation models have high potential for societal impact but
depend critically on key success principles. To capture the nonlinear chains of
causation in the co‐creation pathway, impact metrics must reflect the dynamic
nature and complex interdependencies of health research systems and address
processes as well as outcomes.
Collapse
Affiliation(s)
- Trisha Greenhalgh
- Nuffield Department of Primary Care Health Sciences, University of Oxford
| | - Claire Jackson
- Discipline of General Practice, School of Medicine, University of Queensland
| | - Sara Shaw
- Nuffield Department of Primary Care Health Sciences, University of Oxford
| | - Tina Janamian
- Discipline of General Practice, School of Medicine, University of Queensland
| |
Collapse
|
47
|
Hanney S, Greenhalgh T, Blatch-Jones A, Glover M, Raftery J. The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews. Health Res Policy Syst 2017; 15:26. [PMID: 28351391 PMCID: PMC5371238 DOI: 10.1186/s12961-017-0191-y] [Citation(s) in RCA: 26] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2016] [Accepted: 03/12/2017] [Indexed: 11/21/2022] Open
Abstract
BACKGROUND We sought to analyse the impacts found, and the methods used, in a series of assessments of programmes and portfolios of health research consisting of multiple projects. METHODS We analysed a sample of 36 impact studies of multi-project research programmes, selected from a wider sample of impact studies included in two narrative systematic reviews published in 2007 and 2016. We included impact studies in which the individual projects in a programme had been assessed for wider impact, especially on policy or practice, and where findings had been described in such a way that allowed them to be collated and compared. RESULTS Included programmes were highly diverse in terms of location (11 different countries plus two multi-country ones), number of component projects (8 to 178), nature of the programme, research field, mode of funding, time between completion and impact assessment, methods used to assess impact, and level of impact identified. Thirty-one studies reported on policy impact, 17 on clinician behaviour or informing clinical practice, three on a combined category such as policy and clinician impact, and 12 on wider elements of impact (health gain, patient benefit, improved care or other benefits to the healthcare system). In those multi-programme projects that assessed the respective categories, the percentage of projects that reported some impact was policy 35% (range 5-100%), practice 32% (10-69%), combined category 64% (60-67%), and health gain/health services 27% (6-48%). Variations in levels of impact achieved partly reflected differences in the types of programme, levels of collaboration with users, and methods and timing of impact assessment. Most commonly, principal investigators were surveyed; some studies involved desk research and some interviews with investigators and/or stakeholders. Most studies used a conceptual framework such as the Payback Framework. One study attempted to assess the monetary value of a research programme's health gain. CONCLUSION The widespread impact reported for some multi-project programmes, including needs-led and collaborative ones, could potentially be used to promote further research funding. Moves towards greater standardisation of assessment methods could address existing inconsistencies and better inform strategic decisions about research investment; however, unresolved issues about such moves remain.
Collapse
Affiliation(s)
- Steve Hanney
- Health Economics Research Group (HERG), Institute of Environment, Health and Societies, Brunel University London, London, UB8 3PH United Kingdom
| | - Trisha Greenhalgh
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, OX2 6GG United Kingdom
| | - Amanda Blatch-Jones
- Wessex Institute, Faculty of Medicine, University of Southampton, Southampton, SO16 7NS United Kingdom
| | - Matthew Glover
- Health Economics Research Group (HERG), Institute of Environment, Health and Societies, Brunel University London, London, UB8 3PH United Kingdom
| | - James Raftery
- Primary Care and Population Sciences, Faculty of Medicine, University of Southampton, Southampton General Hospital, Southampton, SO16 6YD United Kingdom
| |
Collapse
|
48
|
Abstract
Impact occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base. Its mechanisms are complex and reflect the multiple ways in which knowledge is generated and utilised. Much progress has been made in measuring both the outcomes of research and the processes and activities through which these are achieved, though the measurement of impact is not without its critics. We review the strengths and limitations of six established approaches (Payback, Research Impact Framework, Canadian Academy of Health Sciences, monetisation, societal impact assessment, UK Research Excellence Framework) plus recently developed and largely untested ones (including metrics and electronic databases). We conclude that (1) different approaches to impact assessment are appropriate in different circumstances; (2) the most robust and sophisticated approaches are labour-intensive and not always feasible or affordable; (3) whilst most metrics tend to capture direct and proximate impacts, more indirect and diffuse elements of the research-impact link can and should be measured; and (4) research on research impact is a rapidly developing field with new methodologies on the horizon.
Collapse
Affiliation(s)
- Trisha Greenhalgh
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Radcliffe Primary Care Building, Woodstock Rd, Oxford, OX2 6GG, UK.
| | - James Raftery
- Primary Care and Population Sciences, Faculty of Medicine, University of Southampton, Southampton General Hospital, Southampton, SO16 6YD, UK
| | - Steve Hanney
- Health Economics Research Group (HERG), Institute of Environment, Health and Societies, Brunel University London, ᅟ, UB8 3PH, UK
| | - Matthew Glover
- Health Economics Research Group (HERG), Institute of Environment, Health and Societies, Brunel University London, ᅟ, UB8 3PH, UK
| |
Collapse
|