The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews

How to strengthen societal impact of research and innovation? Lessons learned from an explanatory research-on-research study on participatory knowledge infrastructures funded by the Netherlands Organization for Health Research and Development

BACKGROUND

Scientific research and innovation can generate societal impact via different pathways. Productive interactions, such as collaboration between researchers and relevant stakeholders, play an important role and have increasingly gained interest of health funders around the globe. What works, how and why in research partnerships to generate societal impact in terms of knowledge utilisation is still not well-known. To explore these issues, the Netherlands Organization for Health Research and Development (ZonMw) initiated an exploratory research-on-research study with a focus on participatory knowledge infrastructures (PKIs) that they fund in the field of public health and healthcare. PKIs are sustainable infrastructures in which knowledge production, dissemination and utilisation takes place via committed collaboration between researchers and stakeholders from policy, practice and/or education. Examples are learning networks, academic collaborative centres, care networks and living labs. The aim of the study was twofold: to gain insights in what constitutes effective collaboration in PKIs; and to learn and improve the research governance, particularly of ZonMw as part of their dissemination and implementation activities.

METHODS

During 2020-2022, we conducted a literature review on long-term research partnerships, analysed available documentation of twenty ZonMw-funded PKIs, surveyed participants of the 2021 European Implementation Event, interviewed steering committee members, organized a Group Decision Room with lecturers, and validated the findings with key experts.

RESULTS

We identified eight mechanisms ('how and why') that are conditional for effective collaboration in PKIs: transdisciplinary collaboration; defining a shared ambition; doing justice to everyone's interests; investing in personal relationships; a professional organisation or structure; a meaningful collaborative process; mutual trust, sufficient time for and continuity of collaboration. Several factors ('what') may hinder (e.g., lack of ownership or structural funding) or facilitate (e.g., stakeholder commitment, embeddedness in an organisation or policy) effective collaboration in research partnerships.

CONCLUSION

To use the study results in policy, practice, education, and/or (further) research, cultural and behavioural change of all stakeholders is needed. To facilitate this, we provide recommendations for funding organisations, particularly ZonMw and its partners within the relevant knowledge ecosystem. It is meant as a roadmap towards the realisation and demonstration of societal impact of (health) research and innovation in the upcoming years.

Advancing health through evidence assisted decisions with health policy and systems research program: a qualitative evaluation of a national health research grant management process in the Philippines

BACKGROUND

Health policy and systems research (HPSR) has influenced Philippine policies, including tobacco control, mental health, and COVID-19. The Department of Health (DOH) Philippines and Philippine Council for Health Research and Development (PCHRD) launched the Advancing Health through Evidence-Assisted Decisions (AHEAD) with HPSR program in 2017, aiming to build a community of researchers and decision-makers committed to evidence production and utilization. Research systems employ grant management processes for transparency and accountability in research funding, preventing waste, fraud, and misuse of funds.

METHODS

This study evaluated AHEAD-HPSR's grant management using surveys, interviews, and focus groups to document (1) grant administration processes implemented by DOH and PCHRD, and (2) experiences of grantees, program managers, staff, and policymakers. Data were initially analyzed through the USA Grant Accountability Office's Federal Grant Life Cycle, with new themes created as they emerged. The study identified processes and gaps in the research grant life cycle stages: design/redesign, pre-award, award, implementation, closeout, and research dissemination and utilization.

RESULTS

Identification of research areas for the grant are identified using national and departmental research priorities. While Calls for Proposals are posted publicly, researchers that have previously worked with policymakers are contacted directly to submit proposals. The evaluation found that research is delayed by bureaucracies in grant administration, particularly in financial reporting and ethics review processes. Complying with the terminal financial report was identified as the most challenging part of the grant process due to immense auditing requirements. Grantees recommend the simplification of bureaucracy for fund release to enable them to focus on research work.

CONCLUSION

This study contributes to the limited literature on health research grant management in developing countries. Valuable information and recommendations were contributed by stakeholders in this evaluation. These are manifestations of a continuing interest and desire to make health policy and systems research in the Philippines more robust and relevant. It is imperative for the program to continually evolve and build systems most applicable to its multidisciplinary context.

Identifying Student Research Project Impact Using the Buxton and Hanney Payback Framework

Objectives. To determine whether evidence of the impact of student quality improvement projects and research projects on practice sites and the community can be identified using the Buxton and Hanney Payback Framework (BHPF).Methods. The BHPF was used to identify the broader impact of quality improvement projects and research projects conducted by the Doctor of Pharmacy (PharmD) class of 2020. The BHPF includes five domains of community impact: knowledge production, benefits to health or the health sector, benefits to future research, economic benefits, and policy and product development. Data were collected by having project preceptors complete a questionnaire and by reviewing student project posters. Data were analyzed by calculating frequencies and percentages for each domain.Results. Projects (N=73) were completed by 107 pharmacy students at health-system sites, community sites, academic sites, and other sites, and most often involved clinical care and pharmacy services (49%). Thirty-three preceptors (55%) responded to the questionnaire, and 73 project posters were reviewed. The most frequently identified impact types were knowledge production (n=43 for questionnaire, n=24 for posters) and health/health sector benefits (n=46 for questionnaire, n=8 for posters). Less frequently identified were economic benefits (total n=19), benefits to future research (total n=13), and policy and product development (total n=10).Conclusions. This study provides evidence that the impact of PharmD student quality improvement and research projects on practice sites and communities can be identified using the BHPF framework, and this impact extends beyond the usual academic outcomes of poster presentations and publications to include benefits related to improving quality of services, improving workflow, and providing opportunity for personal development.

"We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research": A Research Impact Evaluation at a Regional Australian Hospital and Health Service

BACKGROUND

Research is central to high functioning health services alongside clinical care and health professional training. The impact of embedded research includes delivery of high-quality care and improved patient outcomes. Evaluations of research impact help health service leadership ensure investments lead to the greatest healthcare benefits for patients. This study aimed to retrospectively evaluate the impact of research investment from 2008 to 2018 at Townsville Hospital and Health Service (THHS), a regional Hospital and Health Service (HHS) in Queensland, Australia. The evaluation also sought to identify contextual conditions that enable or hinder intended impacts.

METHODS

A mixed-methods realist-informed evaluation was conducted using documentation, interviews with 15 staff and available databases to identify and measure research investments, impacts and contextual conditions influencing impact outcomes.

RESULTS

Between 2008 and 2018, THHS increased resources for research by funding research projects, employing research personnel, building research-enabling facilities, hosting research events, and providing research education and training. Clinical practice, policy and workforce impacts were successful in isolated pockets, championed by individual researchers and facilitated by their policy and community-of-practice networks. However, there was little organisational-level support for continuity of research and implementation into practice and policy. Availability of research supports varied geographically across THHS, and across disciplines.

CONCLUSION

Definitive steps in the development of THHS as a credible and productive research centre and leading hospital research centre in Northern Australia are evident. Continuing investments should address support for the research continuum through to translation and establish ongoing, systematic processes for evaluating research investment and impact.

What funders are doing to assess the impact of their investments in health and biomedical research

As pressures to maximize research funding grow, biomedical research funders are increasingly tasked with demonstrating the long-term and real-world impacts of their funded research investments. Over the past three decades, research impact assessments (RIA) have emerged as an important tool for analysing the impacts of research by incorporating logic models, frameworks and indicators to track measures of knowledge production, capacity-building, development of research products, adoption of research into clinical guidelines and policies, and the realization of health, economic and social benefits. While there are currently several models for RIA within the literature, less attention has been paid to how funders can practically select and implement a RIA model to demonstrate the impacts of their own research portfolios. In this paper, a literature review was performed to understand (1) which research funders have performed RIAs of their research portfolios to date; (2) how funders have designed their assessments, including the models and tools they have used; (3) what challenges to and facilitators of success have funders found when adopting the RIA model to their own portfolio; and (4) who participates in the assessments. Forty-four papers from both published and grey literature were found to meet the review criteria and were examined in detail. There is a growing culture of RIA among funders, and included papers spanned a diverse set of funders from 10 countries or regions. Over half of funders (59.1%) used a framework to conduct their assessment, and a variety of methods for collecting impact data were reported. Issues of methodological rigour were observed across studies in the review, and this was related to numerous challenges funders faced in designing timely RIAs with quality impact data. Over a third of articles (36.4%) included input from stakeholders, yet only one article reported surveying patients and members of the public as part of the assessment. To advance RIA among funders, we offer several recommendations for increasing the methodological rigour of RIAs and suggestions for future research, and call for a careful reflection of the voices needed in an impact assessment to ensure that RIAs are having a meaningful impact on patients and the public.

How successful was the use of a community of practice for the implementation of evidence-based practices for heart failure within the United States Department of Veterans Affairs: Insights from a formative evaluation

Background

Communities of Practice (CoPs) are a promising approach to facilitate the implementation of evidence-based practices (EBPs) to improve care for chronic conditions like heart failure (HF). CoPs involve a complex process of acquiring and converting both explicit and tacit knowledge into clinical activities. This study describes the conceptualization, creation, capacity-building and dissemination of a CoP sustained over 9 years, and evaluates its value and impact on EBP.

Methods

In July 2006, a CoP called the Heart Failure Provider Network (HF Network) was established within the United States Department of Veterans Affairs (VA) with the overarching goal of improving the quality of care for HF patients. We assessed (formative) the HF Network in terms of its various activities (inputs) and proximal impacts (mediators) at the individual level, and its distal impacts (outcomes) at the site level including implementation of new/improved EBPs at the systemwide level.

Results

The HF Network membership grew steadily over the 9 years. The CoP has involved a total of 1341 multidisciplinary and multilevel members at all 144 VA Health Care Systems (sites). Most members were practising clinicians (n = 891, 66.4%), followed by administrators (n = 342, 25.5%), researchers (n = 70, 5.2%) and others (n = 38, 2.8%). Participation was assessed to be “active” for 70.6% versus “passive” for 29.4% of members. The distribution of active members (clinicians 64.7%, administrators 21.6%) was similar to the distribution of overall membership.

Conclusions

Survey respondents perceived the HF Network as useful in terms of its varied activities and resources relevant for patient care. Strong evidence shows that these members, particularly those who considered themselves influential in improving quality of care, noted multiple benefits of membership, which included confirmation of their own clinical practices, evidence-based changes to their practice and help in understanding facilitators and barriers in setting up or running HF clinics and other programmes. Such CoPs have strong impacts on the quality of care being delivered for both mandated and non-mandated initiatives.

Research can be integrated into public health policy-making: global lessons for and from Spanish economic evaluations

WHO promotes the use of research in policy-making to drive improvements in health, including in achieving Sustainable Development Goals such as tobacco control. The European Union’s new €95 billion Horizon Europe research framework programme parallels these aims, and also includes commitments to fund economic evaluations. However, researchers often express frustration at the perceived lack of attention to scientific evidence during policy-making. For example, some researchers claim that evidence regarding the return on investment from optimal implementation of evidence-based policies is frequently overlooked. An increasingly large body of literature acknowledges inevitable barriers to research use, but also analyses facilitators encouraging such use. This opinion piece describes how some research is integrated into policy-making. It highlights two recent reviews. One examines impact assessments of 36 multi-project research programmes and identifies three characteristics of projects more likely to influence policy-making. These include a focus on healthcare system needs, engagement of stakeholders, and research conducted for organizations supported by structures to receive and use evidence. The second review suggests that such characteristics are likely to occur as part of a comprehensive national health research system strategy, especially one integrated into the healthcare system. We also describe two policy-informing economic evaluations conducted in Spain. These examined the most cost-effective package of evidence-based tobacco control interventions and the cost-effectiveness of different strategies to increase screening coverage for cervical cancer. Both projects focused on issues of healthcare concern and involved considerable stakeholder engagement. The Spanish examples reinforce some lessons from the global literature and, therefore, could help demonstrate to authorities in Spain the value of developing comprehensive health research systems, possibly following the interfaces and receptor model. The aim of this would be to integrate needs assessment and stakeholder engagement with structures spanning the research and health systems. In such structures, economic evaluation evidence could be collated, analysed by experts in relation to healthcare needs, and fed into both policy-making as appropriate, and future research calls. The increasingly large local and global evidence base on research utilization could inform detailed implementation of this approach once accepted as politically desirable. Given the COVID-19 pandemic, increasing the cost-effectiveness of healthcare systems and return on investment of public health interventions becomes even more important.

A large National Institute for Health Research (NIHR) Biomedical Research Centre facilitates impactful cross-disciplinary and collaborative translational research publications and research collaboration networks: a bibliometric evaluation study

Background

The evaluation of translational health research is important for various reasons such as the research impact assessment, research funding allocation, accountability, and strategic research policy formulation. The purpose of this study was to evaluate the research productivity, strength and diversity of research collaboration networks and impact of research supported by a large biomedical research centre in the United Kingdom (UK).

Methods

Bibliometric analysis of research publications by translational researchers affiliated with the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC) from April 2012 to March 2017.

Results

Analysis included 2377 translational research publications that were published during the second 5-year funding period of the NIHR Oxford BRC. Author details were available for 99.75% of the publications with DOIs (2359 of 2365 with DOIs), and the number of authors per publication was median 9 (mean  = 18.03, SD  = 3.63, maximum  = 2467 authors). Author lists also contained many consortia, groups, committees, and teams (n  = 165 in total), with 1238 additional contributors, where membership was reported. The BRC co-authorship i.e., research collaboration network for these publications involved 20,229 nodes (authors, of which 1606 nodes had Oxford affiliations), and approximately 4.3 million edges (authorship linkages). Articles with a valid DOIs (2365 of 2377, 99.5%) were collectively cited more than 155,000 times and the average Field Citation Ratio was median 6.75 (geometric mean  = 7.12) while the average Relative Citation Ratio was median 1.50 (geometric mean  = 1.83) for the analysed publications.

Conclusions

The NIHR Oxford BRC generated substantial translational research publications and facilitated a huge collaborative network of translational researchers working in complex structures and consortia, which shows success across the whole of this BRC funding period. Further research involving continued uptake of unique persistent identifiers and the tracking of other research outputs such as clinical innovations and patents would allow a more detailed understanding of large research enterprises such as NIHR BRCs in the UK.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12967-021-03149-x.

Evaluation of the public health achievements made by projects supported by a federal contract mechanism at the Centers for Disease Control and Prevention (CDC), USA

In 2012, the Centers for Disease Control and Prevention (CDC) established the Achieving Public Health Impact through Research (APHIR) contract mechanism. APHIR provides CDC's Centers, Institute, and Offices (CIOs) a mechanism that supports multiyear, high impact public health research. Awarded projects supported research on a wide range of topics (e.g., cancer surveillance, HIV education programs, development of biological assays, and evaluation of traumatic brain injury prevention programs) and achieved diverse outcomes (e.g., contribution to the body of knowledge in their field, changes in practice and health service delivery, and capacity building). This article describes how existing impact frameworks and a variety of methods and tools (key informant interviews, online survey, bibliometric analysis, Altmetric and document reviews) were used to identify the outcomes achieved by awarded projects. The approach discussed in this paper can be used to evaluate projects that involve a diversity of activities and outcomes.

Systematic review and policy dialogue to determine challenges in evidence-informed health policy-making: findings of the SASHA study

Background

Various interventions have been undertaken in Iran to promote evidence-informed health policy-making (EIHP). Identifying the challenges in EIHP is the first step toward strengthening EIHP in each country through the design of tailored interventions. Therefore, the current study was conducted to synthesize the results of earlier studies and to finalize the list of barriers to EIHP in Iran.

Methods

To identify the barriers to EIHP in Iran, two steps were taken: a systematic review and policy dialogue. To conduct the systematic review, three Iranian databases and PubMed, Health Systems Evidence (HSE), Embase, and Scopus were searched. The reference lists of included papers and documentation from some local organizations were hand-searched. Upon conducting the systematic review, given the significance of stakeholders in clarifying the problem of EIHP, policy dialogue was used to complete the list previously extracted and to do advocacy. Selection criteria for the stakeholders included influential and informed individuals from knowledge-producing, knowledge-utilizing, and knowledge-brokering organizations. Semi-structured interviews were held with three important absent stakeholders.

Results

Challenges specific to Iran that were identified included the lack of integration of the health ministry and the medical universities, lack of ties between health knowledge utilization organizations, failure to establish long-term research plans, neglect of national research needs at the time of recruiting human resources in knowledge-producing organizations, and duplication and lack of coordination in routine data obtained from surveillance systems, disease registration systems, and censuses. It seems that some challenges are common across countries, including neglecting the importance of inter- and intra-disciplinary studies, the capacity of policy-makers and managers to utilize evidence, the criteria for evaluating the performance of policy-makers, managers, and academic members, the absence of long-term programmes in knowledge-utilizing organizations, the rapid replacement of policy-makers and managers, and lack of use of evaluation studies.

Conclusions

In this study, we tried to identify the challenges regarding EIHP in Iran using a systematic review and policy dialogue approach. This is the first step toward determining the best interventions to improve evidence-informed policy-making in each country, because these challenges are contextual and need to be investigated contextually.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12961-021-00717-x.

What was the impact of a participatory research project in Australian Indigenous primary healthcare services? Applying a comprehensive framework for assessing translational health research to Lessons for the Best

OBJECTIVES

To (1) apply the Framework to Assess the Impact from Translational health research (FAIT) to Lessons from the Best to Better the Rest (LFTB), (2) report on impacts from LFTB and (3) assess the feasibility and outcomes from a retrospective application of FAIT.

SETTING

Three Indigenous primary healthcare (PHC) centres in the Northern Territory, Australia; project coordinating centre distributed between Townsville, Darwin and Cairns and the broader LFTB learning community across Australia.

PARTICIPANTS

LFTB research team and one representative from each PHC centre.

PRIMARY AND SECONDARY OUTCOME MEASURES

Impact reported as (1) quantitative metrics within domains of benefit using a modified Payback Framework, (2) a cost-consequence analysis given a return on investment was not appropriate and (3) a narrative incorporating qualitative evidence of impact. Data were gathered through in-depth stakeholder interviews and a review of project documentation, outputs and relevant websites.

RESULTS

LFTB contributed to knowledge advancement in Indigenous PHC service delivery; enhanced existing capacity of health centre staff, researchers and health service users; enhanced supportive networks for quality improvement; and used a strengths-based approach highly valued by health centres. LFTB also leveraged between $A1.4 and $A1.6 million for the subsequent Leveraging Effective Ambulatory Practice (LEAP) Project to apply LFTB learnings to resource development and creation of a learning community to empower striving PHC centres.

CONCLUSION

Retrospective application of FAIT to LFTB, although not ideal, was feasible. Prospective application would have allowed Indigenous community perspectives to be included. Greater appreciation of the full benefit of LFTB including a measure of return on investment will be possible when LEAP is complete. Future assessments of impact need to account for the limitations of fully capturing impact when intermediate/final impacts have not yet been realised and captured.

Using a co-design process to develop an integrated model of care for delivering self-management intervention to multi-morbid COPD people in rural Nepal

BACKGROUND

People with chronic obstructive pulmonary disease (COPD) in Nepal are not receiving adequate support to self-manage their chronic conditions, and primary health care can play a key role in the effective management of these. In this study, we aimed to develop a model of care, using a co-design approach, for delivering evidence-based biomedical and psycho-social care to support self-management for people with multi-morbid COPD in rural Nepal.

METHODS

A co-design approach, guided by the five stages of the design thinking model, was used for this study. Layering on "empathize" and "define" phases, we ideated a model of care that was further refined in a "prototype" stage, which included a series of consultative meetings and a 1-day co-design workshop with stakeholders. This co-design process involved a wide range of stakeholders from Nepal, including people with COPD and their families, community representatives, local government representatives, primary care practitioners, community health workers, policymakers, state-level government representatives and academics.

RESULTS

Through our co-design approach, a model of integrated care for delivering evidence-based biomedical and psycho-social care to support self-management for people with multi-morbid COPD was designed. The integrated model of care included: screening of the community members aged > 40 years or exhibiting symptoms for COPD and management of symptomatic patients within primary health care, establishing referral pathways for severe cases to and from secondary/tertiary-level health care and establishing a community-based support system. It involved specific roles for community health workers, patients and their caregivers and community representatives. It was built on existing services and programmes linking primary health care centres and tertiary-level health facilities.

CONCLUSION

The co-design approach is different from the currently dominant approach of rolling out models of care, which were designed elsewhere with minimal community engagement. In our study, the co-design approach was found to be effective in engaging various stakeholders and in developing a model of care for rural Nepal. This grassroots approach is more likely to be acceptable, effective and sustainable in rural Nepal. Further research is required to test the effectiveness of an integrated model of care in delivering self-management support for people with multi-morbid COPD in rural Nepal.

A research impact model for work and health

Research organizations, governments and funding agencies are increasingly interested in the impact of research beyond academia. While a growing literature describes research impacts in healthcare and health services, little has focused on occupational health and safety research. This article describes a research impact model that has been in use for over a decade. The model was developed to track and describe the impact of research conducted by a mid-sized institute that focuses on work and health. Model development was informed by existing models, with the goal of contextualizing the institute's case studies describing three types of research impact: evidence of the diffusion of research; evidence of research informing decision-making; and evidence of societal impact. A logic model describes research actions and outcomes, as well as key audiences and knowledge transfer approaches. A unique element is its indication of the level of difficulty in determining types of impact. The model compares well with current research impact models developed or used in healthcare and health services research, and it has been useful in guiding a mid-sized research organization's process for tracking and describing the impact of its research. It may be useful to other small and mid-sized research organizations that focus on workplace health and safety.

The use of an impact framework to evaluate the impact of research on policy and practice: Screening questionnaires for intellectual disability

There is an increasing emphasis on the importance of research having an impact on policy and practice. This can be more difficult to evidence in intellectual disability services because of the wide range of stakeholders involved. We evaluated whether an impact questionnaire covering: knowledge production, capacity building, informing policy and practice, social and economic benefits could successfully be used to evaluate the impact of research into and use of two screening questionnaires: the Learning Disability Screening Questionnaire (LDSQ) and Child and Adolescent Intellectual Disability Screening Questionnaire (CAIDS-Q). We conducted an online search for published peer reviewed and grey literature, and Internet resources that referenced LDSQ and/or CAIDS-Q. The resultant literature and resources were assessed for relevance and organized according to the categories outlined in the impact questionnaire. Evidence was found for all the areas of impact, with the largest body of evidence being in relation to informing policy and practice and social benefits and the least for economic benefits. The impact questionnaire provided the basis for a comprehensive and useful evaluative framework to assess impact, although there was some overlap between the different categories. The process of using it highlighted some wider issues to consider when attempting to evaluate impact. The results indicated that the research underpinning the LDSQ and CAIDS-Q had resulted in significant, generally positive, and wide-reaching impact on policy and practice in intellectual disability and other services, resulting in a number of positive outcomes.

Employing the Payback Framework to Assess Implementation Science Research Utilization: Lessons From the USAID's PEPFAR HIV/AIDS Implementation Science Awards

BACKGROUND

Stakeholders question whether implementation science (IS) is successful in conducting rigorous science that expedites the inclusion of health innovations into policies and accelerates the pace and scale of health service delivery into clinical and public health practice. Using the Payback Framework (PF) for research utilization (RU), we assessed the impact of USAID's IS investment on a subset of studies examining HIV prevention, care, and treatment.

SETTING

Selected USAID-funded IS awards implemented between 2012 and 2017 in 9 sub-Saharan African countries.

METHODS

A modified version of a RU framework, the PF, was applied to 10 USAID-funded IS awards. A semistructured, self-administered/interviewer-administered questionnaire representing operational items for the 5 categories of the modified PF was used to describe the type and to quantify the level of payback achieved. The raw score was tallied within and across the 5 PF categories, and the percentage of "payback" achieved by category was tabulated. Distribution of payback scores was summarized by tertiles.

RESULTS

Knowledge production had the highest level of payback (75%), followed by benefits to future research (70%), benefits to policy (45%), benefits to health and the health system (18%), and broader economic benefits (5%).

CONCLUSIONS

All awards achieved some level of knowledge production and benefits to future research, but translation to policy and programs was low and variable. We propose the use of policy, health system, and economic monitoring indicators of RU throughout the research process to increase IS studies' impact on health practice, programs, and policy.

Impact of surgical intervention trials on healthcare: A systematic review of assessment methods, healthcare outcomes, and determinants

BACKGROUND

Frameworks used in research impact evaluation studies vary widely and it remains unclear which methods are most appropriate for evaluating research impact in the field of surgical research. Therefore, we aimed to identify and review the methods used to assess the impact of surgical intervention trials on healthcare and to identify determinants for surgical impact.

METHODS

We searched journal databases up to March 10, 2020 for papers assessing the impact of surgical effectiveness trials on healthcare. Two researchers independently screened the papers for eligibility and performed a Risk of Bias assessment. Characteristics of both impact papers and trial papers were summarized. Univariate analyses were performed to identify determinants for finding research impact, which was defined as a change in healthcare practice.

RESULTS

Sixty-one impact assessments were performed in 37 included impact papers. Some surgical trial papers were evaluated in more than one impact paper, which provides a total of 38 evaluated trial papers. Most impact papers were published after 2010 (n = 29). Medical records (n = 10), administrative databases (n = 22), and physician's opinion through surveys (n = 5) were used for data collection. Those data were analyzed purely descriptively (n = 3), comparing data before and after publication (n = 29), or through time series analyses (n = 5). Significant healthcare impact was observed 49 times and more often in more recent publications. Having impact was positively associated with using medical records or administrative databases (ref.: surveys), a longer timeframe for impact evaluation and more months between the publication of the trial paper and the impact paper, data collection in North America (ref.: Europe), no economic evaluation of the intervention, finding no significant difference in surgical outcomes, and suggesting de-implementation in the original trial paper.

CONCLUSIONS AND IMPLICATIONS

Research impact evaluation receives growing interest, but still a small number of impact papers per year was identified. The analysis showed that characteristics of both surgical trial papers and impact papers were associated with finding research impact. We advise to collect data from either medical records or administrative databases, with an evaluation time frame of at least 4 years since trial publication.

Evaluating research investment and impact at a regional Australian Hospital and Health Service: a programme theory and conceptual framework

Background

Health systems in Australia and worldwide are increasingly expected to conduct research and quality improvement activities in addition to delivering clinical care and training health professionals. This study aims to inform a research impact evaluation at a regional Australian Hospital and Health Service by developing a programme theory showing how research investment is expected to have impact.

Methods

This qualitative study, representing the first phase of a larger mixed methods research impact evaluation at the Townsville Hospital and Health Service (THHS), adopts a realist-informed design involving the development of a programme theory. Data were obtained between February and May 2019 from strategic documentation and interviews with six current and former health service executives and senior employees. Inductive themes were integrated into a conceptual framework to visually represent the programme theory.

Results

Research at THHS has developed organically as the service has matured into a regional tertiary referral service serving a diverse rural and remote population across northern Queensland. Throughout this journey, individual THHS leaders often adopted a research development mantle despite disincentives arising from a performance-driven reporting and activity-based funding service context. Impact expectations from research investment at THHS were identified in the categories of enhanced research activity and capacity among clinicians, and improved clinical practice, health workforce capability and stability, and patient and population health. Seven contextual factors were identified as potential enablers or obstacles to these impact expectations and ambitions.

Conclusions

By identifying both relevant impact types and key contextual factors, this study offers programme theory to inform a planned research impact evaluation at THHS. The conceptual framework may be useful in other regionally based health service settings. More broadly, there are opportunities for future research to test and refine hybrid versions of linear and realist research impact evaluation models that combine resource-intensive, theory-driven approaches with policy practicality.

Review of published evidence on knowledge translation capacity, practice and support among researchers and research institutions in low- and middle-income countries

BACKGROUND

Knowledge translation (KT) is a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically sound application of knowledge to yield beneficial outcomes for society. Effective KT requires researchers to play an active role in promoting evidence uptake. This paper presents a systematised review of evidence on low- and middle-income country (LMIC) researchers' KT capacity, practice and interventions for enhancing their KT practice (support) with the aim of identifying gaps and informing future research and interventions.

METHODS

An electronic search for peer-reviewed publications focusing on LMIC researchers' KT capacity, practice and support across all academic fields, authored in English and from the earliest records available to February 2019, was conducted using PubMed and Scopus. Selected studies were appraised using the Mixed Methods Appraisal Tool, data pertaining to publication characteristics and study design extracted, and an a priori thematic analysis of reported research findings completed.

RESULTS

The search resulted in 334 screened articles, of which 66 met the inclusion criteria. Most (n = 43) of the articles presented original research findings, 22 were commentaries and 1 was a structured review; 47 articles reported on researchers' KT practice, 12 assessed the KT capacity of researchers or academic/research institutions and 9 reported on KT support for researchers. More than half (59%) of the articles focused on sub-Saharan Africa and the majority (91%) on health research. Most of the primary studies used the case study design (41%). The findings suggest that LMIC researchers rarely conduct KT and face a range of barriers at individual and institutional levels that limit their KT practice, including inadequate KT knowledge and skills, particularly for communicating research and interacting with research end-users, insufficient funding, and inadequate institutional guidelines, structures and incentives promoting KT practice. Furthermore, the evidence-base on effective interventions for enhancing LMIC researchers' KT practice is insufficient and largely of weak quality.

CONCLUSIONS

More high-quality research on researchers' KT capacity, practice and effective KT capacity strengthening interventions is needed. Study designs that extend beyond case studies and descriptive studies are recommended, including better designed evaluation studies, e.g. use of realist approaches, pragmatic trials, impact evaluations, implementation research and participatory action research.

How do organisations implement research impact assessment (RIA) principles and good practice? A narrative review and exploratory study of four international research funding and administrative organisations

Background

Public research funding agencies and research organisations are increasingly accountable for the wider impacts of the research they support. While research impact assessment (RIA) frameworks and tools exist, little is known and shared of how these organisations implement RIA activities in practice.

Methods

We conducted a review of academic literature to search for research organisations’ published experiences of RIAs. We followed this with semi-structured interviews from a convenience sample (n = 7) of representatives of four research organisations deploying strategies to support and assess research impact.

Results

We found only five studies reporting empirical evidence on how research organisations put RIA principles into practice. From our interviews, we observed a disconnect between published RIA frameworks and tools, and the realities of organisational practices, which tended not to be reported.

We observed varying maturity and readiness with respect to organisations’ structural set ups for conducting RIAs, particularly relating to leadership, skills for evaluation and automating RIA data collection. Key processes for RIA included efforts to engage researcher communities to articulate and plan for impact, using a diversity of methods, frameworks and indicators, and supporting a learning approach. We observed outcomes of RIAs as having supported a dialogue to orient research to impact, underpinned shared learning from analyses of research, and provided evidence of the value of research in different domains and to different audiences.

Conclusions

Putting RIA principles and frameworks into practice is still in early stages for research organisations. We recommend that organisations (1) get set up by considering upfront the resources, time and leadership required to embed impact strategies throughout the organisation and wider research ‘ecosystem’, and develop methodical approaches to assessing impact; (2) work together by engaging researcher communities and wider stakeholders as a core part of impact pathway planning and subsequent assessment; and (3) recognise the benefits that RIA can bring about as a means to improve mutual understanding of the research process between different actors with an interest in research.

Research funding impact and priority setting - advancing universal access and quality healthcare research in Malaysia

BACKGROUND

Health Research Priority Setting (HRPS) in the Ministry of Health (MOH) Malaysia was initiated more than a decade ago to drive effort toward research for informed decision and policy-making. This study assessed the impact of funded prioritised research and identified research gaps to inform future priority setting initiatives for universal access and quality healthcare in Malaysia.

METHODS

Research impact of universal access and quality healthcare projects funded by the National Institutes of Health Malaysia were assessed based on the modified Payback Framework, addressing categories of informing policy, knowledge production, and benefits to health and health sector. For the HRPS process, the Child Health and Nutrition Research Initiative methodology was adapted and adopted, with the incorporation of stakeholder values using weights and monetary allocation survey. Workshop discussions and interviews with stakeholders and research groups were conducted to identify research gaps, with the use of conceptual frameworks to guide the search.

RESULTS

Seventeen ongoing and 50 completed projects were identified for research funding impact analysis. Overall, research fund allocation differed from stakeholders' expectation. For research impact, 48 out of 50 completed projects (96.0%) contributed to some form of policy-making efforts. Almost all completed projects resulted in outputs that contributed to knowledge production and were expected to lead to health and health sector benefits. The HRPS process led to the identification of research priority areas that stemmed from ongoing and new issues identified for universal access and quality healthcare.

CONCLUSION

The concerted efforts of evaluation of research funding impact, prioritisation, dissemination and policy-maker involvement were valuable for optimal health research resource utilisation in a resource constrained developing country. Embedding impact evaluation into a priority setting process and funding research based on national needs could facilitate health research investment to reach its potential.

Development and validation of a questionnaire to measure research impact

Although questionnaires are widely used in research impact assessment, their metric properties are not well known. Our aim is to test the internal consistency and content validity of an instrument designed to measure the perceived impacts of a wide range of research projects. To do so, we designed a questionnaire to be completed by principal investigators in a variety of disciplines (arts and humanities, social sciences, health sciences, and information and communication technologies). The impacts perceived and their associated characteristics were also assessed. This easy-to-use questionnaire demonstrated good internal consistency and acceptable content validity. However, its metric properties were more powerful in areas such as knowledge production, capacity building and informing policy and practice, in which the researchers had a degree of control and influence. In general, the research projects represented an stimulus for the production of knowledge and the development of research skills. Behavioural aspects such as engagement with potential users or mission-oriented projects (targeted to practical applications) were associated with higher social benefits. Considering the difficulties in assessing a wide array of research topics, and potential differences in the understanding of the concept of ‘research impact’, an analysis of the context can help to focus on research needs. Analyzing the metric properties of questionnaires can open up new possibilities for validating instruments used to measure research impact. Further to the methodological utility of the current exercise, we see a practical applicability to specific contexts where multiple discipline research impact is requires.

How are evidence generation partnerships between researchers and policy-makers enacted in practice? A qualitative interview study

Background

Evidence generation partnerships between researchers and policy-makers are a potential method for producing more relevant research with greater potential to impact on policy and practice. Little is known about how such partnerships are enacted in practice, however, or how to increase their effectiveness. We aimed to determine why researchers and policy-makers choose to work together, how they work together, which partnership models are most common, and what the key (1) relationship-based and (2) practical components of successful research partnerships are.

Methods

Semi-structured qualitative interviews were conducted with 18 key informants largely based in New South Wales, Australia, who were (1) researchers experienced in working in partnership with policy in health or health-related areas or (2) policy and programme developers and health system decision-makers experienced in working in partnership with researchers. Data was analysed thematically by two researchers.

Results

Researcher-initiated and policy agency-initiated evidence generation partnerships were common. While policy-initiated partnerships were thought to be the most likely to result in impact, researcher-initiated projects were considered important in advancing the science and were favoured by researchers due to greater perceived opportunities to achieve key academic career metrics. Participants acknowledged that levels of collaboration varied widely in research/policy partnerships from minimal to co-production. Co-production was considered a worthy goal by all, conferring a range of benefits, but one that was difficult to achieve in practice. Some participants asserted that the increased time and resources required for effective co-production meant it was best suited to evaluation and implementation projects where the tacit, experiential knowledge of policy-makers provided critical nuance to underpin study design, implementation and analysis. Partnerships that were mutually considered to have produced the desired outcomes were seen to be underpinned by a range of both relationship-based (such as shared aims and goals and trust) and practical factors (such as sound governance and processes).

Conclusions

Our findings highlight the important role of policy-makers in New South Wales in ensuring the relevance of research. There is still much to understand about how to initiate and sustain successful research/policy partnerships, particularly at the highly collaborative end.

Electronic supplementary material

The online version of this article (10.1186/s12961-019-0441-2) contains supplementary material, which is available to authorized users.

How to engage stakeholders in research: design principles to support improvement

Background

Closing the gap between research production and research use is a key challenge for the health research system. Stakeholder engagement is being increasingly promoted across the board by health research funding organisations, and indeed by many researchers themselves, as an important pathway to achieving impact. This opinion piece draws on a study of stakeholder engagement in research and a systematic literature search conducted as part of the study.

Main body

This paper provides a short conceptualisation of stakeholder engagement, followed by ‘design principles’ that we put forward based on a combination of existing literature and new empirical insights from our recently completed longitudinal study of stakeholder engagement. The design principles for stakeholder engagement are organised into three groups, namely organisational, values and practices. The organisational principles are to clarify the objectives of stakeholder engagement; embed stakeholder engagement in a framework or model of research use; identify the necessary resources for stakeholder engagement; put in place plans for organisational learning and rewarding of effective stakeholder engagement; and to recognise that some stakeholders have the potential to play a key role. The principles relating to values are to foster shared commitment to the values and objectives of stakeholder engagement in the project team; share understanding that stakeholder engagement is often about more than individuals; encourage individual stakeholders and their organisations to value engagement; recognise potential tension between productivity and inclusion; and to generate a shared commitment to sustained and continuous stakeholder engagement. Finally, in terms of practices, the principles suggest that it is important to plan stakeholder engagement activity as part of the research programme of work; build flexibility within the research process to accommodate engagement and the outcomes of engagement; consider how input from stakeholders can be gathered systematically to meet objectives; consider how input from stakeholders can be collated, analysed and used; and to recognise that identification and involvement of stakeholders is an iterative and ongoing process.

Conclusion

It is anticipated that the principles will be useful in planning stakeholder engagement activity within research programmes and in monitoring and evaluating stakeholder engagement. A next step will be to address the remaining gap in the stakeholder engagement literature concerned with how we assess the impact of stakeholder engagement on research use.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0337-6) contains supplementary material, which is available to authorized users.

ISRIA statement: ten-point guidelines for an effective process of research impact assessment

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school's learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders' needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice.The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school's faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.

'Knowledge for better health' revisited - the increasing significance of health research systems: a review by departing Editors-in-Chief

How can nations organise research investments to obtain the best bundle of knowledge and the maximum level of improved health, spread as equitably as possible? This question was the central focus of a major initiative from WHO led by Prof Tikki Pang, which resulted in a range of developments, including the publication of a conceptual framework for national health research systems - Knowledge for better health - in 2003, and in the founding of the journal Health Research Policy and Systems (HARPS). As Editors-in-Chief of the journal since 2006, we mark our retirement by tracking both the progress of the journal and the development of national health research systems. HARPS has maintained its focus on a range of central themes that are key components of a national health research system in any country. These include building capacity to conduct and use health research, identifying appropriate priorities, securing funds and allocating them accountably, producing scientifically valid research outputs, promoting the use of research in polices and practice in order to improve health, and monitoring and evaluating the health research system. Some of the themes covered in HARPS are now receiving increased attention and, for example, with the assessment of research impact and development of knowledge translation platforms, the journal has covered their progress throughout that expansion of interest. In addition, there is increasing recognition of new imperatives, including the importance of promoting gender equality in health research if benefits are to be maximised. In this Editorial, we outline some of the diverse and developing perspectives considered within each theme, as well as considering how they are held together by the growing desire to build effective health research systems in all countries.From 2003 until mid-June 2017, HARPS published 590 articles on the above and related themes, with authors being located in 76 countries. We present quantitative data tracing the journal's growth and the increasing external recognition of its role. We thank the many colleagues who have kindly contributed to the journal's success, and finish on an exciting note by welcoming the new Editors-in-Chief who will take HARPS forward.

Health services research: building capacity to meet the needs of the health care system

Health services researchers have an important role to play in helping health care systems around the world provide high quality, affordable services. However, gaps between the best evidence and current practice suggest that researchers need to work in new ways. The production of research that meets the needs and priorities of the health system requires researchers to work in partnership with decision-makers to conduct research and then mobilize the findings. To do this effectively, researchers require a new set of skills that are not conventionally taught as part of doctoral research programmes. In addition to wider contextual changes, researchers need to understand better the needs of decision-makers, for example through short placements in health system decision-making settings. Second, researchers need to learn to accommodate those needs throughout the research process, including identifying research needs; conducting research collaboratively with decision-makers and producing effective research products.

Transferring research from a university to the United Kingdom National Health Service: the implications for impact

The aim of this article is to inform readers of the author’s reflections on the experience of transferring university-based research into the commercial sector, and of the processes and strategies employed when preparing for impact in so doing. Concepts for the transfer are illustrated by the author’s reflection on aspects that arose during the birthing and subsequent start-up of a university spin-off, Pathways2Wellbeing, a form of reflection-on-action. This is the vehicle for the adaption required to transfer research into the delivery of a specialised clinic in the United Kingdom National Health Service for people with medically unexplained, persistent, bodily symptoms such as fibromyalgia, chronic fatigue and chronic pain. It is hoped that the article will provide readers with an insight into how knowledge transfer can take place through engagement with stakeholders to create an exchange of knowledges to result in impact on health service policy for service users, despite the challenges, and the enablers that facilitated this process. The reflections on the process of knowledge transfer and the implications for impact are underpinned by relevant theory.