Research impact: neither quick nor easy

Prioritising and incentivising productivity within indicator-based approaches to Research Impact Assessment: a commentary

Research Impact Assessment (RIA) represents one of a suite of policies intended to improve the impact generated from investment in health and medical research (HMR). Positivist indicator-based approaches to RIA are widely implemented but increasingly criticised as theoretically problematic, unfair, and burdensome. This commentary proposes there are useful outcomes that emerge from the process of applying an indicator-based RIA framework, separate from those encapsulated in the metrics themselves. The aim for this commentary is to demonstrate how the act of conducting an indicator-based approach to RIA can serve to optimise the productive gains from the investment in HMR. Prior research found that the issues regarding RIA are less about the choice of indicators/metrics, and more about the discussions prompted and activities incentivised by the process. This insight provides an opportunity to utilise indicator-based methods to purposely optimise the research impact. An indicator-based RIA framework specifically designed to optimise research impacts should: focus on researchers and the research process, rather than institution-level measures; utilise a project level unit of analysis that provides control to researchers and supports collaboration and accountability; provide for prospective implementation of RIA and the prospective orientation of research; establish a line of sight to the ultimate anticipated beneficiaries and impacts; Include process metrics/indicators to acknowledge interim steps on the pathway to final impacts; integrate 'next' users and prioritise the utilisation of research outputs as a critical measure; Integrate and align the incentives for researchers/research projects arising from RIA, with those existing within the prevailing research system; integrate with existing peer-review processes; and, adopt a system-wide approach where incremental improvements in the probability of translation from individual research projects, yields higher impact across the whole funding portfolio.Optimisation of the impacts from HMR investment represents the primary purpose of Research Impact policy. The process of conducting an indicator-based approach to RIA, which engages the researcher during the inception and planning phase, can directly contribute to this goal through improvements in the probability that an individual project will generate interim impacts. The research project funding process represents a promising forum to integrate this approach within the existing research system.

How can we achieve impact from public health research? A meta-ethnography of case studies

BACKGROUND

Generation of public health impact from research is challenging. Research of similar quality often has differential uptake and there is considerable lag time between initiation and uptake of research. Improving understanding of how research impact can be achieved may identify areas stakeholders could target.

METHODS

This work uses meta-ethnography to synthesize 21 case studies exploring how researchers have generated public health policy impact.

RESULTS

Eight constructs were identified: expertise; motivation; practical solutions to important problems; support structure and funding; collaboration; wide dissemination and use of media to contribute ideas to the wider narrative; understanding the policy realm; and models of impact. The constructs were combined in a lines-of-argument synthesis, producing a model that seeks to illustrate the diffuse, complex and dynamic nature of the process of generating impact from research.

CONCLUSION

Achieving research impact involves seeking to shape wider debates, building relationships with policy makers, becoming a trusted collaborator and being available to provide relevant and practical solutions to questions of concern to policy makers at the appropriate time.

Mechanisms and pathways to impact in public health research: a preliminary analysis of research funded by the National Institute for Health Research (NIHR)

Background

The mechanisms and pathways to impacts from public health research in the UK have not been widely studied. Through the lens of one funder (NIHR), our aims are to map the diversity of public health research, in terms of funding mechanisms, disciplinary contributions, and public health impacts, identify examples of impacts, and pathways to impact that existing reporting mechanisms may not otherwise have captured, and provide illustrations of how public health researchers perceive the generation of non-academic impact from their work.

Methods

A total of 1386 projects were identified as ‘public health research’ by the NIHR and listed in the NIHR Public Health Overview database (2000–2016). From these, a subset of 857 projects were matched as potentially having begun reporting impacts via an external data-gathering platform (Researchfish). Data on the 857 projects were analyzed quantitatively, and nine projects were selected to investigate further through semi-structured interviews with principal investigators. Two workshops took place to validate emerging and final findings and facilitate analysis.

Results

In addition to the NIHR School for Public Health Research and the NIHR Public Health Research Programme, 89% of projects contained in the NIHR Public Health Overview portfolio as ‘public health research’ are funded via other NIHR research programmes, suggesting significant diversity in disciplines contributing to public health research and outcomes. The pathways to impact observed in our in-depth case studies include contributing to debates on what constitutes appropriate evidence for national policy change, acknowledging local ‘unintended’ impacts, building trusted relationships with stakeholders across health and non-health sectors and actors, collaborating with local authorities, and using non-academic dissemination channels.

Conclusions

Public health as a discipline contributes substantially to impact beyond academia. To support the diversity of these impacts, we need to recognise localized smaller-scale impacts, and the difference in types of evidence required for community and local authority-based impacts. This will also require building capacity and resources to enable impact to take place from public health research. Finally, support is required for engagement with local authorities and working with non-health sectors that contribute to health outcomes.

Evidence for Health II: Overcoming barriers to using evidence in policy and practice

Even the highest quality evidence will have little impact unless it is incorporated into decision-making for health. It is therefore critical to overcome the many barriers to using evidence in decision-making, including (1) missing the window of opportunity, (2) knowledge gaps and uncertainty, (3) controversy, irrelevant and conflicting evidence, as well as (4) vested interests and conflicts of interest. While this is certainly not a comprehensive list, it covers a number of main themes discussed in the knowledge translation literature on this topic, and better understanding these barriers can help readers of the evidence to be more savvy knowledge users and help researchers overcome challenges to getting their evidence into practice. Thus, the first step in being able to use research evidence for improving population health is ensuring that the evidence is available at the right time and in the right format and language so that knowledge users can take the evidence into consideration alongside a multitude of other factors that also influence decision-making. The sheer volume of scientific publications makes it difficult to find the evidence that can actually help inform decisions for health. Policymakers, especially in low- and middle-income countries, require context-specific evidence to ensure local relevance. Knowledge synthesis and dissemination of policy-relevant local evidence is important, but it is still not enough. There are times when the interpretation of the evidence leads to various controversies and disagreements, which act as barriers to the uptake of evidence. Research evidence can also be influenced and misused for various aims and agendas. It is therefore important to ensure that any new evidence comes from reliable sources and is interpreted in light of the overall body of scientific literature. It is not enough to simply produce evidence, nor even to synthesize and package evidence into a more user-friendly format. Particularly at the policy level, political savvy is also needed to ensure that vested interests do not undermine decisions that can impact the health of individuals and populations.