Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Background

Measuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research.

Methods

Keyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework.

Results

We identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilised forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change.

Conclusions

The Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0310-4) contains supplementary material, which is available to authorized users.

Impact assessment of Iran's health technology assessment programme

BACKGROUND

Following approximately 10 years from the beginning of Iran's national Health Technology Assessment (HTA) programme, the present study aims to evaluate its success by examining the impact of HTA and identifying the determinant factors leading to the implementation of HTA report results.

METHODS

The triangulation method was employed herein. HTA reports were initially identified and their impact and determinant factors were then examined from the perspectives of both researchers (by preparing a questionnaire according to the Payback model and sending it to HTA principle investigators) and stakeholders (semi-structured interviews held with each HTA stakeholder). Simultaneously, the quality of the HTA reports was examined with relevant critical appraisal checklists.

RESULTS

The impact of 19 equipment technologies and four pharmaceutical technologies were assessed in this study. Twenty researchers replied (response rate, 86.96%) to the questionnaire on the impact of HTA reports from the researcher's perspective. To assess the impact of HTA reports from the stakeholder's perspective, seven policy-makers were chosen and interviewed as the main target audience. The most common step taken to disseminate the results of the HTA projects was publication. Conducting the HTA had taught researchers and their colleagues' new skills and had facilitated the securing of research grants from other organisations. Most reports had used the systematic review method but the relevant details had been scarcely presented regarding outcomes, costs and analysis. The greatest impact of HTA reports on decision-making had been on policy-makers providing and allocating finances. Barriers in stewardship, identification and prioritisation of topics, performance and dissemination of HTA results were the main barriers of implementing HTAs.

CONCLUSIONS

In most aspects, the status of HTA impact reports need improvement. Thus far, the barriers and facilitators of the HTA programme in Iran have been investigated in other studies. These findings should be pooled to reach a solution that can be actively applied to the health system to improve the status quo of HTA in Iran.

Estimating the returns to United Kingdom publicly funded musculoskeletal disease research in terms of net value of improved health outcomes

BACKGROUND

Building on an approach applied to cardiovascular and cancer research, we estimated the economic returns from United Kingdom public- and charitable-funded musculoskeletal disease (MSD) research that arise from the net value of the improved health outcomes in the United Kingdom.

METHODS

To calculate the economic returns from MSD-related research in the United Kingdom, we estimated (1) the public and charitable expenditure on MSD-related research in the United Kingdom between 1970 and 2013; (2) the net monetary benefit (NMB), derived from the health benefit in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of £25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1994 to 2013; (3) the proportion of NMB attributable to United Kingdom research; and (4) the elapsed time between research funding and health gain. The data collected from these four key elements were used to estimate the internal rate of return (IRR) from MSD-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using a one-way sensitivity analysis.

RESULTS

Expressed in 2013 prices, total expenditure on MSD-related research from 1970 to 2013 was £3.5 billion, and for the period used to estimate the rate of return, 1978-1997, was £1.4 billion. Over the period 1994-2013 the key interventions analysed produced 871,000 QALYs with a NMB of £16 billion, allowing for the net NHS costs resulting from them and valuing a QALY at £25,000. The proportion of benefit attributable to United Kingdom research was 30% and the elapsed time between funding and impact of MSD treatments was 16 years. Our best estimate of the IRR from MSD-related research was 7%, which is similar to the 9% for CVD and 10% for cancer research.

CONCLUSIONS

Our estimate of the IRR from the net health gain to public and charitable funding of MSD-related research in the United Kingdom is substantial, and justifies the research investments made between 1978 and 1997. We also demonstrated the applicability of the approach previously used in assessing the returns from cardiovascular and cancer research. Inevitably, with a study of this kind, there are a number of important assumptions and caveats that we highlight, and these can inform future research.