Research can be integrated into public health policy-making: global lessons for and from Spanish economic evaluations

WHO promotes the use of research in policy-making to drive improvements in health, including in achieving Sustainable Development Goals such as tobacco control. The European Union’s new €95 billion Horizon Europe research framework programme parallels these aims, and also includes commitments to fund economic evaluations. However, researchers often express frustration at the perceived lack of attention to scientific evidence during policy-making. For example, some researchers claim that evidence regarding the return on investment from optimal implementation of evidence-based policies is frequently overlooked. An increasingly large body of literature acknowledges inevitable barriers to research use, but also analyses facilitators encouraging such use. This opinion piece describes how some research is integrated into policy-making. It highlights two recent reviews. One examines impact assessments of 36 multi-project research programmes and identifies three characteristics of projects more likely to influence policy-making. These include a focus on healthcare system needs, engagement of stakeholders, and research conducted for organizations supported by structures to receive and use evidence. The second review suggests that such characteristics are likely to occur as part of a comprehensive national health research system strategy, especially one integrated into the healthcare system. We also describe two policy-informing economic evaluations conducted in Spain. These examined the most cost-effective package of evidence-based tobacco control interventions and the cost-effectiveness of different strategies to increase screening coverage for cervical cancer. Both projects focused on issues of healthcare concern and involved considerable stakeholder engagement. The Spanish examples reinforce some lessons from the global literature and, therefore, could help demonstrate to authorities in Spain the value of developing comprehensive health research systems, possibly following the interfaces and receptor model. The aim of this would be to integrate needs assessment and stakeholder engagement with structures spanning the research and health systems. In such structures, economic evaluation evidence could be collated, analysed by experts in relation to healthcare needs, and fed into both policy-making as appropriate, and future research calls. The increasingly large local and global evidence base on research utilization could inform detailed implementation of this approach once accepted as politically desirable. Given the COVID-19 pandemic, increasing the cost-effectiveness of healthcare systems and return on investment of public health interventions becomes even more important.

Health research systems in change: the case of 'Push the Pace' in the National Institute for Health Research

BACKGROUND

Those running well-organised health research systems are likely to be alert for ways in which they might increase the quality of the services they provide and address any problems identified. This is important because the efficiency of the research system can have a major impact on how long it takes for new treatments to be developed and reach patients. This opinion piece reflects on the experience and learning of the United Kingdom-based National Institute for Health Research (NIHR) when it implemented continuous improvement activity to improve its processes.

DISCUSSION

This paper describes the structure and work of the NIHR and why, despite is successes as a health research system and ongoing local continuous improvement, it believed in the value of an organisation-wide continuous improvement activity. It did this by implementing an approach called 'Push the Pace'. Initially, the organisation focused on reducing the amount of time it took for research to transition from an early concept to evidence that changes lives. This scrutiny enabled the NIHR to realise further areas of improvement it could make - additional goals were increased transparency, process simplification, and improved customer and stakeholder experience. We discuss our experience of Push the Pace with reference to literature on continuous improvement.

CONCLUSION

Continuous improvement is a cycle, an activity that is done constantly and over time, rather than an act or linear activity (such as Push the Pace). We believe that the work of Push the Pace has initiated a strong commitment to a culture of continuous improvement in the NIHR. This is significant because culture change is widely recognised as immensely challenging, particularly in such a large and distributed organisation. However, our biggest challenge will be to enable all staff and stakeholders of the NIHR to participate in the continuous improvement cycle.

How to engage stakeholders in research: design principles to support improvement

Background

Closing the gap between research production and research use is a key challenge for the health research system. Stakeholder engagement is being increasingly promoted across the board by health research funding organisations, and indeed by many researchers themselves, as an important pathway to achieving impact. This opinion piece draws on a study of stakeholder engagement in research and a systematic literature search conducted as part of the study.

Main body

This paper provides a short conceptualisation of stakeholder engagement, followed by ‘design principles’ that we put forward based on a combination of existing literature and new empirical insights from our recently completed longitudinal study of stakeholder engagement. The design principles for stakeholder engagement are organised into three groups, namely organisational, values and practices. The organisational principles are to clarify the objectives of stakeholder engagement; embed stakeholder engagement in a framework or model of research use; identify the necessary resources for stakeholder engagement; put in place plans for organisational learning and rewarding of effective stakeholder engagement; and to recognise that some stakeholders have the potential to play a key role. The principles relating to values are to foster shared commitment to the values and objectives of stakeholder engagement in the project team; share understanding that stakeholder engagement is often about more than individuals; encourage individual stakeholders and their organisations to value engagement; recognise potential tension between productivity and inclusion; and to generate a shared commitment to sustained and continuous stakeholder engagement. Finally, in terms of practices, the principles suggest that it is important to plan stakeholder engagement activity as part of the research programme of work; build flexibility within the research process to accommodate engagement and the outcomes of engagement; consider how input from stakeholders can be gathered systematically to meet objectives; consider how input from stakeholders can be collated, analysed and used; and to recognise that identification and involvement of stakeholders is an iterative and ongoing process.

Conclusion

It is anticipated that the principles will be useful in planning stakeholder engagement activity within research programmes and in monitoring and evaluating stakeholder engagement. A next step will be to address the remaining gap in the stakeholder engagement literature concerned with how we assess the impact of stakeholder engagement on research use.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0337-6) contains supplementary material, which is available to authorized users.

Estimating the returns to United Kingdom publicly funded musculoskeletal disease research in terms of net value of improved health outcomes

BACKGROUND

Building on an approach applied to cardiovascular and cancer research, we estimated the economic returns from United Kingdom public- and charitable-funded musculoskeletal disease (MSD) research that arise from the net value of the improved health outcomes in the United Kingdom.

METHODS

To calculate the economic returns from MSD-related research in the United Kingdom, we estimated (1) the public and charitable expenditure on MSD-related research in the United Kingdom between 1970 and 2013; (2) the net monetary benefit (NMB), derived from the health benefit in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of £25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1994 to 2013; (3) the proportion of NMB attributable to United Kingdom research; and (4) the elapsed time between research funding and health gain. The data collected from these four key elements were used to estimate the internal rate of return (IRR) from MSD-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using a one-way sensitivity analysis.

RESULTS

Expressed in 2013 prices, total expenditure on MSD-related research from 1970 to 2013 was £3.5 billion, and for the period used to estimate the rate of return, 1978-1997, was £1.4 billion. Over the period 1994-2013 the key interventions analysed produced 871,000 QALYs with a NMB of £16 billion, allowing for the net NHS costs resulting from them and valuing a QALY at £25,000. The proportion of benefit attributable to United Kingdom research was 30% and the elapsed time between funding and impact of MSD treatments was 16 years. Our best estimate of the IRR from MSD-related research was 7%, which is similar to the 9% for CVD and 10% for cancer research.

CONCLUSIONS

Our estimate of the IRR from the net health gain to public and charitable funding of MSD-related research in the United Kingdom is substantial, and justifies the research investments made between 1978 and 1997. We also demonstrated the applicability of the approach previously used in assessing the returns from cardiovascular and cancer research. Inevitably, with a study of this kind, there are a number of important assumptions and caveats that we highlight, and these can inform future research.

Does the engagement of clinicians and organisations in research improve healthcare performance: a three-stage review

OBJECTIVE

There is a widely held assumption that engagement by clinicians and healthcare organisations in research improves healthcare performance at various levels, but little direct empirical evidence has previously been collated. The objective of this study was to address the question: Does research engagement (by clinicians and organisations) improve healthcare performance?

METHODS

An hourglass-shaped review was developed, consisting of three stages: (1) a planning and mapping stage; (2) a focused review concentrating on the core question of whether or not research engagement improves healthcare performance; and (3) a wider (but less systematic) review of papers identified during the two earlier stages, focusing on mechanisms.

RESULTS

Of the 33 papers included in the focused review, 28 identified improvements in health services performance. Seven out of these papers reported some improvement in health outcomes, with others reporting improved processes of care. The wider review demonstrated that mechanisms such as collaborative and action research can encourage some progress along the pathway from research engagement towards improved healthcare performance. Organisations that have deliberately integrated the research function into organisational structures demonstrate how research engagement can, among other factors, contribute to improved healthcare performance.

CONCLUSIONS

Current evidence suggests that there is an association between the engagement of individuals and healthcare organisations in research and improvements in healthcare performance. The mechanisms through which research engagement might improve healthcare performance overlap and rarely act in isolation, and their effectiveness often depends on the context in which they operate.

Delivering the aims of the Collaborations for Leadership in Applied Health Research and Care: understanding their strategies and contributions

Background

In 2008, the National Institute for Health Research (NIHR) in England established nine Collaborations for Leadership in Applied Health Research and Care (CLAHRCs) to develop partnerships between universities and local NHS organisations focused on improving patient outcomes through the conduct and application of applied health research.

Objectives

The study explored how effectively the CLAHRCs supported the ‘translation’ of research into patient benefit, and developed ways of doing applied research that maximised its chances of being useful to the service and the capacity of the NHS to respond. It focused on three issues: (1) how the NHS influenced the CLAHRCs, and vice versa; (2) how effective multistakeholder and multidisciplinary research and implementation teams were built in the CLAHRCs; (3) how the CLAHRCs supported the use of research knowledge to change commissioning and clinical behaviour for patient benefit.

Methods

The study adopted an adaptive and emergent approach and incorporated a formative evaluation. An initial phase mapped the landscape of all nine CLAHRCs and the context within which they were established, using document analysis, workshops and interviews, and a literature review. This mapping exercise identified the three research questions that were explored in phase 2 through a stakeholder survey of six CLAHRCs, in-depth case studies of two CLAHRCs, validation interviews with all nine CLAHRCs and the NIHR, and document review.

Results

(1) The local remit and the requirement for matched NHS funding enhanced NHS influence on the CLAHRCs. The CLAHRCs achieved positive change among those most directly involved, but the larger issue of whether or not the CLAHRCs can influence others in and across the NHS remains unresolved. (2) The CLAHRCs succeeded in engaging different stakeholder groups, and explored what encouraged specific groups to become involved. Being responsive to people’s concerns and demonstrating ‘quick wins’ were both important. (3) There was some evidence that academics were becoming more interested in needs-driven research, and that commissioners were seeing the CLAHRCs as a useful source of support. A growing number of completed projects had demonstrated an impact on clinical practice.

Conclusions

The CLAHRCs have included NHS decision-makers in research and researchers in service decision-making, and encouraged research-informed practice. All the CLAHRCs (as collaborations) adopted relationship models. However, as the complexities of the challenges they faced became clearer, it became obvious that a focus on multidisciplinary relationships was necessary, but not sufficient on its own. Attention also has to be paid to the systems within and through which these relationships operate.

Recommendations for research

Future research should compare areas with an Academic Health Science Network (AHSN) and a CLAHRC with areas with just an AHSN, to understand the difference CLAHRCs make. There should be work on understanding implementation, such as the balancing of rigour and relevance in intervention studies; systemic barriers to and facilitators of implementation; and tailoring improvement interventions. There is also a need to better understand the factors that support the explicit use of research evidence across the NHS, and the processes and mechanisms that support the sustainability and scale-up of implementation projects. Research should place emphasis on examining the role of patient and public involvement in CLAHRCs and of the relation between CLAHRCs and NHS commissioners.

Funding

The NIHR Health Services and Delivery Research programme.

Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes

BACKGROUND

Building on an approach developed to assess the economic returns to cardiovascular research, we estimated the economic returns from UK public and charitable funded cancer-related research that arise from the net value of the improved health outcomes.

METHODS

To assess these economic returns from cancer-related research in the UK we estimated: 1) public and charitable expenditure on cancer-related research in the UK from 1970 to 2009; 2) net monetary benefit (NMB), that is, the health benefit measured in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of GB£25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1991 to 2010; 3) the proportion of NMB attributable to UK research; 4) the elapsed time between research funding and health gain; and 5) the internal rate of return (IRR) from cancer-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using sensitivity analyses to illustrate the effect of some key parameters.

RESULTS

In 2011/12 prices, total expenditure on cancer-related research from 1970 to 2009 was £15 billion. The NMB of the 5.9 million QALYs gained from the prioritised interventions from 1991 to 2010 was £124 billion. Calculation of the IRR incorporated an estimated elapsed time of 15 years. We related 17% of the annual NMB estimated to be attributable to UK research (for each of the 20 years 1991 to 2010) to 20 years of research investment 15 years earlier (that is, for 1976 to 1995). This produced a best-estimate IRR of 10%, compared with 9% previously estimated for cardiovascular disease research. The sensitivity analysis demonstrated the importance of smoking reduction as a major source of improved cancer-related health outcomes.

CONCLUSIONS

We have demonstrated a substantive IRR from net health gain to public and charitable funding of cancer-related research in the UK, and further validated the approach that we originally used in assessing the returns from cardiovascular research. In doing so, we have highlighted a number of weaknesses and key assumptions that need strengthening in further investigations. Nevertheless, these cautious estimates demonstrate that the returns from past cancer research have been substantial, and justify the investments made during the period 1976 to 1995.

Investigating Time Lags and Attribution in the Translation of Cancer Research: A Case Study Approach

In 2012, RAND Europe and the Health Economics Research Group (Brunel University) were commissioned by the Wellcome Trust, Cancer Research UK, the National Institute for Health Research and the Academy of Medical Science to conduct a study of the returns to the public/charitable investment in cancer-related research. This study built on previous work published in the 2008 "What's it worth?" report that estimated the economic returns to medical research in terms of spillover benefits and health gain. The 2008 study was extensively quoted and cited as a clear justification for the economic importance of medical research and appears to have played a role in achieving the protection of the medical science budget in the recent public expenditure cuts. This cancer study used a similar approach to that used in the previous study, but with some methodological developments. One of the methodological developments was the inclusion of case studies to examine the validity and variability of the estimates on elapsed time between funding and health gains, and the amount of health gains that can be attributed to UK research. This study provides the full text of the five case studies conducted as well as some discussion of observations emerging across the case study set.

CLAHRCs in practice: combined knowledge transfer and exchange strategies, cultural change, and experimentation

OBJECTIVES

The nine NIHR CLAHRCs are collaborations between universities and local NHS organizations that seek to improve patient outcomes through the conduct and application of applied health research. The theoretical and practical context within which the CLAHRCs were set up was characterized by a considerable degree of uncertainty, and the CLAHRCs were established as a natural experiment.

METHODS

We adopted a formative and emergent evaluation approach. Drawing on in-depth, multi-method case studies of two CLAHRCs we explored how they pursued their remit by supporting efforts to increase the relevance and use of health research, and building relationships.

RESULTS

Both CLAHRCs: strengthened local networks and relationships; built capacity in their local academic and NHS communities to undertake and use research that meets the needs of the service; developed research and implementation methodologies; and added to understanding of the complex relation between research and implementation. There was evidence of impact of CLAHRC projects on health and social care services. Informed by the literature on implementing collaborative research initiatives, knowledge transfer and exchange and cultural change, some key lessons can be drawn.

CONCLUSION

The CLAHRCs pursued a strategy that can be categorized as one of flexible comprehensiveness; i.e. their programmes have been flexible and responsive and they have used a range of approaches that seek to match the diverse aspects of the complex issues they face. Key features include their work on combining a range of knowledge transfer and exchange strategies, their efforts to promote cultural change, and the freedom to experiment, learn and adapt. Although the CLAHRCs do not, by themselves, have the remit or resources to bring about wholesale service improvement in health care, they do have features that would allow them to play a key role in some of the wider initiatives that encourage innovation.

Engagement in research: an innovative three-stage review of the benefits for health-care performance

Background

There is a widely held assumption that research engagement improves health-care performance at various levels, but little direct empirical evidence.

Objectives

To conduct a theoretically and empirically grounded synthesis to map and explore plausible mechanisms through which research engagement might improve health services performance. A review of the effects on patients of their health-care practitioner's or institution's participation in clinical trials was published after submission of the proposal for this review. It identified only 13 relevant papers and, overall, suggested that the evidence that research engagement improves health-care performance was less strong than some thought. We aimed to meet the need for a wider review.

Methods

An hourglass review was developed, consisting of three stages: (1) a planning and mapping stage; (2) a focused review concentrating on the core question of whether or not research engagement improves health care; and (3) a wider (but less systematic) review of papers identified during the two earlier stages. Studies were included inthe focused review if the concept of ‘engagementinresearch’ was an input and some measure of ‘performance’ an output. The search strategy covered the period 1990 to March 2012. MEDLINE, EMBASE, PsycINFO, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science and other relevant databases were searched. A total of 10,239 papers were identified through the database searches, and 159 from other sources. A further relevance and quality check on 473 papers was undertaken, and identified 33 papers for inclusion in the review. A standard meta-analysis was not possible on the heterogeneous mix of papers in the focused review. Therefore an explanatory matrix was developed to help characterise the circumstances in which research engagement might improve health-care performance and the mechanisms that might be at work, identifying two main dimensions along which to categorise the studies:the degree of intentionalityandthe scope of the impact.

Results

Of the 33 papers in the focused review, 28 were positive (of which six were positive/mixed) in relation to the question of whether or not research engagement improves health-care performance. Five papers were negative (of which two were negative/mixed). Seven out of 28 positive papers reported some improvement in health outcomes. For the rest, the improved care took the form of improved processes of care. Nine positive papers were at a clinician level and 19 at an institutional level. The wider review demonstrated, for example, how collaborative and action research can encourage some progress along the pathway from research engagement towards improved health-care performance. There is also evidence that organisations in which the research function is fully integrated into the organisational structure out-perform other organisations that pay less formal heed to research and its outputs. The focused and wider reviews identified the diversity in the mechanisms through which research engagement might improve health care: there are many circumstances and mechanisms at work, more than one mechanism is often operative, and the evidence available for each one is limited.

Limitations

To address the complexities of this evidence synthesis of research we needed to spend significant time mapping the literature, and narrowed the research question to make it feasible. We excluded many potentially relevant papers (though we partially addressed this by conducting a wider additional synthesis). Studies assessing the impact made on clinician behaviour by small, locally conducted pieces of research could be difficult to interpret without full knowledge of the context.

Conclusions

Drawing on the focused and wider reviews, it is suggested that when clinicians and health-care organisations engage in research there is the likelihood of a positive impact on health-care performance. Organisations that have deliberately integrated the research function into organisational structures demonstrate how research engagement can, among other factors, contribute to improved health-care performance. Further explorations are required of research networks and schemes to promote the engagement of clinicians and managers in research. Detailed observational research focusing on research engagement within organisations would build up an understanding of mechanisms.

Study registration

PROSPERO: CRD42012001990.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Ways of assessing the economic value or impact of research: is it a step too far for nursing research?

To identify lessons that could be applied to nursing research, this paper draws on some reviews of the increasing, although apparently still relatively small, number of studies that attempt to assess economic impacts from health research. One review describes several important steps, including identifying the health and other outcomes from specific bodies of research and then valuing the outcomes. We describe major studies in fields such as cardiovascular research that show how the economic value of health research can be demonstrated. In addition, we examine various nursing studies, including ones showing the benefits (especially economic) from nursing itself (as opposed to from nursing research), and also studies describing economic evaluations of new devices and techniques used by nurses, which have the potential to be used when trying to value the research. Currently, such studies rarely go on and demonstrate how the nursing research has had a wider impact on nursing policies and practice, and hence led to outcomes that could be valued. There is, nevertheless, scope to build on these existing nursing studies. Conducting impact assessments could potentially result in a portfolio of examples of nursing research that have informed policies and practices and led to economic impacts in terms of cost savings, and possibly also in terms of health gains that could be valued.

Project Retrosight: Understanding the Returns from Cardiovascular and Stroke Research: The Policy Report

This project explores the impacts arising from cardiovascular and stroke research funded 15-20 years ago and attempts to draw out aspects of the research, researcher or environment that are associated with high or low impact. The project is a case study-based review of 29 cardiovascular and stroke research grants, funded in Australia, Canada and UK between 1989 and 1993. The case studies focused on the individual grants but considered the development of the investigators and ideas involved in the research projects from initiation to the present day. Grants were selected through a stratified random selection approach that aimed to include both high- and low-impact grants. The key messages are as follows: 1) The cases reveal that a large and diverse range of impacts arose from the 29 grants studied. 2) There are variations between the impacts derived from basic biomedical and clinical research. 3) There is no correlation between knowledge production and wider impacts 4) The majority of economic impacts identified come from a minority of projects. 5) We identified factors that appear to be associated with high and low impact. This article presents the key observations of the study and an overview of the methods involved. It has been written for funders of biomedical and health research and health services, health researchers, and policy makers in those fields. It will also be of interest to those involved in research and impact evaluation.

Who needs what from a national health research system: lessons from reforms to the English Department of Health's R&D system

Health research systems consist of diverse groups who have some role in health research, but the boundaries around such a system are not clear-cut. To explore what various stakeholders need we reviewed the literature including that on the history of English health R&D reforms, and we also applied some relevant conceptual frameworks.We first describe the needs and capabilities of the main groups of stakeholders in health research systems, and explain key features of policymaking systems within which these stakeholders operate in the UK. The five groups are policymakers (and health care managers), health professionals, patients and the general public, industry, and researchers. As individuals and as organisations they have a range of needs from the health research system, but should also develop specific capabilities in order to contribute effectively to the system and benefit from it.Second, we discuss key phases of reform in the development of the English health research system over four decades - especially that of the English Department of Health's R&D system - and identify how far legitimate demands of key stakeholder interests were addressed.Third, in drawing lessons we highlight points emerging from contemporary reports, but also attempt to identify issues through application of relevant conceptual frameworks. The main lessons are: the importance of comprehensively addressing the diverse needs of various interacting institutions and stakeholders; the desirability of developing facilitating mechanisms at interfaces between the health research system and its various stakeholders; and the importance of additional money in being able to expand the scope of the health research system whilst maintaining support for basic science.We conclude that the latest health R&D strategy in England builds on recent progress and tackles acknowledged weaknesses. The strategy goes a considerable way to identifying and more effectively meeting the needs of key groups such as medical academics, patients and industry, and has been remarkably successful in increasing the funding for health research. There are still areas that might benefit from further recognition and resourcing, but the lessons identified, and progress made by the reforms are relevant for the design and coordination of national health research systems beyond England.

Developing the protocol for the evaluation of the health foundation's 'engaging with quality initiative' - an emergent approach

In 2004 a UK charity, The Health Foundation, established the 'Engaging with Quality Initiative' to explore and evaluate the benefits of engaging clinicians in quality improvement in healthcare. Eight projects run by professional bodies or specialist societies were commissioned in various areas of acute care. A developmental approach to the initiative was adopted, accompanied by a two level evaluation: eight project self-evaluations and a related external evaluation. This paper describes how the protocol for the external evaluation was developed. The challenges faced included large variation between and within the projects (in approach, scope and context, and in understanding of quality improvement), the need to support the project teams in their self-evaluations while retaining a necessary objectivity, and the difficulty of evaluating the moving target created by the developmental approach adopted in the initiative. An initial period to develop the evaluation protocol proved invaluable in helping us to explore these issues.

An assessment of the impact of the NHS Health Technology Assessment Programme

OBJECTIVES

To consider how the impact of the NHS Health Technology Assessment (HTA) Programme should be measured. To determine what models are available and their strengths and weaknesses. To assess the impact of the first 10 years of the NHS HTA programme from its inception in 1993 to June 2003 and to identify the factors associated with HTA research that are making an impact.

DATA SOURCES

Main electronic databases from 1990 to June 2005. The documentation of the National Coordinating Centre for Health Technology Assessment (NCCHTA). Questionnaires to eligible researchers. Interviews with lead investigators. Case study documentation.

REVIEW METHODS

A literature review of research programmes was carried out, the work of the NCCHTA was reviewed, lead researchers were surveyed and 16 detailed case studies were undertaken. Each case study was written up using the payback framework. A cross-case analysis informed the analysis of factors associated with achieving payback. Each case study was scored for impact before and after the interview to assess the gain in information due to the interview. The draft write-up of each study was checked with each respondent for accuracy and changed if necessary.

RESULTS

The literature review identified a highly diverse literature but confirmed that the 'payback' framework pioneered by Buxton and Hanney was the most widely used and most appropriate model available. The review also confirmed that impact on knowledge generation was more easily quantified than that on policy, behaviour or especially health gain. The review of the included studies indicated a higher level of impact on policy than is often assumed to occur. The survey showed that data pertinent to payback exist and can be collected. The completed questionnaires showed that the HTA Programme had considerable impact in terms of publications, dissemination, policy and behaviour. It also showed, as expected, that different parts of the Programme had different impacts. The Technology Assessment Reports (TARs) for the National Institute for Health and Clinical Excellence (NICE) had the clearest impact on policy in the form of NICE guidance. Mean publications per project were 2.93 (1.98 excluding the monographs), above the level reported for other programmes. The case studies revealed the large diversity in the levels and forms of impacts and the ways in which they arise. All the NICE TARs and more than half of the other case studies had some impact on policy making at the national level whether through NICE, the National Screening Committee, the National Service Frameworks, professional bodies or the Department of Health. This underlines the importance of having a customer or 'receptor' body. A few case studies had very considerable impact in terms of knowledge production and in informing national and international policies. In some of these the principal investigator had prior expertise and/or a research record in the topic. The case studies confirmed the questionnaire responses but also showed how some projects led to further research.

CONCLUSIONS

This study concluded that the HTA Programme has had considerable impact in terms of knowledge generation and perceived impact on policy and to some extent on practice. This high impact may have resulted partly from the HTA Programme's objectives, in that topics tend to be of relevance to the NHS and have policy customers. The required use of scientific methods, notably systematic reviews and trials, coupled with strict peer reviewing, may have helped projects publish in high-quality peer-reviewed journals. Further research should cover more detailed, comprehensive case studies, as well as enhancement of the 'payback framework'. A project that collated health research impact studies in an ongoing manner and analysed them in a consistent fashion would also be valuable.

Payback arising from research funding: evaluation of the Arthritis Research Campaign

OBJECTIVES

Using a structured evaluation framework to systematically review and document the outputs and outcomes of research funded by the Arthritis Research Campaign in the early 1990s. To illustrate the strengths and weaknesses of different modes of research funding.

METHODS

The payback framework was applied to 16 case studies of research grants funded in the early 1990s. Case study methodology included bibliometric analysis, literature and archival document review and key informant interviews.

RESULTS

A range of research paybacks was identified from the 16 research grants. The payback included 302 peer-reviewed papers, postgraduate training and career development, including 28 PhD/MDs, research informing recommendations in clinical guidelines, improved quality of life for people with RA and the reduction of the likelihood of recurrent miscarriage for women with antiphospholipid syndrome. The payback arising from project grants appeared to be similar to that arising from other modes of funding that were better resourced.

CONCLUSIONS

There is a wide diversity of research payback. Short focused project grants seem to provide value for money.

What British psychiatrists read: questionnaire survey of journal usage among clinicians

BACKGROUND

The role of journals in disseminating research to clinicians is increasingly debated. Current measures of esteem for journals (e.g. impact factors) may not indicate clinical penetration.

AIMS

To assess the perceived importance of different mental health journals to psychiatrists' clinical practice and compare this with impact factors.

METHOD

Random samples of psychiatrists providing child and adolescent, adults of working age and old age services chose up to ten journals read or consulted with regard to their clinical work, ranking the top three. For these journals, comparisons were made with impact factors and importance as outlets for UK psychiatry research.

RESULTS

A total of 560 questionnaires were completed (47%). Two membership journals (the British Journal of Psychiatry and the BMJ) were most read and highest ranked. Associations between impact factors, clinicians' ratings and importance as outlets for psychiatry papers varied.

CONCLUSIONS

The results could lead to reconsideration of the importance of some journals. Academic assessments of the status of journals should not be assumed to reflect their influence on clinicians.

Routine monitoring of performance: what makes health research and development different?

Increasing attention is being directed to measuring and monitoring the use of health-related R&D funding, partly to justify this expenditure and partly to ensure that R&D effort is directed to achieving the paybacks desired by funders. These paybacks include contributing to knowledge, contributing to R&D capacity, political benefits, benefits to the health service and to patients, and more general economic benefits. This paper addresses the issues that must be considered when designing a routine performance management system for health R&D. Conventional methods of routine performance management are often rendered inappropriate in this context by the intangible and unpredictable outcomes of research, which are heterogeneous across projects and programmes and which can be hard to attribute to particular R&D support. Instead, to be effective in this context, a routine system must combine quantitative and qualitative indicators, utilising information from a number of different sources. The system must achieve acceptable levels (defined by the funder) on each of the following criteria: it must measure those dimensions of payback that are valued by the funder; it must be decision-relevant; it must be consistent with truthful compliance; it must minimise perverse incentives; and it must have acceptable net costs. It is vitally important that the system itself generates a positive payback. We illustrate these issues by outlining a system that might be used to monitor the payback from government-funded R&D.

How can payback from health services research be assessed?

Throughout the world there is a growing recognition that health care should be research-led. This strengthens the requirement for expenditure on health services research to be justified by demonstrating the benefits it produces. However, payback from health research and development is a complex concept and little used term. Five main categories of payback can be identified: knowledge; research benefits; political and administrative benefits; health sector benefits; and broader economic benefits. Various models of research utilization together with previous assessments of payback from research helped in the development of a new conceptual model of how and where payback may occur. The model combines an input-output perspective with an examination of the permeable interfaces between research and its environment. The model characterizes research projects in terms of Inputs, Processes, and Primary Outputs. The last consist of knowledge and research benefits. There are two interfaces between the project and its environment. The first (Project Specification, Selection and Commissioning) is the link with Research Needs Assessment. The second (Dissemination) should lead to Secondary Outputs (which are policy or administrative decisions), and usually Applications (which take the form of behavioural changes), from which Impacts or Final Outcomes result. It is at this final stage that health and wider economic benefits can be measured. A series of case studies were used to assess the feasibility both of applying the model and the payback categorization. The paper draws various conclusions from the case studies and identifies a range of issues for further work.