Assessing the benefits of health research: lessons from research into the use of antenatal corticosteroids for the prevention of neonatal respiratory distress syndrome

From COVID-19 research to vaccine application: why might it take 17 months not 17 years and what are the wider lessons?

It is often said that it takes 17 years to move medical research from bench to bedside. In a coronavirus disease (COVID-19) world, such time-lags feel intolerable. In these extraordinary circumstances could years be made into months? If so, could those lessons be used to accelerate medical research when the crisis eases?

To measure time-lags in health and biomedical research as well as to identify ways of reducing them, we developed and published (in 2015) a matrix consisting of overlapping tracks (or stages/phases) in the translation from discovery research to developed products, policies and practice. The matrix aids analysis by highlighting the time and actions required to develop research (and its translation) both (1) along each track and (2) from one track to another, e.g. from the discovery track to the research-in-humans track. We noted four main approaches to reducing time-lags, namely increasing resources, working in parallel, starting or working at risk, and improving processes.

Examining these approaches alongside the matrix helps interpret the enormous global effort to develop a vaccine for the 2019 novel coronavirus SARS-CoV-2, the causative agent of COVID-19. Rapid progress in the discovery/basic and human research tracks is being made through a combination of large-scale funding, work being conducted in parallel (between different teams globally and through working in overlapping tracks), working at greater (but proportionate) risk to safety than usual, and adopting various new processes. The overlapping work of some of the teams involves continuing animal research whilst entering vaccine candidates into Phase I trials alongside planning their Phase II trials. The additional funding available helps to reduce some of the usual financial risks in moving so quickly. Going forward through the increasingly large human trials for safety, dosage and efficacy, it will be vital to overlap work in parallel in the often challenging public policy and clinical tracks. Thus, regulatory and reimbursement bodies are beginning and preparing rapid action to pull vaccines proving to be safe and effective through to extraordinarily rapid application to the general population. Monitoring the development of a COVID-19 vaccine using the matrix (modified as necessary) could help identify which of the approaches speeding development and deployment could be usefully applied more widely in the future.

Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

Healthcare resource allocation decisions made under conditions of uncertainty may turn out to be suboptimal. In a resource constrained system in which there is a fixed budget, these suboptimal decisions will result in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to make better resource allocation decisions. This value can be quantified using a value of information (VOI) analysis. This report, from the ISPOR VOI Task Force, introduces VOI analysis, defines key concepts and terminology, and outlines the role of VOI for supporting decision making, including the steps involved in undertaking and interpreting VOI analyses. The report is specifically aimed at those tasked with making decisions about the adoption of healthcare or the funding of healthcare research. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing the results of VOI analyses.

Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Background

Measuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research.

Methods

Keyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework.

Results

We identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilised forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change.

Conclusions

The Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0310-4) contains supplementary material, which is available to authorized users.

Improving Uptake of Key Perinatal Interventions Using Statewide Quality Collaboratives

Regional and statewide quality improvement collaboratives have been instrumental in implementing evidence-based practices and facilitating quality improvement initiatives within neonatology. Statewide collaboratives emerged from larger collaborative organizations, like the Vermont Oxford Network, and play an increasing role in collecting and interpreting data, setting priorities for improvement, disseminating evidence-based clinical practice guidelines, and creating regional networks for synergistic learning. In this review, we highlight examples of successful statewide collaborative initiatives, as well as challenges that exist in initiating and sustaining collaborative efforts.

Preterm birth: the role of knowledge transfer and exchange

BACKGROUND

Preterm birth (PTB) is the leading cause of death in children under age five. Healthcare policy and other decision-making relevant to PTB may rely on obsolete, incomplete or inapplicable research evidence, leading to worsened outcomes. Appropriate knowledge transfer and exchange (KTE) strategies are an important component of efforts to reduce the global PTB burden. We sought to develop a 'landscape' analysis of KTE strategies currently used in PTB and related contexts, and to make recommendations for optimising programmatic implementation and for future research.

METHODS

In the University of California, San Francisco's Preterm Birth Initiative, we convened a multidisciplinary working group and examined KTE frameworks. After selecting a widely-used, adaptable, theoretically-strong framework we reviewed the literature to identify evidence-based KTE strategies. We analysed KTE approaches focusing on key PTB stakeholders (individuals, families and communities, healthcare providers and policymakers). Guided by the framework, we articulated KTE approaches that would likely improve PTB outcomes. We further applied the KTE framework in developing recommendations.

RESULTS

We selected the Linking Research to Action framework. Searches identified 19 systematic reviews, including two 'reviews of reviews'. Twelve reviews provided evidence for KTE strategies in the context of maternal, neonatal and child health, though not PTB specifically; seven reviews provided 'cross-cutting' evidence that could likely be generalised to PTB contexts. For individuals, families and communities, potentially effective KTE strategies include community-based approaches, 'decision aids', regular discussions with providers and other strategies. For providers, KTE outcomes may be improved through local opinion leaders, electronic reminders, multifaceted strategies and other approaches. Policy decisions relevant to PTB may best be informed through the use of evidence briefs, deliberative dialogues, the SUPPORT tools for evidence-informed policymaking and other strategies. Our recommendations for research addressed knowledge gaps in regard to partner engagement, applicability and context, implementation strategy research, monitoring and evaluation, and infrastructure for sustainable KTE efforts.

CONCLUSIONS

Evidence-based KTE, using strategies appropriate to each stakeholder group, is essential to any effort to improve health at the population level. PTB stakeholders should be fully engaged in KTE and programme planning from its earliest stages, and ideally before planning begins.

Mortality rates in OECD countries converged during the period 1990-2010

AIMS

Since the scientific revolution of the 18th century, human health has gradually improved, but there is no unifying theory that explains this improvement in health. Studies of macrodeterminants have produced conflicting results. Most studies have analysed health at a given point in time as the outcome; however, the rate of improvement in health might be a more appropriate outcome.

METHODS

Twenty-eight OECD member countries were selected for analysis in the period 1990-2010. The main outcomes studied, in six age groups, were the national rates of decrease in mortality in the period 1990-2010. The effects of seven potential determinants on the rates of decrease in mortality were analysed in linear multiple regression models using least squares, controlling for country-specific history constants, which represent the mortality rate in 1990.

RESULTS

The multiple regression analyses started with models that only included mortality rates in 1990 as determinants. These models explained 87% of the intercountry variation in the children aged 1-4 years and 51% in adults aged 55-74 years. When added to the regression equations, the seven determinants did not seem to significantly increase the explanatory power of the equations.

CONCLUSIONS

The analyses indicated a decrease in mortality in all nations and in all age groups. The development of mortality rates in the different nations demonstrated significant catch-up effects. Therefore an important objective of the national public health sector seems to be to reduce the delay between international research findings and the universal implementation of relevant innovations.

Which health research gets used and why? An empirical analysis of 30 cases

BACKGROUND

While health research is considered essential for improving health worldwide, it remains unclear how it is best organized to contribute to health. This study examined research that was part of a Ghanaian-Dutch research program that aimed to increase the likelihood that results would be used by funding research that focused on national research priorities and was led by local researchers. The aim of this study was to map the contribution of this research to action and examine which features of research and translation processes were associated with the use of the results.

METHODS

Using Contribution Mapping, we systematically examined how 30 studies evolved and how results were used to contribute to action. We combined interviews with 113 purposively selected key informants, document analysis and triangulation to map how research and translation processes evolved and contributions to action were realized. After each case was analysed separately, a cross-case analysis was conducted to identify patterns in the association between features of research processes and the use of research.

RESULTS

The results of 20 of the 30 studies were used to contribute to action within 12 months. The priority setting and proposal selection process led to the funding of studies which were from the outset closely aligned with health sector priorities. Research was most likely to be used when it was initiated and conducted by people who were in a position to use their results in their own work. The results of 17 out of 18 of these user-initiated studies were translated into action. Other features of research that appeared to contribute to its use were involving potential key users in formulating proposals and developing recommendations.

CONCLUSIONS

Our study underlines the importance of supporting research that meets locally-expressed needs and that is led by people embedded in the contexts in which results can be used. Supporting the involvement of health sector professionals in the design, conduct and interpretation of research appears to be an especially worthwhile investment.

How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice

BACKGROUND

The time taken, or 'time lags', between biomedical/health research and its translation into health improvements is receiving growing attention. Reducing time lags should increase rates of return to such research. However, ways to measure time lags are under-developed, with little attention on where time lags arise within overall timelines. The process marker model has been proposed as a better way forward than the current focus on an increasingly complex series of translation 'gaps'. Starting from that model, we aimed to develop better methods to measure and understand time lags and develop ways to identify policy options and produce recommendations for future studies.

METHODS

Following reviews of the literature on time lags and of relevant policy documents, we developed a new approach to conduct case studies of time lags. We built on the process marker model, including developing a matrix with a series of overlapping tracks to allow us to present and measure elements within any overall time lag. We identified a reduced number of key markers or calibration points and tested our new approach in seven case studies of research leading to interventions in cardiovascular disease and mental health. Finally, we analysed the data to address our study's key aims.

RESULTS

The literature review illustrated the lack of agreement on starting points for measuring time lags. We mapped points from policy documents onto our matrix and thus highlighted key areas of concern, for example around delays before new therapies become widely available. Our seven completed case studies demonstrate we have made considerable progress in developing methods to measure and understand time lags. The matrix of overlapping tracks of activity in the research and implementation processes facilitated analysis of time lags along each track, and at the cross-over points where the next track started. We identified some factors that speed up translation through the actions of companies, researchers, funders, policymakers, and regulators. Recommendations for further work are built on progress made, limitations identified and revised terminology.

CONCLUSIONS

Our advances identify complexities, provide a firm basis for further methodological work along and between tracks, and begin to indicate potential ways of reducing lags.

Estimating the returns to UK publicly funded cancer-related research in terms of the net value of improved health outcomes

BACKGROUND

Building on an approach developed to assess the economic returns to cardiovascular research, we estimated the economic returns from UK public and charitable funded cancer-related research that arise from the net value of the improved health outcomes.

METHODS

To assess these economic returns from cancer-related research in the UK we estimated: 1) public and charitable expenditure on cancer-related research in the UK from 1970 to 2009; 2) net monetary benefit (NMB), that is, the health benefit measured in quality adjusted life years (QALYs) valued in monetary terms (using a base-case value of a QALY of GB£25,000) minus the cost of delivering that benefit, for a prioritised list of interventions from 1991 to 2010; 3) the proportion of NMB attributable to UK research; 4) the elapsed time between research funding and health gain; and 5) the internal rate of return (IRR) from cancer-related research investments on health benefits. We analysed the uncertainties in the IRR estimate using sensitivity analyses to illustrate the effect of some key parameters.

RESULTS

In 2011/12 prices, total expenditure on cancer-related research from 1970 to 2009 was £15 billion. The NMB of the 5.9 million QALYs gained from the prioritised interventions from 1991 to 2010 was £124 billion. Calculation of the IRR incorporated an estimated elapsed time of 15 years. We related 17% of the annual NMB estimated to be attributable to UK research (for each of the 20 years 1991 to 2010) to 20 years of research investment 15 years earlier (that is, for 1976 to 1995). This produced a best-estimate IRR of 10%, compared with 9% previously estimated for cardiovascular disease research. The sensitivity analysis demonstrated the importance of smoking reduction as a major source of improved cancer-related health outcomes.

CONCLUSIONS

We have demonstrated a substantive IRR from net health gain to public and charitable funding of cancer-related research in the UK, and further validated the approach that we originally used in assessing the returns from cardiovascular research. In doing so, we have highlighted a number of weaknesses and key assumptions that need strengthening in further investigations. Nevertheless, these cautious estimates demonstrate that the returns from past cancer research have been substantial, and justify the investments made during the period 1976 to 1995.

Identification of potential opinion leaders in child health promotion in Sweden using network analysis

BACKGROUND

Opinion leaders are often local individuals with high credibility who can influence other people. Robust effects using opinion leaders in diffusing innovations have been shown in several randomized controlled trials, for example regarding sexually transmitted infections (STI), human immunodeficiency virus (HIV) prevention, mammography rates and caesarean birth delivery rates. In a Cochrane review 2010 it was concluded that the use of opinion leaders can successfully promote evidence-based practice. Thus, using opinion leaders within the public health sector might be one means to speed up the dissemination of health promoting and disease preventing innovations. Social network analysis has been used to trace and map networks, with focus on relationships and positions, in widely spread arenas and topics. The purpose of this study was to use social network analysis in order to identify potential opinion leaders at the arena of child health promotion in Sweden.

RESULTS

By using snowball technique a short e-mail question was spread in up to five links, starting from seven initially invited persons. This inquiry resulted in a network consisting of 153 individuals. The most often mentioned actors were researchers, public health officials and paediatricians, or a combination of these professions. Four single individuals were mentioned by five to seven other persons in the network. These individuals obviously possess qualities that make other professionals within the public health sector listen to and trust them.

CONCLUSIONS

Social network analysis seemed to be a useful method to identify influential persons with high credibility, i.e. potential opinion leaders, at the arena of child health promotion in Sweden. If genuine opinion leaders could be identified directed measures can be carried out in order to spread new and relevant knowledge. This may facilitate for public health actors at the local, regional and national level to more rapidly progress innovations into everyday practice. However, effectiveness studies of opinion leaders in the public health sector still have to be performed.

Measuring the payback of research activities: a feasible ex-post evaluation methodology in epidemiology and public health

Most ex-post evaluations of research funding programs are based on bibliometric methods and, although this approach has been widely used, it only examines one facet of the project's impact, that is, scientific productivity. More comprehensive models of payback assessment of research activities are designed for large-scale projects with extensive funding. The purpose of this study was to design and implement a methodology for the ex-post evaluation of small-scale projects that would take into account both the fulfillment of projects' stated objectives as well as other wider benefits to society as payback measures. We used a two-phase ex-post approach to appraise impact for 173 small-scale projects funded in 2007 and 2008 by a Spanish network center for research in epidemiology and public health. In the internal phase we used a questionnaire to query the principal investigator (PI) on the outcomes as well as actual and potential impact of each project; in the external phase we sent a second questionnaire to external reviewers with the aim of assessing (by peer-review) the performance of each individual project. Overall, 43% of the projects were rated as having completed their objectives "totally", and 40% "considerably". The research activities funded were reported by PIs as socially beneficial their greatest impact being on research capacity (50% of payback to society) and on knowledge translation (above 11%). The method proposed showed a good discriminating ability that makes it possible to measure, reliably, the extent to which a project's objectives were met as well as the degree to which the project contributed to enhance the group's scientific performance and of its social payback.

Ways of assessing the economic value or impact of research: is it a step too far for nursing research?

To identify lessons that could be applied to nursing research, this paper draws on some reviews of the increasing, although apparently still relatively small, number of studies that attempt to assess economic impacts from health research. One review describes several important steps, including identifying the health and other outcomes from specific bodies of research and then valuing the outcomes. We describe major studies in fields such as cardiovascular research that show how the economic value of health research can be demonstrated. In addition, we examine various nursing studies, including ones showing the benefits (especially economic) from nursing itself (as opposed to from nursing research), and also studies describing economic evaluations of new devices and techniques used by nurses, which have the potential to be used when trying to value the research. Currently, such studies rarely go on and demonstrate how the nursing research has had a wider impact on nursing policies and practice, and hence led to outcomes that could be valued. There is, nevertheless, scope to build on these existing nursing studies. Conducting impact assessments could potentially result in a portfolio of examples of nursing research that have informed policies and practices and led to economic impacts in terms of cost savings, and possibly also in terms of health gains that could be valued.

Setting priorities in clinical and health services research: Properties of an adapted and updated method

Objectives: The objectives of this study is to review the set of criteria of the Institute of Medicine (IOM) for priority-setting in research with addition of new criteria if necessary, and to develop and evaluate the reliability and validity of the final priority score.

Methods: Based on the evaluation of 199 research topics, forty-five experts identified additional criteria for priority-setting, rated their relevance, and ranked and weighted them in a three-round modified Delphi technique. A final priority score was developed and evaluated. Internal consistency, test–retest and inter-rater reliability were assessed. Correlation with experts’ overall qualitative topic ratings were assessed as an approximation to validity.

Results: All seven original IOM criteria were considered relevant and two new criteria were added (“potential for translation into practice”, and “need for knowledge”). Final ranks and relative weights differed from those of the original IOM criteria: “research impact on health outcomes” was considered the most important criterion (4.23), as opposed to “burden of disease” (3.92). Cronbach's alpha (0.75) and test–retest stability (interclass correlation coefficient = 0.66) for the final set of criteria were acceptable. The area under the receiver operating characteristic curve for overall assessment of priority was 0.66.

Conclusions: A reliable instrument for prioritizing topics in clinical and health services research has been developed. Further evaluation of its validity and impact on selecting research topics is required.

Systematic review of methods for evaluating healthcare research economic impact

BACKGROUND

The economic benefits of healthcare research require study so that appropriate resources can be allocated to this research, particularly in developing countries. As a first step, we performed a systematic review to identify the methods used to assess the economic impact of healthcare research, and the outcomes.

METHOD

An electronic search was conducted in relevant databases using a combination of specific keywords. In addition, 21 relevant Web sites were identified.

RESULTS

The initial search yielded 8,416 articles. After studying titles, abstracts, and full texts, 18 articles were included in the analysis. Eleven other reports were found on Web sites. We found that the outcomes assessed as healthcare research payback included direct cost-savings, cost reductions in healthcare delivery systems, benefits from commercial advancement, and outcomes associated with improved health status. Two methods were used to study healthcare research payback: macro-economic studies, which examine the relationship between research studies and economic outcome at the aggregated level, and case studies, which examine specific research projects to assess economic impact.

CONCLUSIONS

Our study shows that different methods and outcomes can be used to assess the economic impacts of healthcare research. There is no unique methodological approach for the economic evaluation of such research. In our systematic search we found no research that had evaluated the economic return of research in low and middle income countries. We therefore recommend a consensus on practical guidelines at international level on the basis of more comprehensive methodologies (such as Canadian Academic of Health Science and payback frameworks) in order to build capacity, arrange for necessary informative infrastructures and promote necessary skills for economic evaluation studies.

[Developing and applying the Payback Framework to assess the socioeconomic impact of health research]

There is increasing pressure for assessments of the wider socioeconomic impacts of health research. Governments are making greater demands to justify the expenditure of public money. However, there is also a belief that assessing how the wider effects or benefits of health research arise should help to inform the management and organization of health research so as to increase future impacts. Since the mid- 1990s, Buxton and Hanney at the Health Economics Research Group, Brunel University, have been developing and applying the Payback Framework to assess the impacts of health research. Together with their colleagues, these researchers have applied this model in a series of studies to assess the payback from research programs in various fields (including diabetes, arthritis and cardiovascular disease) and in various countries (including the United Kingdom, The Netherlands, Ireland, Australia and Canada). Other teams of researchers have applied the Payback Framework in, for example, Spain and Hong Kong. The Payback Framework consists of two elements, the first being the multi-dimensional categorization of the benefits of health research, which covers five main categories ranging from traditional knowledge production and research training and targeting, to impacts on policy and product development through to health and economic gains. The second element is a logic model of how best to assess these impacts. Application of this framework can be resource intensive, but has provided illustrative 'good news' stories on the payback resulting from research and has helped to inform research management.

Access, sources and value of new medical information: views of final year medical students at the University of Nairobi

OBJECTIVES

To evaluate final year medical students' access to new medical information.

METHOD

Cross-sectional survey of final year medical students at the University of Nairobi using anonymous, self-administered questionnaires.

RESULTS

Questionnaires were distributed to 85% of a possible 343 students and returned by 44% (152). Half reported having accessed some form of new medical information within the previous 12 months, most commonly from books and the internet. Few students reported regular access; and specific, new journal articles were rarely accessed. Absence of internet facilities, slow internet speed and cost impeded access to literature; and current training seems rarely to encourage students to seek new information.

CONCLUSION

Almost half the students had not accessed any new medical information in their final year in medical school. This means they are ill prepared for a career that may increasingly demand life-long, self-learning.

A systematic evaluation of payback of publicly funded health and health services research in Hong Kong

BACKGROUND

The Health and Health Services Research Fund (HHSRF) is dedicated to support research related to all aspects of health and health services in Hong Kong. We evaluated the fund's outcomes and explored factors associated with the translation of research findings to changes in health policy and provider behaviour.

METHODS

A locally suitable questionnaire was developed based on the "payback" evaluation framework and was sent to principal investigators of the completed research projects supported by the fund since 1993. Research "payback" in six outcome areas was surveyed, namely knowledge production, use of research in the research system, use of research project findings in health system policy/decision making, application of the research findings through changed behaviour, factors influencing the utilization of research, and health/health service/economic benefits.

RESULTS

Principal investigators of 178 of 205 (87%) completed research projects returned the questionnaire. Investigators reported research publications in 86.5% (mean = 5.4 publications per project), career advancement 34.3%, acquisition of higher qualifications 38.2%, use of results in policy making 35.4%, changed behaviour in light of findings 49.4%, evidence of health service benefit 42.1% and generated subsequent research in 44.9% of the projects. Payback outcomes were positively associated with the amount of funding awarded. Multivariate analysis found participation of investigators in policy committees and liaison with potential users were significantly associated with reported health service benefit (odds ratio [OR]participation = 2.86, 95% confidence interval [CI] 1.28-6.40; ORliaison = 2.03, 95% CI 1.05-3.91), policy and decision-making (ORparticipation = 10.53, 95% CI 4.13-26.81; ORliaison = 2.52, 95% CI 1.20-5.28), and change in behavior (ORparticipation = 3.67, 95% CI 1.53-8.81).

CONCLUSION

The HHSRF has produced substantial outcomes and compared favourably with similar health research funds in other developed economies. Further studies are needed to better understand the factors and pathways associated with the translation of research findings into practice.

The information sources and journals consulted or read by UK paediatricians to inform their clinical practice and those which they consider important: a questionnaire survey

Background

Implementation of health research findings is important for medicine to be evidence-based. Previous studies have found variation in the information sources thought to be of greatest importance to clinicians but publication in peer-reviewed journals is the traditional route for dissemination of research findings. There is debate about whether the impact made on clinicians should be considered as part of the evaluation of research outputs. We aimed to determine first which information sources are generally most consulted by paediatricians to inform their clinical practice, and which sources they considered most important, and second, how many and which peer-reviewed journals they read.

Methods

We enquired, by questionnaire survey, about the information sources and academic journals that UK medical paediatric specialists generally consulted, attended or read and considered important to their clinical practice.

Results

The same three information sources – professional meetings & conferences, peer-reviewed journals and medical colleagues – were, overall, the most consulted or attended and ranked the most important. No one information source was found to be of greatest importance to all groups of paediatricians. Journals were widely read by all groups, but the proportion ranking them first in importance as an information source ranged from 10% to 46%. The number of journals read varied between the groups, but Archives of Disease in Childhood and BMJ were the most read journals in all groups. Six out of the seven journals previously identified as containing best paediatric evidence are the most widely read overall by UK paediatricians, however, only the two most prominent are widely read by those based in the community.

Conclusion

No one information source is dominant, therefore a variety of approaches to Continuing Professional Development and the dissemination of research findings to paediatricians should be used. Journals are an important information source. A small number of key ones can be identified and such analysis could provide valuable additional input into the evaluation of clinical research outputs.

Prioridades de investigación en servicios sanitarios en el Sistema Nacional de Salud. Una aproximación por consenso de expertos

OBJECTIVES

The aim of the present paper was to identify and prioritise health services research issues, from the perspective of Spanish NHS enrollees' health needs.

METHODS

13 experts attended to a structured consensual meeting by using a two-phase process, sharing both Nominal Group and Delphi Technique features. Health Services Research Priorities were identified and scored (7 to 9, high relevance; 4 to 6 mild relevance and 1 to 3, low or no relevance). Results were sorted regarding the median score and its dispersion.

RESULTS

53 research issues were identified. Strategies to improve effectiveness in health care, information to patients, health care quality and safety, equity, access and sustainability were considered of high priority and high consensus; just 4 items had remarkable dispersion (Intercuartile interval [ICI] > or = 2). Organizational improvement, productivity and efficiency enhancement, workforce and equipments were classified of mild priority. Additionally, this group of items reached very low consensus: 59% of them reached ICI > or = 2.

CONCLUSIONS

Our approach offers reference about priorities to Spanish health services researchers, and also, orientation to financers and users of this kind of research, though several limitations have been described.

Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's 'Arthritis Research Campaign'

Background

External and internal factors are increasingly encouraging research funding bodies to demonstrate the outcomes of their research. Traditional methods of assessing research are still important, but can be merged into broader multi-dimensional categorisations of research benefits. The onus has hitherto been on public sector funding bodies, but in the UK the role of medical charities in funding research is particularly important and the Arthritis Research Campaign, the leading medical charity in its field in the UK, commissioned a study to identify the outcomes from research that it funds. This article describes the methods to be used.

Methods

A case study approach will enable narratives to be told, illuminating how research funded in the early 1990s was (or was not) translated into practice. Each study will be organised using a common structure, which, with careful selection of cases, should enable cross-case analysis to illustrate the strengths of different modes and categories of research. Three main interdependent methods will be used: documentary and literature review; semi-structured interviews; and bibliometric analysis. The evaluative framework for organising the studies was previously used for assessing the benefits from health services research. Here, it has been specifically amended for a medical charity that funds a wide range of research and is concerned to develop the careers of researchers. It was further refined in three pilot studies. The framework has two main elements. First, a multi-dimensional categorisation of benefits going from the knowledge produced in peer reviewed journal articles through to the health and potential economic gain. The second element is a logic model, which, with various stages, should provide a way of organising the studies. The stock of knowledge is important: much research, especially basic, will feed into it and influence further research rather than directly lead to health gains. The cross-case analysis will look for factors associated with outcomes.

Conclusions

The pilots confirmed the applicability of the methods for a full study which should assist the Arthritis Research Campaign to demonstrate the outcomes from its funding, and provide it with evidence to inform its own policies.