Use of research to inform public policymaking

Community capacity to acquire, assess, adapt, and apply research evidence: a survey of Ontario's HIV/AIDS sector

Background

Community-based organizations (CBOs) are important stakeholders in health systems and are increasingly called upon to use research evidence to inform their advocacy, program planning, and service delivery. To better support CBOs to find and use research evidence, we sought to assess the capacity of CBOs in the HIV/AIDS sector to acquire, assess, adapt, and apply research evidence in their work.

Methods

We invited executive directors of HIV/AIDS CBOs in Ontario, Canada (n = 51) to complete the Canadian Health Services Research Foundation's "Is Research Working for You?" survey.

Findings

Based on responses from 25 organizations that collectively provide services to approximately 32,000 clients per year with 290 full-time equivalent staff, we found organizational capacity to acquire, assess, adapt, and apply research evidence to be low. CBO strengths include supporting a culture that rewards flexibility and quality improvement, exchanging information within their organization, and ensuring that their decision-making processes have a place for research. However, CBO Executive Directors indicated that they lacked the skills, time, resources, incentives, and links with experts to acquire research, assess its quality and reliability, and summarize it in a user-friendly way.

Conclusion

Given the limited capacity to find and use research evidence, we recommend a capacity-building strategy for HIV/AIDS CBOs that focuses on providing the tools, resources, and skills needed to more consistently acquire, assess, adapt, and apply research evidence. Such a strategy may be appropriate in other sectors and jurisdictions as well given that CBO Executive Directors in the HIV/AIDS sector in Ontario report low capacity despite being in the enviable position of having stable government infrastructure in place to support them, benefiting from long-standing investment in capacity building, and being part of an active provincial network. CBOs in other sectors and jurisdictions that have fewer supports may have comparable or lower capacity. Future research should examine a larger sample of CBO Executive Directors from a range of sectors and jurisdictions.

Use of a knowledge synthesis by decision makers and planners to facilitate system level integration in a large Canadian provincial health authority

Purpose

The study is an examination of how a knowledge synthesis, conducted to fill an information gap identified by decision makers and planners responsible for integrating health systems in a western Canadian health authority, is being used within that organization.

Methods

Purposive sampling and snowball technique were used to identify 13 participants who were interviewed about how they are using the knowledge synthesis for health services planning and decision-making.

Results

The knowledge synthesis is used by those involved in the strategic direction of the provincial healthcare organization and those tasked with the operationalization of integration at the provincial or local level. Both groups most frequently use the 10 key principles for integration, followed by the sections on integration processes, strategies and models. The key principles facilitate discussion on priority areas to be considered and provide a reference point for a desired future state. Perceived information gaps relate to a lack of detail on ‘how to’ strategies, tools and processes that would lead to successful integration.

Discussion and conclusion

The current project demonstrates that decision makers and planners will effectively use a knowledge synthesis if it is timely, relevant and accessible. The information can be applied at strategic and operations levels. Attention needs to be paid to include more information on implementation strategies and processes. Including knowledge users in identifying research questions will increase information uptake.

Making targeted screening for infant hearing loss an effective option in less developed countries

Developing countries account for a disproportionate burden of infant hearing loss globally but the prospects of the more ideal universal newborn hearing screening (UNHS) have been debated. The Joint Committee on Infant Hearing (JCIH) of USA has consistently proposed targeted newborn hearing screening (TNHS) for such countries. This study therefore set out to examine the appropriateness of JCIH risk factors as a basis for TNHS in Sub-Saharan Africa and Southeast Asia. From a review of relevant literature published in PubMed in the last 10 years, evidence on the effectiveness of TNHS based on JCIH or other risk factors is sparse or limited. Consistent with the prevailing epidemiological profile of these countries additional putative risk factors not listed or more prevalent than those listed by JCIH such as maternal hypertensive disorders in pregnancy, lack of skilled attendant at delivery, non-elective cesarean delivery and infant undernutrition have been demonstrated besides consanguinity. While TNHS has intuitive appeal in resource-poor settings, it is likely to be fraught with diverse operational constraints that could significantly curtail its effectiveness in these two regions. Well-conducted pilot UNHS studies to determine context-specific risk factors, screening efficiency and the potential trade-offs are warranted in each country prior to embarking on TNHS where UNHS is not immediately practicable.

Evidence summaries tailored to health policy-makers in low- and middle-income countries

OBJECTIVE

To describe how the SUPPORT collaboration developed a short summary format for presenting the results of systematic reviews to policy-makers in low- and middle-income countries (LMICs).

METHODS

We carried out 21 user tests in six countries to explore users' experiences with the summary format. We modified the summaries based on the results and checked our conclusions through 13 follow-up interviews. To solve the problems uncovered by the user testing, we also obtained advisory group feedback and conducted working group workshops.

FINDINGS

Policy-makers liked a graded entry format (i.e. short summary with key messages up front). They particularly valued the section on the relevance of the summaries for LMICs, which compensated for the lack of locally-relevant detail in the original review. Some struggled to understand the text and numbers. Three issues made redesigning the summaries particularly challenging: (i) participants had a poor understanding of what a systematic review was; (ii) they expected information not found in the systematic reviews and (iii) they wanted shorter, clearer summaries. Solutions included adding information to help understand the nature of a systematic review, adding more references and making the content clearer and the document quicker to scan.

CONCLUSION

Presenting evidence from systematic reviews to policy-makers in LMICs in the form of short summaries can render the information easier to assimilate and more useful, but summaries must be clear and easy to read or scan quickly. They should also explain the nature of the information provided by systematic reviews and its relevance for policy decisions.

Supporting the use of health technology assessments in policy making about health systems

Objectives:The objective of this study is to profile the health technology assessments (HTAs) produced in Canada and other selected countries and assess their potential to inform policy making about health systems in jurisdictions other than the ones for which they were produced, and to develop and pilot test prototypes for packaging and assessing the relevance of HTAs for health system managers and policy makers.

Methods:We compiled an inventory of all HTAs that were produced by nine HTA agencies between September 2003 and August 2006; coded the title and abstract of each HTA according to the technologies assessed, methods used, and whether or not context-specific actionable messages were provided; developed a prototype for a structured, decision-relevant HTA summary and for a relevance-assessment form; and pilot-tested the prototypes using semistructured telephone interviews with a purposive sample of Canadian healthcare managers and policy makers.

Results:Our review of the 223 HTAs identified that: (i) 44 HTAs addressed health system arrangements (20 percent); (ii) 205 incorporated a systematic review (92 percent), whereas only 12 incorporated a sociopolitical assessment using explicit methods (5 percent); and (iii) 50 contained context-specific actionable messages (22 percent). Our interviews identified significant support for both the general idea of an HTA summary and the prototype's specific elements, but mixed views about using peer assessments of relevance.

Conclusions:Those involved in supporting the use of HTAs in policy making about health systems may wish to produce structured decision-relevant summaries for their systematic review-containing HTAs to increase the prospects for their HTAs being used outside the jurisdiction for which they were produced.

The relationship between economic evaluations and HIV and AIDS treatment policies

PURPOSE OF REVIEW

Economics, and specifically economic evaluations, are increasingly being utilized to provide treatment policy guidance to decision makers. This article reviews work that has contributed to understanding of the relationship.

RECENT FINDINGS

There is a paucity of research explicitly investigating the association between economic evaluations and HIV and AIDS treatment policy. Where it does exist, it is weak. Factors contributing to the limited impact include lack of reliable and trusted data; absence of local cost-effectiveness data for different interventions; contradictory results; challenges associated with understanding complex economic/mathematical models; inefficient implementation of HIV and AIDS policies; inability to pursue long-term health planning needs; and political will.

SUMMARY

Consideration of the ways in which economic evaluations can have greater influence over HIV and AIDS policies is needed. The weak relationship between the two reflects the complicated and multifaceted decision-making process that is often influenced by socioeconomic and political factors. If an economic evaluation is to influence policy, then cognizance of this is important. Extending the economic toolkit to include broader-based models that incorporate political economy variables, but do not compromise on comprehension, validity and robustness, will offer better informed policy recommendations.

A knowledge management tool for public health: health-evidence.ca

BACKGROUND

The ultimate goal of knowledge translation and exchange (KTE) activities is to facilitate incorporation of research knowledge into program and policy development decision making. Evidence-informed decision making involves translation of the best available evidence from a systematically collected, appraised, and analyzed body of knowledge. Knowledge management (KM) is emerging as a key factor contributing to the realization of evidence-informed public health decision making. The goal of health-evidence.ca is to promote evidence-informed public health decision making through facilitation of decision maker access to, retrieval, and use of the best available synthesized research evidence evaluating the effectiveness of public health interventions.

METHODS

The systematic reviews that populate health evidence.ca are identified through an extensive search (1985-present) of 7 electronic databases: MEDLINE, EMBASE, CINAHL, PsycINFO, Sociological Abstracts, BIOSIS, and SportDiscus; handsearching of over 20 journals; and reference list searches of all relevant reviews. Reviews are assessed for relevance and quality by two independent reviewers. Commonly-used public health terms are used to assign key words to each review, and project staff members compose short summaries highlighting results and implications for policy and practice.

RESULTS

As of June 2010, there are 1913 reviews in the health-evidence.ca registry in 21 public health and health promotion topic areas. Of these, 78% have been assessed as being of strong or moderate methodological quality. Health-evidence.ca receives approximately 35,000 visits per year, 20,596 of which are unique visitors, representing approximately 100 visits per day. Just under half of all visitors return to the site, with the average user spending six minutes and visiting seven pages per visit. Public health nurses, program managers, health promotion workers, researchers, and program coordinators are among the largest groups of registered users, followed by librarians, dieticians, medical officers of health, and nutritionists. The majority of users (67%) access the website from direct traffic (e.g., have the health-evidence.ca webpage bookmarked, or type it directly into their browser).

CONCLUSIONS

Consistent use of health-evidence.ca and particularly the searching for reviews that correspond with current public health priorities illustrates that health-evidence.ca may be playing an important role in achieving evidence-informed public health decision making.

National policy-makers speak out: are researchers giving them what they need?

The objective of this empirical study was to understand the perspectives and attitudes of policy-makers towards the use and impact of research in the health sector in low- and middle-income countries. The study used data from 83 semi-structured, in-depth interviews conducted with purposively selected policy-makers at the national level in Argentina, Egypt, Iran, Malawi, Oman and Singapore. The interviews were structured around an interview guide developed based on existing literature and in consultation with all six country investigators. Transcripts were processed using a thematic-analysis approach. Policy-makers interviewed for this study were unequivocal in their support for health research and the high value they attribute to it. However, they stated that there were structural and informal barriers to research contributing to policy processes, to the contribution research makes to knowledge generally, and to the use of research in health decision-making specifically. Major findings regarding barriers to evidence-based policy-making included poor communication and dissemination, lack of technical capacity in policy processes, as well as the influence of the political context. Policy-makers had a variable understanding of economic analysis, equity and burden of disease measures, and were vague in terms of their use in national decisions. Policy-maker recommendations regarding strategies for facilitating the uptake of research into policy included improving the technical capacity of policy-makers, better packaging of research results, use of social networks, and establishment of fora and clearinghouse functions to help assist in evidence-based policy-making.

Systematic reviews, systematic error and the acquisition of clinical knowledge

BACKGROUND

Since its inception, evidence-based medicine and its application through systematic reviews, has been widely accepted. However, it has also been strongly criticised and resisted by some academic groups and clinicians. One of the main criticisms of evidence-based medicine is that it appears to claim to have unique access to absolute scientific truth and thus devalues and replaces other types of knowledge sources.

DISCUSSION

The various types of clinical knowledge sources are categorised on the basis of Kant's categories of knowledge acquisition, as being either 'analytic' or 'synthetic'. It is shown that these categories do not act in opposition but rather, depend upon each other. The unity of analysis and synthesis in knowledge acquisition is demonstrated during the process of systematic reviewing of clinical trials. Systematic reviews constitute comprehensive synthesis of clinical knowledge but depend upon plausible, analytical hypothesis development for the trials reviewed. The dangers of systematic error regarding the internal validity of acquired knowledge are highlighted on the basis of empirical evidence. It has been shown that the systematic review process reduces systematic error, thus ensuring high internal validity. It is argued that this process does not exclude other types of knowledge sources. Instead, amongst these other types it functions as an integrated element during the acquisition of clinical knowledge.

CONCLUSIONS

The acquisition of clinical knowledge is based on interaction between analysis and synthesis. Systematic reviews provide the highest form of synthetic knowledge acquisition in terms of achieving internal validity of results. In that capacity it informs the analytic knowledge of the clinician but does not replace it.

Bridging the gaps between research, policy and practice in low- and middle-income countries: a survey of researchers

BACKGROUND

Many international statements have urged researchers, policy-makers and health care providers to collaborate in efforts to bridge the gaps between research, policy and practice in low- and middle-income countries. We surveyed researchers in 10 countries about their involvement in such efforts.

METHODS

We surveyed 308 researchers who conducted research on one of four clinical areas relevant to the Millennium Development Goals (prevention of malaria, care of women seeking contraception, care of children with diarrhea and care of patients with tuberculosis) in each of 10 low- and middle-income countries (China, Ghana, India, Iran, Kazakhstan, Laos, Mexico, Pakistan, Senegal and Tanzania). We focused on their engagement in three promising bridging activities and examined system-level, organizational and individual correlates of these activities.

RESULTS

Less than half of the researchers surveyed reported that they engaged in one or more of the three promising bridging activities: 27% provided systematic reviews of the research literature to their target audiences, 40% provided access to a searchable database of research products on their topic, and 43% established or maintained long-term partnerships related to their topic with representatives of the target audience. Three factors emerged as statistically significant predictors of respondents' engagement in these activities: the existence of structures and processes to link researchers and their target audiences predicted both the provision of access to a database (odds ratio [OR] 2.62, 95% CI 1.30-5.27) and the establishment or maintenance of partnerships (OR 2.65, 95% CI 1.25-5.64); stability in their contacts predicted the provision of systematic reviews (OR 2.88, 95% CI 1.35-6.13); and having managers and public (government) policy-makers among their target audiences predicted the provision of both systematic reviews (OR 4.57, 95% CI 1.78-11.72) and access to a database (OR 2.55, 95% CI 1.20-5.43).

INTERPRETATION

Our findings suggest potential areas for improvement in light of the bridging strategies targeted at health care providers that have been found to be effective in some contexts and the factors that appear to increase the prospects for using research in policy-making.

Community-based knowledge transfer and exchange: helping community-based organizations link research to action

BACKGROUND

Community-based organizations (CBOs) are important stakeholders in health systems and are increasingly called upon to use research evidence to inform their advocacy, program planning, and service delivery efforts. CBOs increasingly turn to community-based research (CBR) given its participatory focus and emphasis on linking research to action. In order to further facilitate the use of research evidence by CBOs, we have developed a strategy for community-based knowledge transfer and exchange (KTE) that helps CBOs more effectively link research evidence to action. We developed the strategy by: outlining the primary characteristics of CBOs and why they are important stakeholders in health systems; describing the concepts and methods for CBR and for KTE; comparing the efforts of CBR to link research evidence to action to those discussed in the KTE literature; and using the comparison to develop a framework for community-based KTE that builds on both the strengths of CBR and existing KTE frameworks.

DISCUSSION

We find that CBR is particularly effective at fostering a climate for using research evidence and producing research evidence relevant to CBOs through community participation. However, CBOs are not always as engaged in activities to link research evidence to action on a larger scale or to evaluate these efforts. Therefore, our strategy for community-based KTE focuses on: an expanded model of 'linkage and exchange' (i.e., producers and users of researchers engaging in a process of asking and answering questions together); a greater emphasis on both producing and disseminating systematic reviews that address topics of interest to CBOs; developing a large-scale evidence service consisting of both 'push' efforts and efforts to facilitate 'pull' that highlight actionable messages from community relevant systematic reviews in a user-friendly way; and rigorous evaluations of efforts for linking research evidence to action.

SUMMARY

Through this type of strategy, use of research evidence for CBO advocacy, program planning, and service delivery efforts can be better facilitated and continually refined through ongoing evaluations of its impact.

Ten key principles for successful health systems integration

Integrated health systems are considered part of the solution to the challenge of sustaining Canada's healthcare system. This systematic literature review was undertaken to guide decision-makers and others to plan for and implement integrated health systems. This review identified 10 universal principles of successfully integrated healthcare systems that may be used by decision-makers to assist with integration efforts. These principles define key areas for restructuring and allow organizational flexibility and adaptation to local context. The literature does not contain a one-size-fits-all model or process for successful integration, nor is there a firm empirical foundation for specific integration strategies and processes.

Why isn't more injury prevention evidence-based?

The focus on evidence-based practice is critical to addressing the issue of injuries, yet advances in the science of injury prevention have not always led to advances in practice. Effective approaches are not always adopted, or when adopted and transferred from one setting to another, they do not always achieve expected results. These challenges were the basis of two breakout sessions at the second European Injury Control and Safety Promotion Conference in Paris, France (October 2008). In summarising the key issues raised during those sessions, this article describes what is meant by evidence-based practice, discusses why evidence-based practice tends not to occur and considers approaches that may facilitate the adoption and implementation of evidence-based strategies. To address the challenge, specific action is required, both on the part of the research community and those responsible for developing and implementing injury prevention policies and programmes.

In for the long haul: knowledge translation between academic and nonprofit organizations

Although scientists are continually refining existing knowledge and producing new evidence to improve health care and health care delivery, far too little scientific output finds its way into the tool kits of practitioners. Likewise, the questions that clinicians would like to be answered all too rarely get taken up by researchers. In this article we focus on knowledge translation challenges accompanying a longitudinal research program with nonprofit organizations providing direct and indirect health and social services to disadvantaged groups in one region of Canada. Three essential factors influencing authentic and reciprocal knowledge transfer and utilization between nonprofit service providers and researchers are discussed: strong institutional partnerships, the use of skilled knowledge brokers, and the meaningful involvement of frontline personnel.

Translating research into policy: lessons learned from eclampsia treatment and malaria control in three southern African countries

BACKGROUND

Little is known about the process of knowledge translation in low- and middle-income countries. We studied policymaking processes in Mozambique, South Africa and Zimbabwe to understand the factors affecting the use of research evidence in national policy development, with a particular focus on the findings from randomized control trials (RCTs). We examined two cases: the use of magnesium sulphate (MgSO(4)) in the treatment of eclampsia in pregnancy (a clinical case); and the use of insecticide treated bed nets and indoor residual household spraying for malaria vector control (a public health case).

METHODS

We used a qualitative case-study methodology to explore the policy making process. We carried out key informants interviews with a range of research and policy stakeholders in each country, reviewed documents and developed timelines of key events. Using an iterative approach, we undertook a thematic analysis of the data.

FINDINGS

Prior experience of particular interventions, local champions, stakeholders and international networks, and the involvement of researchers in policy development were important in knowledge translation for both case studies. Key differences across the two case studies included the nature of the evidence, with clear evidence of efficacy for MgSO(4 )and ongoing debate regarding the efficacy of bed nets compared with spraying; local researcher involvement in international evidence production, which was stronger for MgSO(4 )than for malaria vector control; and a long-standing culture of evidence-based health care within obstetrics. Other differences were the importance of bureaucratic processes for clinical regulatory approval of MgSO(4), and regional networks and political interests for malaria control. In contrast to treatment policies for eclampsia, a diverse group of stakeholders with varied interests, differing in their use and interpretation of evidence, was involved in malaria policy decisions in the three countries.

CONCLUSION

Translating research knowledge into policy is a complex and context sensitive process. Researchers aiming to enhance knowledge translation need to be aware of factors influencing the demand for different types of research; interact and work closely with key policy stakeholders, networks and local champions; and acknowledge the roles of important interest groups.

SUPPORT Tools for evidence-informed health Policymaking (STP) 8: Deciding how much confidence to place in a systematic review

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. The reliability of systematic reviews of the effects of health interventions is variable. Consequently, policymakers and others need to assess how much confidence can be placed in such evidence. The use of systematic and transparent processes to determine such decisions can help to prevent the introduction of errors and bias in these judgements. In this article, we suggest five questions that can be considered when deciding how much confidence to place in the findings of a systematic review of the effects of an intervention. These are: 1. Did the review explicitly address an appropriate policy or management question? 2. Were appropriate criteria used when considering studies for the review? 3. Was the search for relevant studies detailed and reasonably comprehensive? 4. Were assessments of the studies' relevance to the review topic and of their risk of bias reproducible? 5. Were the results similar from study to study?

SUPPORT Tools for evidence-informed health Policymaking (STP) 14: Organising and using policy dialogues to support evidence-informed policymaking

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. Policy dialogues allow research evidence to be considered together with the views, experiences and tacit knowledge of those who will be involved in, or affected by, future decisions about a high-priority issue. Increasing interest in the use of policy dialogues has been fuelled by a number of factors: 1. The recognition of the need for locally contextualised 'decision support' for policymakers and other stakeholders 2. The recognition that research evidence is only one input into the decision-making processes of policymakers and other stakeholders 3. The recognition that many stakeholders can add significant value to these processes, and 4. The recognition that many stakeholders can take action to address high-priority issues, and not just policymakers. In this article, we suggest questions to guide those organising and using policy dialogues to support evidence-informed policymaking. These are: 1. Does the dialogue address a high-priority issue? 2. Does the dialogue provide opportunities to discuss the problem, options to address the problem, and key implementation considerations? 3. Is the dialogue informed by a pre-circulated policy brief and by a discussion about the full range of factors that can influence the policymaking process? 4. Does the dialogue ensure fair representation among those who will be involved in, or affected by, future decisions related to the issue? 5. Does the dialogue engage a facilitator, follow a rule about not attributing comments to individuals, and not aim for consensus? 6. Are outputs produced and follow-up activities undertaken to support action?

SUPPORT Tools for evidence-informed health Policymaking (STP) 1: What is evidence-informed policymaking?

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. In this article, we discuss the following three questions: What is evidence? What is the role of research evidence in informing health policy decisions? What is evidence-informed policymaking? Evidence-informed health policymaking is an approach to policy decisions that aims to ensure that decision making is well-informed by the best available research evidence. It is characterised by the systematic and transparent access to, and appraisal of, evidence as an input into the policymaking process. The overall process of policymaking is not assumed to be systematic and transparent. However, within the overall process of policymaking, systematic processes are used to ensure that relevant research is identified, appraised and used appropriately. These processes are transparent in order to ensure that others can examine what research evidence was used to inform policy decisions, as well as the judgements made about the evidence and its implications. Evidence-informed policymaking helps policymakers gain an understanding of these processes.

SUPPORT tools for evidence-informed policymaking in health 6: Using research evidence to address how an option will be implemented

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. After a policy decision has been made, the next key challenge is transforming this stated policy position into practical actions. What strategies, for instance, are available to facilitate effective implementation, and what is known about the effectiveness of such strategies? We suggest five questions that can be considered by policymakers when implementing a health policy or programme. These are: 1. What are the potential barriers to the successful implementation of a new policy? 2. What strategies should be considered in planning the implementation of a new policy in order to facilitate the necessary behavioural changes among healthcare recipients and citizens? 3. What strategies should be considered in planning the implementation of a new policy in order to facilitate the necessary behavioural changes in healthcare professionals? 4. What strategies should be considered in planning the implementation of a new policy in order to facilitate the necessary organisational changes? 5. What strategies should be considered in planning the implementation of a new policy in order to facilitate the necessary systems changes?

SUPPORT Tools for evidence-informed health Policymaking (STP) 9: Assessing the applicability of the findings of a systematic review

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers.

Differences between health systems may often result in a policy or programme option that is used in one setting not being feasible or acceptable in another. Or these differences may result in an option not working in the same way in another setting, or even achieving different impacts in another setting. A key challenge that policymakers and those supporting them must face is therefore the need to understand whether research evidence about an option can be applied to their setting. Systematic reviews make this task easier by summarising the evidence from studies conducted in a variety of different settings. Many systematic reviews, however, do not provide adequate descriptions of the features of the actual settings in which the original studies were conducted. In this article, we suggest questions to guide those assessing the applicability of the findings of a systematic review to a specific setting. These are: 1. Were the studies included in a systematic review conducted in the same setting or were the findings consistent across settings or time periods? 2. Are there important differences in on-the-ground realities and constraints that might substantially alter the feasibility and acceptability of an option? 3. Are there important differences in health system arrangements that may mean an option could not work in the same way? 4. Are there important differences in the baseline conditions that might yield different absolute effects even if the relative effectiveness was the same? 5. What insights can be drawn about options, implementation, and monitoring and evaluation? Even if there are reasonable grounds for concluding that the impacts of an option might differ in a specific setting, insights can almost always be drawn from a systematic review about possible options, as well as approaches to the implementation of options and to monitoring and evaluation.

SUPPORT Tools for evidence-informed health Policymaking (STP) 7: Finding systematic reviews

This article is part of a series written for people responsible for making decisions about health policies and programmes and for those who support these decision makers. Systematic reviews are increasingly seen as a key source of information in policymaking, particularly in terms of assisting with descriptions of the impacts of options. Relative to single studies they offer a number of advantages related to understanding impacts and are also seen as a key source of information for clarifying problems and providing complementary perspectives on options. Systematic reviews can be undertaken to place problems in comparative perspective and to describe the likely harms of an option. They also assist with understanding the meanings that individuals or groups attach to a problem, how and why options work, and stakeholder views and experiences related to particular options. A number of constraints have hindered the wider use of systematic reviews in policymaking. These include a lack of awareness of their value and a mismatch between the terms employed by policymakers, when attempting to retrieve systematic reviews, and the terms used by the original authors of those reviews. Mismatches between the types of information that policymakers are seeking, and the way in which authors fail to highlight (or make obvious) such information within systematic reviews have also proved problematic. In this article, we suggest three questions that can be used to guide those searching for systematic reviews, particularly reviews about the impacts of options being considered. These are: 1. Is a systematic review really what is needed? 2. What databases and search strategies can be used to find relevant systematic reviews? 3. What alternatives are available when no relevant review can be found?

The National Strategic Plan of South Africa: what are the prospects of success after the repeated failure of previous AIDS policy?

Hitherto, the story of HIV/AIDS in South Africa is, to a large extent, one of lost opportunities. Whereas the country has one of the worst epidemics in the world, consecutive national AIDS strategies have been repeatedly marked by failure over almost three decades. Understandably, South Africa's most recent HIV/AIDS policy, the HIV & AIDS and STI Strategic Plan for South Africa, 2007-2011 (NSP), has been greeted with general acclaim. However, what are its real prospects of success against the backdrop of the repeated failures of the past? The first objective of this review is to systematically identify the core reasons for past policy failures. Using a comprehensive analytical framework, this article presents a systematic review of the literature on postapartheid AIDS policy in South Africa. The analysis demonstrates that a complex interplay among the content, context, actors and process of AIDS policy created a gap between policy making and policy implementation, which rendered near-ideal AIDS policies ineffective. Secondly, we evaluate the chances of success of the current NSP by examining both the policy-making phase and the resulting policy document in light of the reasons for past policy failures. Our analysis shows that the NSP contains dynamic and comprehensive policy content, sensitive to the socio-economic and cultural dimensions of HIV/AIDS. However, many of the political actors that hampered treatment implementation in the past, and who deepened the gap between government and civil society, are still in office. Monetary and human resource shortages also create a policy context that is infertile for the implementation of a comprehensive HIV/AIDS strategy, as envisaged in the NSP. Finally, these health system restrictions have a clear negative impact on the process of policy implementation. Without the mobilization of people living with HIV/AIDS and their communities, the NSP will be ineffective in bridging the gap between policy intentions and policy implementation. The strength of this article lies in its systematic analysis of previous policy responses, as a basis for appraising current AIDS policy. Although such an approach tends to simplify the complexities of the actual policy environment, it nonetheless draws to the attention of policy participants the importance of and the complex interrelationships among the different dimensions of AIDS policy.

How can we support the use of systematic reviews in policymaking?

John Lavis discusses how health policymakers and their stakeholders need research evidence, and the best ways evidence can be synthesized and packaged to optimize its use.

Predicting medical students' intentions to take up rural practice after graduation

OBJECTIVES

Using a novel longitudinal tracking project, this study develops and evaluates the performance of a predictive model and index of rural medical practice intention based on the characteristics of incoming medical students.

METHODS

Medical school entry survey data were obtained from the Medical Schools Outcome Database (MSOD) project implemented in all Australian and New Zealand medical schools and coordinated through Medical Deans Australia and New Zealand, the representative body for the Deans of 18 Australian and two New Zealand medical schools and faculties. The medical school commencement survey collects data on students' education and family background, including rural upbringing, personal circumstances and scholarships, and on their practice intentions in terms of location and specialty. The MSOD will also allow tracking of medical graduates after graduation. Logistic regression modelling was used to develop a predictive model of rural practice intention. Split-sample validation was used to gain some insight into the stability of performance of the model.

RESULTS

Response rates to the MSOD survey exceeded 90% on average. The model findings confirm and extend previous research examining the association of medical student characteristics with intention to take up rural medical practice. The statistically significant independent factors in the model included students' rural backgrounds, financial arrangements and intentions regarding specialist versus generalist practice upon graduation. Model performance was good, with an area under the receiver-operator characteristics curve of 0.86, and reproducible, with an area in a validation sample of 0.83.

CONCLUSIONS

The model and related index provide important insights into individual factors associated with rural practice intention among students commencing medical studies. The model can also provide a means for optimising the use of scarce medical programme resources, thereby helping to improve the supply of rural medical practitioners. This study illustrates the power and potential of a robust, consistent, systematic longitudinal tracking project.

Can we systematically review studies that evaluate complex interventions?

BACKGROUND TO THE DEBATE

The UK Medical Research Council defines complex interventions as those comprising "a number of separate elements which seem essential to the proper functioning of the interventions although the 'active ingredient' of the intervention that is effective is difficult to specify." A typical example is specialist care on a stroke unit, which involves a wide range of health professionals delivering a variety of treatments. Michelle Campbell and colleagues have argued that there are "specific difficulties in defining, developing, documenting, and reproducing complex interventions that are subject to more variation than a drug". These difficulties are one of the reasons why it is challenging for researchers to systematically review complex interventions and synthesize data from separate studies. This PLoS Medicine Debate considers the challenges facing systematic reviewers and suggests several ways of addressing them.

Health systems integration: state of the evidence

INTRODUCTION

Integrated health systems are considered a solution to the challenge of maintaining the accessibility and integrity of healthcare in numerous jurisdictions worldwide. However, decision makers in a Canadian health region indicated they were challenged to find evidence-based information to assist with the planning and implementation of integrated healthcare systems.

METHODS

A systematic literature review of peer-reviewed literature from health sciences and business databases, and targeted grey literature sources.

RESULTS

Despite the large number of articles discussing integration, significant gaps in the research literature exist. There was a lack of high quality, empirical studies providing evidence on how health systems can improve service delivery and population health. No universal definition or concept of integration was found and multiple integration models from both the healthcare and business literature were proposed in the literature. The review also revealed a lack of standardized, validated tools that have been systematically used to evaluate integration outcomes. This makes measuring and comparing the impact of integration on system, provider and patient level challenging.

DISCUSSION AND CONCLUSION

Healthcare is likely too complex for a one-size-fits-all integration solution. It is important for decision makers and planners to choose a set of complementary models, structures and processes to create an integrated health system that fits the needs of the population across the continuum of care. However, in order to have evidence available, decision makers and planners should include evaluation for accountability purposes and to ensure a better understanding of the effectiveness and impact of health systems integration.

Making trials matter: pragmatic and explanatory trials and the problem of applicability

Randomised controlled trials are the best research design for decisions about the effect of different interventions but randomisation does not, of itself, promote the applicability of a trial's results to situations other than the precise one in which the trial was done. While methodologists and trialists have rightly paid great attention to internal validity, much less has been given to applicability. This narrative review is aimed at those planning to conduct trials, and those aiming to use the information in them. It is intended to help the former group make their trials more widely useful and to help the latter group make more informed decisions about the wider use of existing trials. We review the differences between the design of most randomised trials (which have an explanatory attitude) and the design of trials more able to inform decision making (which have a pragmatic attitude) and discuss approaches used to assert applicability of trial results. If we want evidence from trials to be used in clinical practice and policy, trialists should make every effort to make their trial widely applicable, which means that more trials should be pragmatic in attitude.

Human resource management interventions to improve health workers' performance in low and middle income countries: a realist review

Background

Improving health workers' performance is vital for achieving the Millennium Development Goals. In the literature on human resource management (HRM) interventions to improve health workers' performance in Low and Middle Income Countries (LMIC), hardly any attention has been paid to the question how HRM interventions might bring about outcomes and in which contexts. Such information is, however, critical to assess the transferability of results. Our aim was to explore if realist review of published primary research provides better insight into the functioning of HRM interventions in LMIC.

Methodology

A realist review not only asks whether an intervention has shown to be effective, but also through which mechanisms an intervention produces outcomes and which contextual factors appear to be of critical influence. Forty-eight published studies were reviewed.

Results

The results show that HRM interventions can improve health workers' performance, but that different contexts produce different outcomes. Critical implementation aspects were involvement of local authorities, communities and management; adaptation to the local situation; and active involvement of local staff to identify and implement solutions to problems. Mechanisms that triggered change were increased knowledge and skills, feeling obliged to change and health workers' motivation. Mechanisms to contribute to motivation were health workers' awareness of local problems and staff empowerment, gaining acceptance of new information and creating a sense of belonging and respect. In addition, staff was motivated by visible improvements in quality of care and salary supplements. Only a limited variety of HRM interventions have been evaluated in the health sector in LMIC. Assumptions underlying HRM interventions are usually not made explicit, hampering our understanding of how HRM interventions work.

Conclusion

Application of a realist perspective allows identifying which HRM interventions might improve performance, under which circumstances, and for which groups of health workers. To be better able to contribute to an understanding of how HRM interventions could improve health workers' performance, a combination of qualitative and quantitative research methods would be needed and the use of common indicators for evaluation and a common reporting format would be required.

Maternal and child health in the occupied Palestinian territory

The Countdown to 2015 intervention coverage indicators in the occupied Palestinian territory are similar to those of other Arab countries, although there are gaps in continuity and quality of services across the continuum of the perinatal period. Since the mid 1990s, however, access to maternity facilities has become increasingly unpredictable. Mortality rates for infants (age </=1 year) and children younger than 5 years have changed little, and the prevalence of stunting in children has increased. Living conditions have worsened since 2006, when the elected Palestinian administration became politically and economically boycotted, resulting in unprecedented levels of Palestinian unemployment, poverty, and internal conflict, and increased restrictions to health-care access. Although a political solution is imperative for poverty alleviation, sustainable development, and the universal right to health care, women and children should not have to wait. Urgent action from international and local decision makers is needed for sustainable access to high-quality care and basic health entitlements.

Building blocks for meta-synthesis: data integration tables for summarising, mapping, and synthesising evidence on interventions for communicating with health consumers

Background

Systematic reviews have developed into a powerful method for summarising and synthesising evidence. The rise in systematic reviews creates a methodological opportunity and associated challenges and this is seen in the development of overviews, or reviews of systematic reviews. One of these challenges is how to summarise evidence from systematic reviews of complex interventions for inclusion in an overview. Interventions for communicating with and involving consumers in their care are frequently complex. In this article we outline a method for preparing data integration tables to enable review-level synthesis of the evidence on interventions for communication and participation in health.

Methods and Results

Systematic reviews published by the Cochrane Consumers and Communication Review Group were utilised as the basis from which to develop linked steps for data extraction, evidence assessment and synthesis. The resulting output is called a data integration table. Four steps were undertaken in designing the data integration tables: first, relevant information for a comprehensive picture of the characteristics of the review was identified from each review, extracted and summarised. Second, results for the outcomes of the review were assessed and translated to standardised evidence statements. Third, outcomes and evidence statements were mapped into an outcome taxonomy that we developed, using language specific to the field of interventions for communication and participation. Fourth, the implications of the review were assessed after the mapping step clarified the level of evidence available for each intervention.

Conclusion

The data integration tables represent building blocks for constructing overviews of review-level evidence and for the conduct of meta-synthesis. Individually, each table aims to improve the consistency of reporting on the features and effects of interventions for communication and participation; provides a broad assessment of the strength of evidence derived from different methods of analysis; indicates a degree of certainty with results; and reports outcomes and gaps in the evidence in a consistent and coherent way. In addition, individual tables can serve as a valuable tool for accurate dissemination of large amounts of complex information on communication and participation to professionals as well as to members of the public.

An international survey indicated that unpublished systematic reviews exist

OBJECTIVE

To determine the frequency of unpublished systematic reviews (SRs) and explore factors contributing to their occurrence.

STUDY DESIGN AND SETTING

First or corresponding authors from a sample of SRs published in 2005 were asked to participate in a 26-item survey administered through the Internet, facsimile, and postal mail. Outcomes included median and range of published and unpublished SRs, and barriers, facilitators, and reasons for not publishing SRs. Descriptive analyses were performed.

RESULTS

55.7% (348 of 625) of those invited participated, half of which were from Europe and 22.7% were from the United States. Participants reported 1,405 published (median: 2.0, range: 1-150) and 199 unpublished (median: 2.0, range: 1-33) SRs. Lack of time and lack of funding and organizational support were barriers, whereas time availability and self-motivation were facilitators to publishing reviews. For most recent unpublished SRs (n=52), the reasons for not publishing included lack of time (12 of 52, 23.0%), the manuscript being rejected (10 of 52, 19.0%), and operational issues (six of 52, 11.5%).

CONCLUSION

Unpublished SRs do exist. Lack of time, funding, and organizational support were consistent reasons for not publishing SRs. Statistical significance of SR results was not reported as being a major barrier or reason for not publishing. Further research on unpublished SRs is warranted.

Developing a national dissemination plan for collaborative care for depression: QUERI Series

Background

Little is known about effective strategies for disseminating and implementing complex clinical innovations across large healthcare systems. This paper describes processes undertaken and tools developed by the U.S. Department of Veterans Affairs (VA) Mental Health Quality Enhancement Research Initiative (MH-QUERI) to guide its efforts to partner with clinical leaders to prepare for national dissemination and implementation of collaborative care for depression.

Methods

An evidence-based quality improvement (EBQI) process was used to develop an initial set of goals to prepare the VA for national dissemination and implementation of collaborative care. The resulting product of the EBQI process is referred to herein as a "National Dissemination Plan" (NDP). EBQI participants included: a) researchers with expertise on the collaborative care model for depression, clinical quality improvement, and implementation science, and b) VA clinical and administrative leaders with experience and expertise on how to adapt research evidence to organizational needs, resources and capacity. Based on EBQI participant feedback, drafts of the NDP were revised and refined over multiple iterations before a final version was approved by MH-QUERI leadership. 'Action Teams' were created to address each goal. A formative evaluation framework and related tools were developed to document processes, monitor progress, and identify and act upon barriers and facilitators in addressing NDP goals.

Results

The National Dissemination Plan suggests that effectively disseminating collaborative care for depression in the VA will likely require attention to: Guidelines and Quality Indicators (4 goals), Training in Clinical Processes and Evidence-based Quality Improvement (6 goals), Marketing (7 goals), and Informatics Support (1 goal). Action Teams are using the NDP as a blueprint for developing infrastructure to support system-wide adoption and sustained implementation of collaborative care for depression. To date, accomplishments include but are not limited to: conduct of a systematic review of the literature to update VA depression treatment guidelines to include the latest evidence on collaborative care for depression; training for clinical staff on TIDES (Translating Initiatives for Depression into Effective Solutions project) care; spread of TIDES care to new VA facilities; and integration of TIDES depression assessment tools into a planned update of software used in delivery of VA mental health services. Thus far, common barriers encountered by Action Teams in addressing NDP goals include: a) limited time to address goals due to competing tasks/priorities, b) frequent turnover of key organizational leaders/stakeholders, c) limited skills and training among team members for addressing NDP goals, and d) difficulty coordinating activities across Action Teams on related goals.

Conclusion

MH-QUERI has partnered with VA organizational leaders to develop a focused yet flexible plan to address key factors to prepare for national dissemination and implementation of collaborative care for depression. Early indications suggest that the plan is laying an important foundation that will enhance the likelihood of successful implementation and spread across the VA healthcare system.

Evidence-informed health policy 1 - synthesis of findings from a multi-method study of organizations that support the use of research evidence

Background

Organizations have been established in many countries and internationally to support the use of research evidence by producing clinical practice guidelines, undertaking health technology assessments, and/or directly supporting the use of research evidence in developing health policy on an international, national, and state or provincial level. Learning from these organizations can reduce the need to 'reinvent the wheel' and inform decisions about how best to organize support for such organizations, particularly in low- and middle-income countries (LMICs).

Methods

We undertook a multi-method study in three phases – a survey, interviews, and case descriptions that drew on site visits – and in each of the second and third phases we focused on a purposive sample of those involved in the previous phase. We used the seven main recommendations that emerged from the advice offered in the interviews to organize much of the synthesis of findings across phases and methods. We used a constant comparative method to identify themes from across phases and methods.

Results

Seven recommendations emerged for those involved in establishing or leading organizations that support the use of research evidence in developing health policy: 1) collaborate with other organizations; 2) establish strong links with policymakers and involve stakeholders in the work; 3) be independent and manage conflicts of interest among those involved in the work; 4) build capacity among those working in the organization; 5) use good methods and be transparent in the work; 6) start small, have a clear audience and scope, and address important questions; and 7) be attentive to implementation considerations, even if implementation is not a remit. Four recommendations emerged for the World Health Organization (WHO) and other international organizations and networks: 1) support collaborations among organizations; 2) support local adaptation efforts; 3) mobilize support; and 4) create global public goods.

Conclusion

This synthesis of findings from a multi-method study, along with the more detailed findings from each of the three phases of the study (which are reported in the three following articles in the series), provide a strong basis on which researchers, policymakers, international organizations (and networks) like WHO can respond to the growing chorus of voices calling for efforts to support the use of research evidence in developing health policy.

Translating research into maternal health care policy: a qualitative case study of the use of evidence in policies for the treatment of eclampsia and pre-eclampsia in South Africa

Background

Few empirical studies of research utilisation have been conducted in low and middle income countries. This paper explores how research information, in particular findings from randomised controlled trials and systematic reviews, informed policy making and clinical guideline development for the use of magnesium sulphate in the treatment of eclampsia and pre-eclampsia in South Africa.

Methods

A qualitative case-study approach was used to examine the policy process. This included a literature review, a policy document review, a timeline of key events and the collection and analysis of 15 interviews with policy makers and academic clinicians involved in these policy processes and sampled using a purposive approach. The data was analysed thematically and explored theoretically through the literature on agenda setting and the policy making process.

Results

Prior to 1994 there was no national maternal care policy in South Africa. Consequently each tertiary level institution developed its own care guidelines and these recommended a range of approaches to the management of pre-eclampsia and eclampsia. The subsequent emergence of new national policies for maternal care, including for the treatment of pre-eclampsia and eclampsia, was informed by evidence from randomised controlled trials and systematic reviews. This outcome was influenced by a number of factors. The change to a democratic government in the mid 1990s, and the health reforms that followed, created opportunities for maternal health care policy development. The new government was open to academic involvement in policy making and recruited academics from local networks into key policy making positions in the National Department of Health. The local academic obstetric network, which placed high value on evidence-based practice, brought these values into the policy process and was also linked strongly to international evidence based medicine networks. Within this context of openness to policy development, local researchers acted as policy entrepreneurs, bringing attention to priority health issues, and to the use of research evidence in addressing these. This resulted in the new national maternity care guidelines being informed by evidence from randomised controlled trials and recommending explicitly the use of magnesium sulphate for the management of eclampsia.

Conclusion

Networks of researchers were important not only in using research information to shape policy but also in placing issues on the policy agenda. A policy context which created a window of opportunity for new research-informed policy development was also crucial.

Patient evaluations of accessibility and co-ordination in general practice in Europe

OBJECTIVE

To determine whether patient evaluations of the accessibility to general practice and co-ordination with other care providers were associated with characteristics of general practice organizations.

BACKGROUND

In 1998 patients across Europe perceived that small general practices have better accessibility than large practices. Since then a number of changes in primary care have had impact on accessibility and co-ordination of care.

DESIGN, SETTING AND PARTICIPANTS

The study was based on data from the European Practice Assessment study, an observational study in 284 general practices in 10 countries in 2004.

MAIN OUTCOME MEASURES

Patient evaluations of general practice were measured with the 23-item Europep instrument, from which seven items on accessibility and co-ordination were selected in a principal factor analysis. Six practice characteristics were examined: percentage of female general practitioners, mean age of physicians, mean number of physician hours worked per week, number of general practitioners, number of care providers, urbanization level. Mixed regression models were applied, in which patients were clustered within practices, and practices within countries.

RESULTS

Practices with a higher numbers of care providers received less positive patient evaluations (b= -0.112, P=0.004). The other practice characteristics were not related to patient evaluations. Only a small proportion of the total variation in patient evaluations of accessibility and co-ordination (1.8%) was explained by characteristics of the general practice organizations.

CONCLUSIONS

General practices have become larger in most developed countries in recent years, but patients seemed to prefer general practice organizations with fewer health professionals.

Sustainability science: an integrated approach for health-programme planning

Planning for programme sustainability is a key contributor to health and development, especially in low-income and middle-income countries. A consensus evidence-based operational framework would facilitate policy and research advances in understanding, measuring, and improving programme sustainability. We did a systematic review of both conceptual frameworks and empirical studies about health-programme sustainability. On the basis of the review, we propose that sustainable health programmes are regarded as complex systems that encompass programmes, health problems targeted by programmes, and programmes' drivers or key stakeholders, all of which interact dynamically within any given context. We show the usefulness of this approach with case studies drawn from the authors' experience.

Supporting the delivery of cost-effective interventions in primary health-care systems in low-income and middle-income countries: an overview of systematic reviews

Strengthening health systems is a key challenge to improving the delivery of cost-effective interventions in primary health care and achieving the vision of the Alma-Ata Declaration. Effective governance, financial and delivery arrangements within health systems, and effective implementation strategies are needed urgently in low-income and middle-income countries. This overview summarises the evidence from systematic reviews of health systems arrangements and implementation strategies, with a particular focus on evidence relevant to primary health care in such settings. Although evidence is sparse, there are several promising health systems arrangements and implementation strategies for strengthening primary health care. However, their introduction must be accompanied by rigorous evaluations. The evidence base needs urgently to be strengthened, synthesised, and taken into account in policy and practice, particularly for the benefit of those who have been excluded from the health care advances of recent decades.

Primary health care: making Alma-Ata a reality

The principles agreed at Alma-Ata 30 years ago apply just as much now as they did then. "Health for all" by the year 2000 was not achieved, and the Millennium Development Goals (MDGs) for 2015 will not be met in most low-income countries without substantial acceleration of primary health care. Factors have included insufficient political prioritisation of health, structural adjustment policies, poor governance, population growth, inadequate health systems, and scarce research and assessment on primary health care. We propose the following priorities for revitalising primary health care. Health-service infrastructure, including human resources and essential drugs, needs strengthening, and user fees should be removed for primary health-care services to improve use. A continuum of care for maternal, newborn, and child health services, including family planning, is needed. Evidence-based, integrated packages of community and primary curative and preventive care should be adapted to country contexts, assessed, and scaled up. Community participation and community health workers linked to strengthened primary-care facilities and first-referral services are needed. Furthermore, intersectoral action linking health and development is necessary, including that for better water, sanitation, nutrition, food security, and HIV control. Chronic diseases, mental health, and child development should be addressed. Progress should be measured and accountability assured. We prioritise research questions and suggest actions and measures for stakeholders both locally and globally, which are required to revitalise primary health care.

"Developing good taste in evidence": facilitators of and hindrances to evidence-informed health policymaking in state government

CONTEXT

Policymaking is a highly complex process that is often difficult to predict or influence. Most of the scholarship examining the role of research evidence in policymaking has focused narrowly on characteristics of the evidence and the interactions between scientists and government officials. The real-life context in which policymakers are situated and make decisions also is crucial to the development of evidence-informed policy.

METHODS

This qualitative study expands on other studies of research utilization at the state level through interviews with twenty-eight state legislators and administrators about their real-life experiences incorporating evidence into policymaking. The interviews were coded inductively into the following categories: (1) the important or controversial issue or problem being addressed, (2) the information that was used, (3) facilitators, and (4) hindrances.

FINDINGS

Hindrances to evidence-informed policymaking included institutional features; characteristics of the evidence supply, such as research quantity, quality, accessibility, and usability; and competing sources of influence, such as interest groups. The policymakers identified a number of facilitators to the use of evidence, including linking research to concrete impacts, costs, and benefits; reframing policy issues to fit the research; training to use evidence-based skills; and developing research venues and collaborative relationships in order to generate relevant evidence.

CONCLUSIONS

Certain hindrances to the incorporation of research into policy, like limited budgets, are systemic and not readily altered. However, some of the barriers and facilitators of evidence-informed health policymaking are amenable to change. Policymakers could benefit from evidence-based skills training to help them identify and evaluate high-quality information. Researchers and policymakers thus could collaborate to develop networks for generating and sharing relevant evidence for policy.

Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services

Scoping studies have been used across a range of disciplines for a wide variety of purposes. However, their value is increasingly limited by a lack of definition and clarity of purpose. The UK's Service Delivery and Organisation Research Programme (SDO) has extensive experience of commissioning and using such studies; twenty four have now been completed.

This review article has four objectives; to describe the nature of the scoping studies that have been commissioned by the SDO Programme; to consider the impact of and uses made of such studies; to provide definitions for the different elements that may constitute a scoping study; and to describe the lessons learnt by the SDO Programme in commissioning scoping studies.

Scoping studies are imprecisely defined but usually consist of one or more discrete components; most commonly they are non-systematic reviews of the literature, but other important elements are literature mapping, conceptual mapping and policy mapping. Some scoping studies also involve consultations with stakeholders including the end users of research.

Scoping studies have been used for a wide variety of purposes, although a common feature is to identify questions and topics for future research. The reports of scoping studies often have an impact that extends beyond informing research commissioners about future research areas; some have been published in peer reviewed journals, and others have been published in research summaries aimed at a broader audience of health service managers and policymakers.

Key lessons from the SDO experience are the need to relate scoping studies to a particular health service context; the need for scoping teams to be multi-disciplinary and to be given enough time to integrate diverse findings; and the need for the research commissioners to be explicit not only about the aims of scoping studies but also about their intended uses. This necessitates regular contact between researchers and commissioners.

Scoping studies are an essential element in the portfolio of approaches to research, particularly as a mechanism for helping research commissioners and policy makers to ask the right questions. Their utility will be further enhanced by greater recognition of the individual components, definitions for which are provided.