Do evidence summaries increase health policy-makers' use of evidence from systematic reviews? A systematic review

What happened next? A survey of review clients evaluating impacts of rapid reviews

OBJECTIVES

End-user evaluation of the impact of evidence syntheses is critical to demonstrating value. This study presents results of a survey evaluating the impact of rapid reviews undertaken by two teams based in Melbourne, Australia, and Hamilton, Canada.

METHODS

Clients were invited to participate in a short written survey following delivery of a rapid review. Survey items encompassed reach, usefulness and format; interactions with the review teams; and overall satisfaction.

RESULTS

Twenty-five completed surveys from 53 invitations were received pertaining to 19 rapid reviews conducted between September 2021 and October 2023. Topics encompassed COVID-19, health and behavior change; reports were an average of 31 pages; and were delivered over an average of 62 days. Evaluation findings were positive, with high satisfaction with reports and service delivery; very high satisfaction with report structure and length; good evidence of reach (reports read by decision makers and cited in other documents); and evidence that the rapid reviews made contributions to strategic planning, policy and program funding decisions.

CONCLUSION

Rapid reviews are making impactful contributions, alongside other inputs, to policy and practice. Further research is required to build this evaluation dataset; examine the balance between timeliness and methodological rigor in evidence synthesis; and explore models of delivery and capacity within and outside of government. It is also critical to promote implementation efforts to harness the full potential of rapid reviews and other evidence syntheses to impact the lives of citizens.

A plan for systematic reviews for high-need areas in forensic science

Forensic scientific practitioners and researchers must navigate a rapidly growing body of research. This makes it increasingly challenging to inform courts, lawyers, and other decision makers about the state of the field, thus heightening the chances of wrongful convictions and acquittals. When similar challenges have arisen in other fields, they have turned to systematic reviews, which are research reviews that use formal, articulated methods to provide a comprehensive summary of the literature on a specific research question. Systematic reviews allow users to evaluate how the authors identified research and synthesised its findings, making them more transparent than informal literature reviews. This article lays out a justification and plan for systematic reviews in forensic science.

Signposting services for people with health and care needs: a rapid realist review

Background

Signposting typically refers to an informal process that involves giving information to patients to enable them to access external services and support. It is perceived to reduce demand on primary care and other urgent care services.

Methods

This focused realist review was conducted rapidly within time constraints. Searches to identify theory were undertaken on MEDLINE, Cumulative Index to Nursing and Allied Health Literature and Social Sciences Citation Index in June 2022 for research published in English from 2016. We selected 22 publications and extracted programme theories from these to develop three priority questions: Question 1: What do people with health and social care needs require from a signposting service to believe it is valuable? Question 2: What resources do providers require to confidently deliver an effective signposting service? Question 3: Under what circumstances should commissioners commission generic or specialist signposting services? Purposive searching was conducted to find a rich sample of studies. UK studies were prioritised to optimise the applicability of synthesis findings.

Results

The review included 27 items, 4 reviews and 23 studies, a mix of qualitative, evaluations and case studies. Service users value a joined-up response that helps them to navigate the available resources. Key features include an understanding of their needs, suggestion of different options and a summary of recommended actions. Only a small number of service user needs are met by signposting services alone; people with complex health and social care needs often require extended input and time. Front-line providers of signposting services require appropriate training, ongoing support and supervision, good knowledge of relevant and available activities and an ability to match service users to appropriate resources. Front-line providers need to offer a flexible response targeted at user needs. Commissioned signposting services in England (no studies from Scotland, Wales and Northern Ireland) are highly diverse in terms of client groups, staff delivering the service, referral routes and role descriptions. A lack of service evaluation poses a potential barrier to effective commissioning. A shortage of available services in the voluntary and community sector may limit the effectiveness of signposting services. Commissioners should ensure that referrals target intensive support at patients most likely to benefit in the longer term.

Conclusions

Signposting services need greater clarity of roles and service expectations to facilitate evaluation. Users with complex health and social care needs require intensive, repeat support from specialist services equipped with specific knowledge and situational understanding. A tension persists between efficient (transactional) service provision with brief referral and effective (relational) service provision, underpinned by competing narratives. Do signposting services represent 'diversion of unwanted demand from primary care/urgent care services' or 'improved quality of care through a joined-up response by health, social care and community/voluntary services'?

Limitations

This realist review was conducted within a tight time frame with a potential impact on methodology; for example, the use of purposive searching may have resulted in omission of relevant evidence.

Future work

Signposting services require service evaluation and consideration of the issue of diversity.

Study registration

This study is registered as PROSPERO CRD42022348200.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: NIHR130588) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 26. See the NIHR Funding and Awards website for further award information.

Various application roles for Campbell systematic reviews: a citation analysis

OBJECTIVES

Systematic reviews (SRs) are becoming essential evidence in the decision-making process within the field of social sciences. This study aimed to investigate how Campbell SRs were cited and explore their specific application roles.

STUDY DESIGN AND SETTING

We included Campbell SRs published between 2016 and 2020 by searching the Wiley online library, and retrieved the articles and documents citing Campbell SRs from the Web of Science and Google Scholar by December 31, 2021. We described the characteristics of the SRs and citations, and formulated a set of application roles by analyzing the sentences or paragraphs where the SRs were cited.

RESULTS

Sixty nine Campbell SRs were published between 2016 and 2020; they were cited in 641 articles or documents a total of 1,289 times. The primary types of articles that cited Campbell SRs were cross-sectional studies (n = 226, 35.3%), SRs (n = 112, 17.5%), randomized controlled trials (n = 77, 12.0%), and policy reports (n = 57, 8.8%). Articles utilizing Campbell SRs were predominantly led by authors from the United States (n = 184, 28.7%), the United Kingdom (n = 98, 15.3%), and Australia (n = 51, 8.0%). We formulated a set of 10 application roles for the Campbell SRs, of which the most frequent were: describing the current status in the field of interest (n = 691, 53.6%), corroboration of the results (n = 140, 10.9%), identifying research gaps (n = 130, 10.1%), and providing methodological references (n = 126, 9.8%); the role of supporting policy recommendations or decisions accounted for 6.0% (n = 77) of the citations. Approximately 12% of Campbell SRs were used to support policy recommendations or decisions.

CONCLUSION

Campbell SRs are widely applied, particularly in scientific research, to describe the current status in the field of interest. Although the current application of Campbell SRs in supporting policy recommendations and decisions may not be predominant, there is a growing recognition of their value in using Campbell SRs to inform decision-making.

Decision-makers' experiences with rapid evidence summaries to support real-time evidence informed decision-making in crises: a mixed methods study

BACKGROUND

There is a clear need for research evidence to drive policymaking and emergency responses so that lives are saved and resources are not wasted. The need for evidence support for health and humanitarian crisis is even more pertinent because of the time and practical constraints that decision-makers in these settings face. To improve the use of research evidence in policy and practice, it is important to provide evidence resources tailored to the target audience. This study aims to gain real-world insights from decision-makers about how they use evidence summaries to inform real-time decision-making in crisis-settings, and to use our findings to improve the format of evidence summaries.

METHODS

This study used an explanatory sequential mixed method study design. First, we used a survey to identify the views and experiences of those who were directly involved in crisis response in different contexts, and who may or may not have used evidence summaries. Second, we used the insights generated from the survey to help inform qualitative interviews with decision-makers in crisis-settings to derive an in-depth understanding of how they use evidence summaries and their desired format for evidence summaries.

RESULTS

We interviewed 26 decision-makers working in health and humanitarian emergencies. The study identified challenges decision-makers face when trying to find and use research evidence in crises, including insufficient time and increased burden of responsibilities during crises, limited access to reliable internet connection, large volume of data not translated into user friendly summaries, and little information available on preparedness and response measures. Decision-makers preferred the following components in evidence summaries: title, target audience, presentation of key findings in an actionable checklist or infographic format, implementation considerations, assessment of the quality of evidence presented, citation and hyperlink to the full review, funding sources, language of full review, and other sources of information on the topic. Our study developed an evidence summary template with accompanying training material to inform real-time decision-making in crisis-settings.

CONCLUSIONS

Our study provided a deeper understanding of the preferences of decision-makers working in health and humanitarian emergencies about the format of evidence summaries to enable real-time evidence informed decision-making.

The Impact of Information Presentation and Cognitive Dissonance on Processing Systematic Review Summaries: A Randomized Controlled Trial on Bicycle Helmet Legislation

Background: Summaries of systematic reviews are a reference method for the dissemination of research evidence on the effectiveness of public health interventions beyond the scientific community. Motivated reasoning and cognitive dissonance may interfere with readers' ability to process the information included in such summaries. Methods: We conducted a web experiment on a panel of university-educated North Americans (N = 259) using a systematic review of the effectiveness of bicycle helmet legislation as a test case. The outcome variables were the perceived tentativeness of review findings and attitude toward bicycle helmet legislation. We manipulated two types of uncertainty: (i) deficient uncertainty (inclusion vs. non-inclusion of information on limitations of the studies included in the review) and (ii) consensus uncertainty (consensual findings showing legislation effectiveness vs. no evidence of effectiveness). We also examined whether reported expertise in helmet legislation and the frequency of wearing a helmet while cycling interact with the experimental factors. Results: None of the experimental manipulations had a main effect on the perceived tentativeness. The presentation of consensual efficacy findings had a positive main effect on the attitude toward the legislation. Self-reported expertise had a significant main effect on the perceived tentativeness, and exposing participants with reported expertise to results showing a lack of evidence of efficacy increased their favorable attitude toward the legislation. Participants' helmet use was positively associated with their attitude toward the legislation (but not with perceived tentativeness). Helmet use did not interact with the experimental manipulations. Conclusions: Motivated reasoning and cognitive dissonance influence a reader's ability to process information contained in a systematic review summary.

Within study comparisons and risk of bias in international development: Systematic review and critical appraisal

Background

Many systematic reviews incorporate nonrandomised studies of effects, sometimes called quasi-experiments or natural experiments. However, the extent to which nonrandomised studies produce unbiased effect estimates is unclear in expectation or in practice. The usual way that systematic reviews quantify bias is through "risk of bias assessment" and indirect comparison of findings across studies using meta-analysis. A more direct, practical way to quantify the bias in nonrandomised studies is through "internal replication research", which compares the findings from nonrandomised studies with estimates from a benchmark randomised controlled trial conducted in the same population. Despite the existence of many risks of bias tools, none are conceptualised to assess comprehensively nonrandomised approaches with selection on unobservables, such as regression discontinuity designs (RDDs). The few that are conceptualised with these studies in mind do not draw on the extensive literature on internal replications (within-study comparisons) of randomised trials.

Objectives

Our research objectives were as follows:Objective 1: to undertake a systematic review of nonrandomised internal study replications of international development interventions.Objective 2: to develop a risk of bias tool for RDDs, an increasingly common method used in social and economic programme evaluation.

Methods

We used the following methods to achieve our objectives.Objective 1: we searched systematically for nonrandomised internal study replications of benchmark randomised experiments of social and economic interventions in low- and middle-income countries (L&MICs). We assessed the risk of bias in benchmark randomised experiments and synthesised evidence on the relative bias effect sizes produced by benchmark and nonrandomised comparison arms.Objective 2: We used document review and expert consultation to develop further a risk of bias tool for quasi-experimental studies of interventions (ROBINS-I) for RDDs.

Results

Objective 1: we located 10 nonrandomised internal study replications of randomised trials in L&MICs, six of which are of RDDs and the remaining use a combination of statistical matching and regression techniques. We found that benchmark experiments used in internal replications in international development are in the main well-conducted but have "some concerns" about threats to validity, usually arising due to the methods of outcomes data collection. Most internal replication studies report on a range of different specifications for both the benchmark estimate and the nonrandomised replication estimate. We extracted and standardised 604 bias coefficient effect sizes from these studies, and present average results narratively.Objective 2: RDDs are characterised by prospective assignment of participants based on a threshold variable. Our review of the literature indicated there are two main types of RDD. The most common type of RDD is designed retrospectively in which the researcher identifies post-hoc the relationship between outcomes and a threshold variable which determines assignment to intervention at pretest. These designs usually draw on routine data collection such as administrative records or household surveys. The other, less common, type is a prospective design where the researcher is also involved in allocating participants to treatment groups from the outset. We developed a risk of bias tool for RDDs.

Conclusions

Internal study replications provide the grounds on which bias assessment tools can be evidenced. We conclude that existing risk of bias tools needs to be further developed for use by Campbell collaboration authors, and there is a wide range of risk of bias tools and internal study replications to draw on in better designing these tools. We have suggested the development of a promising approach for RDD. Further work is needed on common methodologies in programme evaluation, for example on statistical matching approaches. We also highlight that broader efforts to identify all existing internal replication studies should consider more specialised systematic search strategies within particular literatures; so as to overcome a lack of systematic indexing of this evidence.