The impact of Cochrane Reviews: a mixed-methods evaluation of outputs from Cochrane Review Groups supported by the National Institute for Health Research

Various application roles for Campbell systematic reviews: a citation analysis

OBJECTIVES

Systematic reviews (SRs) are becoming essential evidence in the decision-making process within the field of social sciences. This study aimed to investigate how Campbell SRs were cited and explore their specific application roles.

STUDY DESIGN AND SETTING

We included Campbell SRs published between 2016 and 2020 by searching the Wiley online library, and retrieved the articles and documents citing Campbell SRs from the Web of Science and Google Scholar by December 31, 2021. We described the characteristics of the SRs and citations, and formulated a set of application roles by analyzing the sentences or paragraphs where the SRs were cited.

RESULTS

Sixty nine Campbell SRs were published between 2016 and 2020; they were cited in 641 articles or documents a total of 1,289 times. The primary types of articles that cited Campbell SRs were cross-sectional studies (n = 226, 35.3%), SRs (n = 112, 17.5%), randomized controlled trials (n = 77, 12.0%), and policy reports (n = 57, 8.8%). Articles utilizing Campbell SRs were predominantly led by authors from the United States (n = 184, 28.7%), the United Kingdom (n = 98, 15.3%), and Australia (n = 51, 8.0%). We formulated a set of 10 application roles for the Campbell SRs, of which the most frequent were: describing the current status in the field of interest (n = 691, 53.6%), corroboration of the results (n = 140, 10.9%), identifying research gaps (n = 130, 10.1%), and providing methodological references (n = 126, 9.8%); the role of supporting policy recommendations or decisions accounted for 6.0% (n = 77) of the citations. Approximately 12% of Campbell SRs were used to support policy recommendations or decisions.

CONCLUSION

Campbell SRs are widely applied, particularly in scientific research, to describe the current status in the field of interest. Although the current application of Campbell SRs in supporting policy recommendations and decisions may not be predominant, there is a growing recognition of their value in using Campbell SRs to inform decision-making.

A meta-review of the impact of compression therapy on venous leg ulcer healing

This meta-review aimed to appraise and synthesise findings from existing systematic reviews that measured the impact of compression therapy on venous leg ulcers healing. We searched five databases to identify potential papers; three authors extracted data, and a fourth author adjudicated the findings. The AMSTAR-2 tool was used for quality appraisal and the certainty of the evidence was appraised using GRADEpro. Data analysis was undertaken using RevMan. We identified 12 systematic reviews published between 1997 and 2021. AMSTAR-2 assessment identified three as high quality, five as moderate quality, and four as low quality. Seven comparisons were reported, with a meta-analysis undertaken for five of these comparisons: compression vs no compression (risk ratio [RR]: 1.55; 95% confidence interval [CI] 1.34-1.78; P < .00001; moderate-certainty evidence); elastic compression vs inelastic compression (RR: 1.02; 95% CI: 0.96-1.08; P < .61 moderate-certainty evidence); four layer vs <four-layer bandage systems (RR: 1.07; 95% CI: 0.82-1.40; P < .63; moderate-certainty evidence); comparison between different four-layer bandage systems (RR: 1.08; 95% CI: 0.93-1.25; P = .34; moderate-certainty evidence); compression bandage vs compression stocking (RR 0.95; 95% CI 0.87-1.03; P = .18; moderate-certainty evidence). The main conclusion from this review is that there is a statistically significant difference in healing rates when compression is used compared with no compression, with moderate-certainty evidence. Otherwise, there is no statistically different difference in healing rates using elastic compression vs inelastic compression, four layer vs <four-layer bandage systems, different four-layer bandage systems, or compression bandages vs compression stockings.

Investigator initiated trials versus industry sponsored trials - translation of randomized controlled trials into clinical practice (IMPACT)

Background

Healthcare decisions are ideally based on clinical trial results, published in study registries, as journal articles or summarized in secondary research articles. In this research project, we investigated the impact of academically and commercially sponsored clinical trials on medical practice by measuring the proportion of trials published and cited by systematic reviews and clinical guidelines.

Methods

We examined 691 multicenter, randomized controlled trials that started in 2005 or later and were completed by the end of 2016. To determine whether sponsorship/funding and place of conduct influence a trial’s impact, we created four sub-cohorts of investigator initiated trials (IITs) and industry sponsored trials (ISTs): 120 IITs and 171 ISTs with German contribution compared to 200 IITs and 200 ISTs without German contribution. We balanced the groups for study phase and place of conduct. German IITs were funded by the German Research Foundation (DFG), the Federal Ministry of Education and Research (BMBF), or by another non-commercial research organization. All other trials were drawn from the German Clinical Trials Register or ClinicalTrials.gov. We investigated, to what extent study characteristics were associated with publication and impact using multivariable logistic regressions.

Results

For 80% of the 691 trials, results were published as result articles in a medical journal and/or study registry, 52% were cited by a systematic review, and 26% reached impact in a clinical guideline. Drug trials and larger trials were associated with a higher probability to be published and to have an impact than non-drug trials and smaller trials. Results of IITs were more often published as a journal article while results of ISTs were more often published in study registries. International ISTs less often gained impact by inclusion in systematic reviews or guidelines than IITs.

Conclusion

An encouraging high proportion of the clinical trials were published, and a considerable proportion gained impact on clinical practice. However, there is still room for improvement. For publishing study results, study registries have become an alternative or complement to journal articles, especially for ISTs. IITs funded by governmental bodies in Germany reached an impact that is comparable to international IITs and ISTs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01359-x.

Over half of clinical practice guidelines use non-systematic methods to inform recommendations: A methods study

INTRODUCTION

Assessing the process used to synthesize the evidence in clinical practice guidelines enables users to determine the trustworthiness of the recommendations. Clinicians are increasingly dependent on guidelines to keep up with vast quantities of medical literature, and guidelines are followed to avoid malpractice suits. We aimed to assess whether systematic methods were used when synthesizing the evidence for guidelines; and to determine the type of review cited in support of recommendations.

METHODS

Guidelines published in 2017 and 2018 were retrieved from the TRIP and Epistemonikos databases. We randomly sorted and sequentially screened clinical guidelines on all topics to select the first 50 that met our inclusion criteria. Our primary outcomes were the number of guidelines using either a systematic or non-systematic process to gather, assess, and synthesise evidence; and the numbers of recommendations within guidelines based on different types of evidence synthesis (systematic or non-systematic reviews). If a review was cited, we looked for evidence that it was critically appraised, and recorded which quality assessment tool was used. Finally, we examined the relation between the use of the GRADE approach, systematic review process, and type of funder.

RESULTS

Of the 50 guidelines, 17 (34%) systematically synthesised the evidence to inform recommendations. These 17 guidelines clearly reported their objectives and eligibility criteria, conducted comprehensive search strategies, and assessed the quality of the studies. Of the 29/50 guidelines that included reviews, 6 (21%) assessed the risk of bias of the review. The quality of primary studies was reported in 30/50 (60%) guidelines.

CONCLUSIONS

High quality, systematic review products provide the best available evidence to inform guideline recommendations. Using non-systematic methods compromises the validity and reliability of the evidence used to inform guideline recommendations, leading to potentially misleading and untrustworthy results.

Quaternary prevention: a concept relevant to public health? A bibliometric and descriptive content analysis

According to the World Organization of Family Doctors (WONCA), quaternary prevention (P4) is a recent concept that aims to prevent medical overuse. Thus, this study aimed to measure and map research output on P4 as outline research trends, evaluating its current international status through a bibliometric and descriptive content analysis. We reviewed scientific articles on P4 recorded in PubMed, LILACS, SciELO or CINAHL with the outcomes: publication year, first authors' name and nationality, journals' name, country and ranking, publication language, used methods and main reported subjects. The analysis included 65 articles published in 33 journals of 16 countries between 2003 and 2018 with a peak of publications in 2015. The first authors came from 17 different countries, 23% Brazilian, with Uruguay as the leading nation in scientific production per capita. Q1 or Q2 journals amassed 28% of published papers. Bibliographic research comprised 88% of articles and 38% of all focused on specific examples of medical overuse. P4 represents an ethical and valid approach to prevent iatrogenic events and achieve equal and fair access to health services. Conceptual, geographical, and linguistic elements, as well as WONCA conferences and type of healthcare systems in the authors' country were fundamental factors that affected research output. The available studies are still of limited quality and quantity, with further investigations needed to assess the effective impact of P4 on public health.

"It Would've Been Nice if They Interpreted the Data a Little Bit. It Didn't Really Say Much, and It Didn't Really Help Us.": A Qualitative Study of VA Health System Evidence Needs

Supplemental Digital Content is available in the text.

Patient health outcomes improve when learning health care systems use evidence to implement promising services and allocate resources effectively. Here, we examine the unique environment in which Veterans Health Administration (VHA) leadership use evidence and the facilitators and barriers to using evidence synthesis products in decision-making. We end by describing the steps researchers can take to better support the needs of health system leadership.

Impact and use of reviews and 'overviews of reviews' to inform clinical practice guideline recommendations: protocol for a methods study

INTRODUCTION

Guidelines are systematically developed recommendations to assist practitioner and patient decisions about treatments for clinical conditions. High quality and comprehensive systematic reviews and 'overviews of systematic reviews' (overviews) represent the best available evidence. Many guideline developers, such as the WHO and the Australian National Health and Medical Research Council, recommend the use of these research syntheses to underpin guideline recommendations. We aim to evaluate the impact and use of systematic reviews with and without pairwise meta-analysis or network meta-analyses (NMAs) and overviews in clinical practice guideline (CPG) recommendations.

METHODS AND ANALYSIS

CPGs will be retrieved from Turning Research Into Practice and Epistemonikos (2017-2018). The retrieved citations will be sorted randomly and then screened sequentially by two independent reviewers until 50 CPGs have been identified. We will include CPGs that provide at least two explicit recommendations for the management of any clinical condition. We will assess whether reviews or overviews were cited in a recommendation as part of the development process for guidelines. Data extraction will be done independently by two authors and compared. We will assess the risk of bias by examining how each guideline developed clinical recommendations. We will calculate the number and frequency of citations of reviews with or without pairwise meta-analysis, reviews with NMAs and overviews, and whether they were systematically or non-systematically developed. Results will be described, tabulated and categorised based on review type (reviews or overviews). CPGs reporting the use of the Grading of Recommendations, Assessment, Development and Evaluation approach will be compared with those using a different system, and pharmacological versus non-pharmacological CPGs will be compared.

ETHICS AND DISSEMINATION

No ethics approval is required. We will present at the Cochrane Colloquium and the Guidelines International Network conference.

"Implementation Is so Difficult": Survey of National Learning Health System Decision-makers Identifies Need For Implementation Information in Evidence Reviews

BACKGROUND

Evidence use within learning health care systems can improve patient health outcomes. Embedded in the Veterans Health Administration (VHA) since 2007, the Veterans Affairs Evidence Synthesis Program (ESP) provides tailored evidence synthesis services to support VHA's learning health care system goals. As part of the ESP's ongoing quality improvement efforts, we have been surveying our users since 2016.

METHODS

We reviewed data from a survey of end-users received between September 5, 2016, and June 10, 2019. The survey assessed: (1) nature of decision-making needs; (2) actions resulting from the report's findings; (3) implementation timeframe; and (4) overall perception of report content.

RESULTS

Feedback was received from 66 of the 138 operational partners requesting ESP products during the fiscal year 2015 through 2018. Requests commonly informed clinical guidance (58%), identified future research needs (58%), and determined VHA-specific implementation strategy (47%). A total of 91% of responders used reports, typically within 3 months after completion (82%). Use was typically for VHA publications and/or presentations to inform VHA policy or guidance (26%), to inform intervention/strategy adoption decisions (23%) and for medical device and therapy procurement decisions (21%). Over half (53%) of respondents indicated that it would be useful for ESP reports to include more guidance on implementing findings.

CONCLUSION

Our survey of learning health system decision-makers' actual patterns and timing of evidence use provides valuable new information that can further support development of other health system and evidence producer partnerships and identifies key needs for better supporting health systems' uptake of evidence.

Convergent and sequential synthesis designs: implications for conducting and reporting systematic reviews of qualitative and quantitative evidence

BACKGROUND

Systematic reviews of qualitative and quantitative evidence can provide a rich understanding of complex phenomena. This type of review is increasingly popular, has been used to provide a landscape of existing knowledge, and addresses the types of questions not usually covered in reviews relying solely on either quantitative or qualitative evidence. Although several typologies of synthesis designs have been developed, none have been tested on a large sample of reviews. The aim of this review of reviews was to identify and develop a typology of synthesis designs and methods that have been used and to propose strategies for synthesizing qualitative and quantitative evidence.

METHODS

A review of systematic reviews combining qualitative and quantitative evidence was performed. Six databases were searched from inception to December 2014. Reviews were included if they were systematic reviews combining qualitative and quantitative evidence. The included reviews were analyzed according to three concepts of synthesis processes: (a) synthesis methods, (b) sequence of data synthesis, and (c) integration of data and synthesis results.

RESULTS

A total of 459 reviews were included. The analysis of this literature highlighted a lack of transparency in reporting how evidence was synthesized and a lack of consistency in the terminology used. Two main types of synthesis designs were identified: convergent and sequential synthesis designs. Within the convergent synthesis design, three subtypes were found: (a) data-based convergent synthesis design, where qualitative and quantitative evidence is analyzed together using the same synthesis method, (b) results-based convergent synthesis design, where qualitative and quantitative evidence is analyzed separately using different synthesis methods and results of both syntheses are integrated during a final synthesis, and (c) parallel-results convergent synthesis design consisting of independent syntheses of qualitative and quantitative evidence and an interpretation of the results in the discussion.

CONCLUSIONS

Performing systematic reviews of qualitative and quantitative evidence is challenging because of the multiple synthesis options. The findings provide guidance on how to combine qualitative and quantitative evidence. Also, recommendations are made to improve the conducting and reporting of this type of review.