What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary

Synthesising evidence on patient portals: a protocol for an umbrella review

INTRODUCTION

Over the last two decades, patient portals have emerged as a noticeable eHealth strategy. To date, research on patient portals has been rapidly increasing. Our umbrella review aims to provide a meta-level synthesis to make sense of the evidence on patient portals from published systematic reviews (SRs).

METHODS

We will employ a modified version of the Joanna Briggs Institute umbrella review method. The search strategy encompasses multiple databases. The inclusion criterion is specific to SRs focused on patient portal. Patients or public were not involved in this work.

ANALYSIS

Two researchers will independently screen titles/abstracts and then full-text articles against the inclusion/exclusion criteria. Methodological quality of included reviews will be assessed and data will be extracted from the final selection of reviews. These reviews will be categorised into quantitative, qualitative and/or mixed-synthesis groups based on information about the design of primary studies provided in the reviews. Correspondingly, we will create quantitative, qualitative and/or mixed-synthesis Excel data-extraction tables. Within each table, data will be extracted with the reference to primary studies as reported in the reviews, and will be synthesised into themes and then a smaller number of findings/outcomes. Modified Grading of Recommendations Assessment, Development and Evaluation (GRADE) and Confidence in the Evidence from Reviews of Qualitative research (CERQual) tools will be applied to assess the strength of evidence at the level of each finding/outcome. The output of our umbrella review will consist of summary of findings tables and evidence profile tables. A narrative meta-level synthesis will be provided. We will use the clinical adoption meta-model as an organising framework.

ETHICS AND DISSEMINATION

As an outcome of this review, we will create a guidance and roadmap to be used in a future Delphi study to gather feedback from Canadian eHealth stakeholders. We will also present at conferences and publish the final report. The umbrella review does not require ethical approval.

PROSPERO REGISTRATION NUMBER

CRD42018096657.

Network meta-analysis of randomised trials of pharmacological, psychotherapeutic, exercise and collaborative care interventions for depressive symptoms in patients with coronary artery disease: hybrid systematic review of systematic reviews protocol

BACKGROUND

Depression is common in patients with coronary artery disease (CAD) and is associated with poorer outcomes and higher costs. Several randomised controlled trials (RCTs) targeting depression, of various modalities (including pharmacological, psychotherapeutic and other approaches), have been conducted and summarised in pairwise meta-analytic reviews. However, no study has considered the cumulative evidence within a network, which can provide valuable indirect comparisons and information about the relative efficacy of interventions. Therefore, we will adopt a review of review methodology to develop a network meta-analysis (NMA) of depression interventions for depression in CAD.

METHODS

We will search relevant databases from inception for systematic reviews of RCTs of depression treatments for people with CAD, supplementing this with comprehensive searches for recent or ongoing studies. We will extract data from and summarise characteristics of individual RCTs, including participants, study characteristics, outcome measures and adverse events. Cochrane risk of bias ratings will also be extracted or if not present will be conducted by the authors. RCTs that compare depression treatments (grouped as pharmacological, psychotherapeutic, combined pharmacological/psychotherapeutic, exercise, collaborative care) to placebo, usual care, waitlist control or attention controls, or directly in head-to-head comparisons, will be included. Primary outcomes will be the change in depressive symptoms (summarised with a standardised mean difference) and treatment acceptability (treatment discontinuation: % of people who withdrew). Secondary outcomes will include change in 6-month depression outcomes, health-related quality of life (HRQoL), mortality, cardiovascular morbidity, health services use and adverse events. Secondary analyses will form further networks with individual anti-depressants and psychotherapies. We will use frequentist, random effects multivariate network meta-analysis to synthesise the evidence for depression intervention and to achieve a ranking of treatments, using Stata. Rankograms and surface under the cumulative ranking curves will be used for treatment ranking. Local and global methods will evaluate consistency. GRADE will be used to assess evidence quality for primary outcomes.

DISCUSSION

The present review will address uncertainties about the evidence in terms of depression management in CAD and may allow for a ranking of treatments, including providing important information for future research efforts.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42018108293.

Effects of prevention and harm reduction interventions on gambling behaviours and gambling related harm: An umbrella review

BACKGROUND

Harms related to gambling have been found not only to affect problem gamblers, but also to occur amongst low- and moderate-risk gamblers. This has resulted in calls for a public health approach to address a possible 'prevention paradox' in gambling related harm. The aim of this study was to evaluate the systematic review evidence base on the effects of prevention and harm reduction interventions on gambling behaviours, and gambling related harm. We also aimed to examine differential effects of interventions across socio-demographic groups.

METHODS

Systematic methods were used to locate and evaluate published systematic reviews of prevention and harm reduction interventions. We designed the review using the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) Equity extension Guidelines. Four databases were searched from their start date until May 2018. The quality of the included articles was determined using A MeaSurement Tool to Assess systematic Reviews (AMSTAR 2).

RESULTS

Ten systematic reviews were identified reporting 55 unique relevant primary studies. Much of the review evidence-base related to pre-commitment and limit setting (24%), self-exclusion (20%), youth prevention programmes (20%), and machine messages/feedback (20%). The effectiveness of harm reduction interventions are limited by the extent to which users adhere to voluntary systems. Less than half of studies examining youth prevention programmes demonstrated positive effects on behaviour. No review extracted data or reported on the differential effects of intervention strategies across sociodemographic groups. The quality of the included reviews (and their primary studies) were generally poor and clear gaps in the evidence base have been highlighted.

CONCLUSIONS

The evidence base is dominated by evaluations of individual-level harm reduction interventions, with a paucity of research on supply reduction interventions. Review conclusions are limited by the quality and robustness of the primary research. Future research should consider the equity effects of intervention strategies.

Knowledge Synthesis in Evidence-Based Medicine

Systematic reviews are the most common form of knowledge synthesis and remain a cornerstone of the practice of evidence-based medicine. They offer enhanced rigor and validity relative to traditional narrative review articles by reducing bias and increasing objectivity. In answering focused research questions, systematic reviews are directly applicable to clinical practice as well as the development of clinical guidelines and the identification of knowledge gaps, which may drive future primary research directions. Typically, such a rigorous process necessarily requires substantive time to carefully and systematically identify, screen, and synthesize all relevant available primary research on a topic. Further, other knowledge synthesis methods have emerged to address the varying needs of decision makers with respect to condensed timelines and more diverse research questions, as well as to allow incorporation of already synthesized evidence into reviews. These alternative methods include rapid reviews, scoping reviews, and overviews of systematic reviews, which are being used with increasing frequency by clinicians, decision-makers, and researchers. We encourage clinicians and researchers in nuclear medicine and other imaging sciences to acquire a greater familiarity with these methods and to consider them in clinical decision making, the development of clinical guidelines, and the planning of future research activities.

The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective

INTRODUCTION

Evidence-to-decision (EtD) frameworks intend to ensure that all criteria of relevance to a health decision are systematically considered. This paper, part of a series commissioned by the WHO, reports on the development of an EtD framework that is rooted in WHO norms and values, reflective of the changing global health landscape, and suitable for a range of interventions and complexity features. We also sought to assess the value of this framework to decision-makers at global and national levels, and to facilitate uptake through suggestions on how to prioritise criteria and methods to collect evidence.

METHODS

In an iterative, principles-based approach, we developed the framework structure from WHO norms and values. Preliminary criteria were derived from key documents and supplemented with comprehensive subcriteria obtained through an overview of systematic reviews of criteria employed in health decision-making. We assessed to what extent the framework can accommodate features of complexity, and conducted key informant interviews among WHO guideline developers. Suggestions on methods were drawn from the literature and expert consultation.

RESULTS

The new WHO-INTEGRATE (INTEGRATe Evidence) framework comprises six substantive criteria-balance of health benefits and harms, human rights and sociocultural acceptability, health equity, equality and non-discrimination, societal implications, financial and economic considerations, and feasibility and health system considerations-and the meta-criterion quality of evidence. It is intended to facilitate a structured process of reflection and discussion in a problem-specific and context-specific manner from the start of a guideline development or other health decision-making process. For each criterion, the framework offers a definition, subcriteria and example questions; it also suggests relevant primary research and evidence synthesis methods and approaches to assessing quality of evidence.

CONCLUSION

The framework is deliberately labelled version 1.0. We expect further modifications based on focus group discussions in four countries, example applications and input across concerned disciplines.

A decision tool to help researchers make decisions about including systematic reviews in overviews of reviews of healthcare interventions

BACKGROUND

Overviews of reviews of healthcare interventions (overviews) integrate information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. Overviews may identify multiple SRs that examine the same intervention for the same condition and include some, but not all, of the same primary studies. Different researchers use different approaches to manage these "overlapping SRs," but each approach has advantages and disadvantages. This study aimed to develop an evidence-based decision tool to help researchers make informed inclusion decisions when conducting overviews of healthcare interventions.

METHODS

We used a two-stage process to develop the decision tool. First, we conducted a multiple case study to obtain empirical evidence upon which the tool is based. We systematically conducted seven overviews five times each, making five different decisions about which SRs to include in the overviews, for a total of 35 overviews; we then examined the impact of the five inclusion decisions on the overviews' comprehensiveness and challenges, within and across the seven overview cases. Second, we used a structured, iterative process to transform the evidence obtained from the multiple case study into an empirically based decision tool with accompanying descriptive text.

RESULTS

The resulting decision tool contains four questions: (1) Do Cochrane SRs likely examine all relevant intervention comparisons and available data? (2) Do the Cochrane SRs overlap? (3) Do the non-Cochrane SRs overlap? (4) Are researchers prepared and able to avoid double-counting outcome data from overlapping SRs, by ensuring that each primary study's outcome data are extracted from overlapping SRs only once? Guidance is provided to help researchers answer each question, and empirical evidence is provided regarding the advantages, disadvantages, and potential trade-offs of the different inclusion decisions.

CONCLUSIONS

This evidence-based decision tool is designed to provide researchers with the knowledge and means to make informed inclusion decisions in overviews. The tool can provide practical guidance and support for overview authors by helping them consider questions that could affect the comprehensiveness and complexity of their overviews. We hope this tool will be a useful resource for researchers conducting overviews, and we welcome discussion, testing, and refinement of the proposed tool.

Effectiveness of stop smoking interventions among adults: protocol for an overview of systematic reviews and an updated systematic review

BACKGROUND

Tobacco smoking is the leading cause of cancer, preventable death, and disability. Smoking cessation can increase life expectancy by nearly a decade if achieved in the third or fourth decades of life. Various stop smoking interventions are available including pharmacotherapies, electronic cigarettes, behavioural support, and alternative therapies. This protocol outlines an evidence review which will evaluate the benefits and harms of stop smoking interventions in adults.

METHODS

The evidence review will consist of two stages. First, an overview of systematic reviews evaluating the benefits and harms of various stop smoking interventions delivered in or referred from the primary care setting will be conducted. The second stage will involve updating a systematic review on electronic cigarettes identified in the overview; randomized controlled trials will be considered for outcomes relating to benefits while randomized controlled trials, non-randomized controlled trials, and comparative observational studies will be considered for evaluating harms. Search strategies will be developed and peer-reviewed by medical information specialists. The search strategy for the updated review on e-cigarettes will be developed using that of the candidate systematic review. The MEDLINE®, PsycINFO, Embase, and the Cochrane Library electronic databases will be searched as of 2008 for the overview of reviews and from the last search date of the selected review for the updated review. Organizational websites and trial registries will be searched for unpublished or ongoing reviews/studies. Two reviewers will independently screen the title and abstracts of citations using the liberal accelerated method. Full-text screening will be performed independently by two reviewers. Extracted data will be verified by a second reviewer. Disagreements regarding full-text screening and data extraction will be resolved by consensus or third-party adjudication. The methodological quality of systematic reviews, risk of bias of randomized and non-randomized trials, and methodological quality of cohort studies will be evaluated using AMSTAR 2, the Cochrane risk of bias tool, and a modified version of the Scottish Intercollegiate Guidelines Network critical appraisal tool, respectively. The GRADE framework will be used to assess the quality of the evidence for outcomes.

DISCUSSION

The evidence review will evaluate the benefits and harms of various stop smoking interventions for adults. Findings will be used to inform a national tobacco cessation guideline by the Canadian Task Force on Preventive Health Care.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO (CRD42018099691, CRD42018099692).

The impact of different inclusion decisions on the comprehensiveness and complexity of overviews of reviews of healthcare interventions

BACKGROUND

Overviews of reviews (overviews) compile information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. Overviews may identify multiple SRs that examine the same intervention for the same condition and include some, but not all, of the same primary studies. There is currently limited guidance on whether and how to include these overlapping SRs in overviews. Our objectives were to assess how different inclusion decisions in overviews of healthcare interventions affect their comprehensiveness and results, and document challenges encountered when making different inclusion decisions in overviews.

METHODS

We used five inclusion decisions to conduct overviews across seven topic areas, resulting in 35 overviews. The inclusion decisions were (1) include all Cochrane and non-Cochrane SRs, (2) include only Cochrane SRs, or consider all Cochrane and non-Cochrane SRs but include only non-overlapping SRs, and in the case of overlapping SRs, select (3) the Cochrane SR, (4) the most recent SR (by publication or search date), or (5) the highest quality SR (assessed using AMSTAR). For each topic area and inclusion scenario, we documented the amount of outcome data lost and changed and the challenges involved.

RESULTS

When conducting overviews, including only Cochrane SRs, instead of all SRs, often led to loss/change of outcome data (median 31% of outcomes lost/changed; range 0-100%). Considering all Cochrane and non-Cochrane SRs but including only non-overlapping SRs and selecting the Cochrane SR for groups of overlapping SRs (instead of the most recent or highest quality SRs) allowed the most outcome data to be recaptured (median 42% of lost/changed outcome recaptured; range 28-86%). Across all inclusion scenarios, challenges were encountered when extracting data from overlapping SRs.

CONCLUSIONS

Overlapping SRs present a methodological challenge for overview authors. This study demonstrates that different inclusion decisions affect the comprehensiveness and results of overviews in different ways, depending in part on whether Cochrane SRs examine all intervention comparisons relevant to the overview. Study results were used to develop an evidence-based decision tool that provides practical guidance for overview authors.

A review protocol on research partnerships: a Coordinated Multicenter Team approach

BACKGROUND

Research partnership approaches, in which researchers and stakeholders work together collaboratively on a research project, are an important component of research, knowledge translation, and implementation. Despite their growing use, a comprehensive understanding of the principles, strategies, outcomes, and impacts of different types of research partnerships is lacking. Generating high-quality evidence in this area is challenging due to the breadth and diversity of relevant literature. We established a Coordinated Multicenter Team approach to identify and synthesize the partnership literature and better understand the evidence base. This review protocol outlines an innovative approach to locating, reviewing, and synthesizing the literature on research partnerships.

METHODS

Five reviews pertaining to research partnerships are proposed. The Coordinated Multicenter Team developed a consensus-driven conceptual framework to guide the reviews. First, a review of reviews will comparatively describe and synthesize key domains (principles, strategies, outcomes, and impacts) for different research partnership approaches, within and beyond health (e.g., integrated knowledge translation, participatory action research). After identifying commonly used search terminology, three complementary scoping reviews will describe and synthesize these domains in the health research partnership literature. Finally, an umbrella review will amalgamate and reflect on the collective findings and identify research gaps and future directions. We will develop a collaborative review methodology, comprising search strategy efficiencies, terminology standardization, and the division of screening, extraction, and synthesis to optimize feasibility and literature capture. A series of synthesis and scoping manuscripts will emerge from this Coordinated Multicenter Team approach.

DISCUSSION

Comprehensively describing and differentiating research partnership terminology and its domains will address well-documented gaps in the literature. These efforts will contribute to and improve the quality, conduct, and reporting of research partnership literature. The collaborative review methodology will help identify and establish common terms, leverage efficiencies (e.g., expertise, experience, search and protocol design, resources) and optimize research feasibility and quality. Our approach allows for enhanced scope and inclusivity of all research user groups and domains, thereby contributing uniquely to the literature. This multicenter, efficiency and quality-focused approach may serve to inspire researchers across the globe in addressing similar domain challenges, as exist in this rapidly expanding field.

Medication adherence interventions and outcomes: an overview of systematic reviews

Objective

To present evidence for healthcare-provided medication adherence interventions on clinical, economic and humanistic outcomes among patients.

Methods

Literature search of systematic reviews in Medline, Embase and CINAHL (2007-2017), validation of quality using A Measurement Tool to Assess Systematic Reviews (AMSTAR) 2 and Preferred Reporting Items for Systematic Reviews and Meta-Analyses questionnaires and, finally, extraction, combination and tabulation of results for included studies.

Results

From eight systematic reviews with medium to high AMSTAR 2 score, 37 randomised controlled studies involving 28 600 participants were extracted. Patient education and counselling showed some positive effects on medication adherence. Patient education also showed some positive effects on morbidity, healthcare utilities and patient satisfaction. Counselling had some benefit on mortality and healthcare utilisation. Simplifying doses was shown to have some benefit on morbidity and patient satisfaction. Interventions delivered by pharmacists and nurses showed a better result in improving adherence and outcomes than interventions delivered by general practitioners.

Conclusions

Some interventions were found to have positive effect on adherence and outcomes, but no single strategy showed improvement in all settings. For future research patients should be screened for non-adherence to reveal both if they are non-adherent and type of non-adherence, as well as bigger sample sizes and longer duration of follow-up.

Interventions during pregnancy to prevent preterm birth: an overview of Cochrane systematic reviews

BACKGROUND

Preterm birth (PTB) is a major factor contributing to global rates of neonatal death and to longer-term health problems for surviving infants. Both the World Health Organization and the United Nations consider prevention of PTB as central to improving health care for pregnant women and newborn babies. Current preventative clinical strategies show varied efficacy in different populations of pregnant women, frustrating women and health providers alike, while researchers call for better understanding of the underlying mechanisms that lead to PTB.

OBJECTIVES

We aimed to summarise all evidence for interventions relevant to the prevention of PTB as reported in Cochrane systematic reviews (SRs). We intended to highlight promising interventions and to identify SRs in need of an update.

METHODS

We searched the Cochrane Database of Systematic Reviews (2 November 2017) with key words to capture any Cochrane SR that prespecified or reported a PTB outcome. Inclusion criteria focused on pregnant women without signs of preterm labour or ruptured amniotic membranes. We included reviews of interventions for pregnant women irrespective of their risk status. We followed standard Cochrane methods.We applied GRADE criteria to evaluate the quality of SR evidence. We assigned graphic icons to classify the effectiveness of interventions as: clear evidence of benefit; clear evidence of harm; clear evidence of no effect or equivalence; possible benefit; possible harm; or unknown benefit or harm. We defined clear evidence of benefit and clear evidence of harm to be GRADE moderate- or high-quality evidence with a confidence interval (CI) that does not cross the line of no effect. Clear evidence of no effect or equivalence is GRADE moderate- or high-quality evidence with a narrow CI crossing the line of no effect. Possible benefit and possible harm refer to GRADE low-quality evidence with a clear effect (CI does not cross the line of no effect) or GRADE moderate- or high-quality evidence with a wide CI. Unknown harm or benefit refers to GRADE low- or very low-quality evidence with a wide CI.

MAIN RESULTS

We included 83 SRs; 70 had outcome data. Below we highlight key results from a subset of 36 SRs of interventions intended to prevent PTB.

OUTCOME

preterm birthClear evidence of benefitFour SRs reported clear evidence of benefit to prevent specific populations of pregnant women from giving birth early, including midwife-led continuity models of care versus other models of care for all women; screening for lower genital tract infections for pregnant women less than 37 weeks' gestation and without signs of labour, bleeding or infection; and zinc supplementation for pregnant women without systemic illness. Cervical cerclage showed clear benefit for women with singleton pregnancy and high risk of PTB only.Clear evidence of harmNo included SR reported clear evidence of harm.No effect or equivalenceFor pregnant women at high risk of PTB, bedrest for women with singleton pregnancy and antibiotic prophylaxis during the second and third trimester were of no effect or equivalent to a comparator.Possible benefitFour SRs found possible benefit in: group antenatal care for all pregnant women; antibiotics for pregnant women with asymptomatic bacteriuria; pharmacological interventions for smoking cessation for pregnant women who smoke; and vitamin D supplements alone for women without pre-existing conditions such as diabetes.Possible harmOne SR reported possible harm (increased risk of PTB) with intramuscular progesterone, but this finding is only relevant to women with multiple pregnancy and high risk of PTB. Another review found possible harm with vitamin D, calcium and other minerals for pregnant women without pre-existing conditions.

OUTCOME

perinatal deathClear evidence of benefitTwo SRs reported clear evidence of benefit to reduce pregnant women's risk of perinatal death: midwife-led continuity models of care for all pregnant women; and fetal and umbilical Doppler for high-risk pregnant women.Clear evidence of harmNo included SR reported clear evidence of harm.No effect or equivalenceFor pregnant women at high risk of PTB, antibiotic prophylaxis during the second and third trimester was of no effect or equivalent to a comparator.Possible benefitOne SR reported possible benefit with cervical cerclage for women with singleton pregnancy and high risk of PTB.Possible harmOne SR reported possible harm associated with a reduced schedule of antenatal visits for pregnant women at low risk of pregnancy complications; importantly, these women already received antenatal care in settings with limited resources.

OUTCOMES

preterm birth and perinatal deathUnknown benefit or harmFor pregnant women at high risk of PTB for any reason including multiple pregnancy, home uterine monitoring was of unknown benefit or harm. For pregnant women at high risk due to multiple pregnancy: bedrest, prophylactic oral betamimetics, vaginal progesterone and cervical cerclage were all of unknown benefit or harm.

AUTHORS' CONCLUSIONS

Implications for practiceThe overview serves as a map and guide to all current evidence relevant to PTB prevention published in the Cochrane Library. Of 70 SRs with outcome data, we identified 36 reviews of interventions with the aim of preventing PTB. Just four of these SRs had evidence of clear benefit to women, with an additional four SRs reporting possible benefit. No SR reported clear harm, which is an important finding for women and health providers alike.The overview summarises no evidence for the clinically important interventions of cervical pessary, cervical length assessment and vaginal progesterone because these Cochrane Reviews were not current. These are active areas for PTB research.The graphic icons we assigned to SR effect estimates do not constitute clinical guidance or an endorsement of specific interventions for pregnant women. It remains critical for pregnant women and their healthcare providers to carefully consider whether specific strategies to prevent PTB will be of benefit for individual women, or for specific populations of women.Implications for researchFormal consensus work is needed to establish standard language for overviews of reviews and to define the limits of their interpretation.Clinicians, researchers and funders must address the lack of evidence for interventions relevant to women at high risk of PTB due to multiple pregnancy.

Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence

BACKGROUND

Overviews of systematic reviews (SRs) attempt to systematically retrieve and summarise the results of multiple systematic reviews. This is the second of two papers from a study aiming to develop a comprehensive evidence map of the methods used in overviews. Our objectives were to (a) develop a framework of methods for conducting, interpreting and reporting overviews (stage I)-the Methods for Overviews of Reviews (MOoR) framework-and (b) to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II). In the first paper, we reported findings for the four initial steps of an overview (specification of purpose, objectives and scope; eligibility criteria; search methods; data extraction). In this paper, we report the remaining steps: assessing risk of bias; synthesis, presentation and summary of the findings; and assessing certainty of the evidence arising from the overview.

METHODS

In stage I, we identified cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews. Based on these studies, we developed a framework of possible methods for overviews, categorised by the steps in conducting an overview. Multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified studies evaluating methods and mapped these evaluations to the framework.

RESULTS

Forty-two stage I studies described methods relevant to one or more of the latter steps of an overview. Six studies evaluating methods were included in stage II. These mapped to steps involving (i) the assessment of risk of bias (RoB) in SRs (two SRs and three primary studies, all reporting evaluation of RoB tools) and (ii) the synthesis, presentation and summary of the findings (one primary study evaluating methods for measuring overlap).

CONCLUSION

Many methods have been described for use in the latter steps in conducting an overview; however, evaluation and guidance for applying these methods is sparse. The exception is RoB assessment, for which a multitude of tools exist-several with sufficient evaluation and guidance to recommend their use. Evaluation of other methods is required to provide a comprehensive evidence map.

Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: protocol for a descriptive analytic study

BACKGROUND

Systematic reviews (SRs) of randomised controlled trials (RCTs) can provide the best evidence to inform decision-making, but their methodological and reporting quality varies. Tools exist to guide the critical appraisal of quality and risk of bias in SRs, but evaluations of their measurement properties are limited. We will investigate the interrater reliability (IRR), usability, and applicability of A MeaSurement Tool to Assess systematic Reviews (AMSTAR), AMSTAR 2, and Risk Of Bias In Systematic reviews (ROBIS) for SRs in the fields of biomedicine and public health.

METHODS

An international team of researchers at three collaborating centres will undertake the study. We will use a random sample of 30 SRs of RCTs investigating therapeutic interventions indexed in MEDLINE in February 2014. Two reviewers at each centre will appraise the quality and risk of bias in each SR using AMSTAR, AMSTAR 2, and ROBIS. We will record the time to complete each assessment and for the two reviewers to reach consensus for each SR. We will extract the descriptive characteristics of each SR, the included studies, participants, interventions, and comparators. We will also extract the direction and strength of the results and conclusions for the primary outcome. We will summarise the descriptive characteristics of the SRs using means and standard deviations, or frequencies and proportions. To test for interrater reliability between reviewers and between the consensus agreements of reviewer pairs, we will use Gwet's AC₁ statistic. For comparability to previous evaluations, we will also calculate weighted Cohen's kappa and Fleiss' kappa statistics. To estimate usability, we will calculate the mean time to complete the appraisal and to reach consensus for each tool. To inform applications of the tools, we will test for statistical associations between quality scores and risk of bias judgments, and the results and conclusions of the SRs.

DISCUSSION

Appraising the methodological and reporting quality of SRs is necessary to determine the trustworthiness of their conclusions. Which tool may be most reliably applied and how the appraisals should be used is uncertain; the usability of newly developed tools is unknown. This investigation of common (AMSTAR) and newly developed (AMSTAR 2, ROBIS) tools will provide empiric data to inform their application, interpretation, and refinement.

Quality ratings of reviews in overviews: a comparison of reviews with and without dual (co-)authorship

BACKGROUND

Previous research shows that many authors of Cochrane overviews were also involved in some of the included systematic reviews (SRs). This type of dual (co-)authorship (DCA) may be a conflict of interest and a potential source of bias. Our objectives were to (1) additionally investigate DCA in non-Cochrane overviews; (2) investigate whether there is an association between DCA and quality assessments of SRs in Cochrane and non-Cochrane overviews.

METHODS

We selected a sample of Cochrane (n = 20) and non-Cochrane (n = 78) overviews for analysis. We extracted data on the number of reviews affected by DCA and whether quality assessment of included reviews was conducted independently. Differences in mean quality scores between SRs with and without DCA were calculated in each overview. These differences were standardized (using the standardized mean difference (SMD)) and meta-analyzed using a random effects model.

RESULTS

Forty out of 78 non-Cochrane overviews (51%) and 18 out of 20 Cochrane overviews (90%) had included at least one SR with DCA. For Cochrane overviews, a median of 5 [interquartile range (IQR) 2.5 to 7] SRs were affected by DCA (median of included reviews 10). For non-Cochrane overviews a median of 1 [IQR 0 to 2] of the included SRs were affected (median of included reviews 14). The meta-analysis showed a SMD of 0.58 (95% confidence interval (CI) 0.27 to 0.90) indicating higher quality scores in reviews with overlapping authors. The test for subgroup differences shows no evidence of a difference between Cochrane (SMD 0.44; 95% CI 0.07 to 0.81) and non-Cochrane overviews (SMD 0.62; 95% CI 0.06 to 1.17).

CONCLUSIONS

Many authors of overviews also often have an authorship on one or more of the underlying reviews. Our analysis shows that, on average, authors of overviews give higher quality ratings to SRs in which they were involved themselves than to other SRs. Conflict of interest is one explanation, but there are several others such as reviewer expertise. Independent and blinded reassessments of the reviews would provide more robust evidence on potential bias arising from DCA.

Long-acting insulin analogues for type 1 diabetes: An overview of systematic reviews and meta-analysis of randomized controlled trials

Background

The comparison between long acting insulin analogues (LAIA) and human insulin (NPH) has been investigated for decades, with many randomized controlled trials (RCTs) and systematic reviews giving mixed results. This overlapping and contradictory evidence has increased uncertainty on coverage decisions at health systems level.

Aim

To conduct an overview of systematic reviews and update existing reviews, preparing new meta-analysis to determine whether LAIA are effective for T1D patients compared to NPH.

Methods

We identified systematic reviews of RCTs that evaluated the efficacy of LAIA glargine or detemir, compared to NPH insulin for T1D, assessing glycated hemoglobin (A1C) and hypoglycemia. Data sources included Pubmed, Cochrane Library, EMBASE and hand-searching. The methodological quality of studies was independently assessed by two reviewers, using AMSTAR and Jadad scale. We found 11 eligible systematic reviews that contained a total of 25 relevant clinical trials. Two reviewers independently abstracted data.

Results

We found evidence that LAIA are efficacious compared to NPH, with estimates showing a reduction in nocturnal hypoglycemia episodes (RR 0.66; 95% CI 0.57; 0.76) and A1C (95% CI 0.23; 0.12). No significance was found related to severe hypoglycemia (RR 0.94; 95% CI 0.71; 1.24).

Conclusion

This study design has allowed us to carry out the most comprehensive assessment of RCTs on this subject, filling a gap in diabetes research. Our paper addresses a question that is important not only for decision makers but also for clinicians.

Quality assessment versus risk of bias in systematic reviews: AMSTAR and ROBIS had similar reliability but differed in their construct and applicability

OBJECTIVES

The objective of the study was to assess the inter-rater reliability (IRR) of AMSTAR and ROBIS in judging individual domains and overall methodological quality/risk of bias of systematic reviews, the concurrent validity of the tools, and the time required to apply them.

STUDY DESIGN AND SETTING

This is a cross-sectional study. Five raters independently read 31 systematic reviews and applied AMSTAR and ROBIS. Fleiss' k for multiple raters for individual domains and overall methodological quality/risk of bias was calculated. Similar domains assessed by both tools and final scores were matched to explore the concurrent validity, using the Kendall tau correlation.

RESULTS

IRR ranged from fair to perfect for AMSTAR and from moderate to substantial for ROBIS. Kappa for overall quality/risk of bias was 0.73 (95% confidence interval [CI] 0.65-0.81) for AMSTAR and 0.64 (95% CI 0.54-0.74) for ROBIS. We judged most of the reviews at intermediate quality with AMSTAR (53%), while judgments were split in high (53%) and low (47%) risk of bias with ROBIS. The correlation between judgments on similar domains ranged from moderate to high, while it was fair on the overall judgment (K = 0.35, 95% CI 0.21-0.49). The mean time to complete ROBIS was about double that for AMSTAR.

CONCLUSION

AMSTAR and ROBIS offer similar IRR but differ in their construct and applicability.

An introduction to overviews of reviews: planning a relevant research question and objective for an overview

BACKGROUND

Overviews of systematic reviews are a relatively new approach to synthesising evidence, and research methods and associated guidance are developing. Within this paper we aim to help readers understand key issues which are essential to consider when taking the first steps in planning an overview. These issues relate to the development of clear, relevant research questions and objectives prior to the development of an overview protocol.

METHODS

Initial discussions and key concepts for this paper were formed during a workshop on overview methods at the 2016 UK Cochrane Symposium, at which all members of this author group presented work and contributed to wider discussions. Detailed descriptions of the various key features of overviews and their different objectives were created by the author group based upon current evidence (Higgins J, Green S. Cochrane Handbook Syst Rev Interv. 2011;4:5, Pollock M, et al. Sys Rev. 2016;5:190-205, Pollock A, et al. Cochrane overviews of reviews: exploring the methods and challenges. UK and Ireland: Cochrane Symposium; 2016, Pieper D, et al. Res Syn Meth. 2014;5:187-99, Lunny C, et al. Sys Rev. 2016;5:4-12, Hartling L, et al. Comparing multiple treatments: an introduction to overviews of reviews. In 23rd Cochrane Colloquium; 2015, Hartling L, et al. Plos One. 2012;7:1-8, Ballard M, Montgomery P. Res Syn Meth. 2017;8:92-108) and author experiences conducting overviews.

RESULTS

Within this paper we introduce different types of overviews and suggest common research questions addressed by these overviews. We briefly reflect on the key features and objectives of the example overviews discussed.

CONCLUSIONS

Clear decisions relating to the research questions and objectives are a fundamental first step during the initial planning stages for an overview. Key stakeholders should be involved at the earliest opportunity to ensure that the planned overview is relevant and meaningful to the potential end users of the overview. Following best practice in common with other forms of systematic evidence synthesis, an overview protocol should be published, ensuring transparency and reducing opportunities for introduction of bias in the conduct of the overview.

Same family, different species: methodological conduct and quality varies according to purpose for five types of knowledge synthesis

OBJECTIVES

The aim of the study was to characterize methodological conduct, reporting, and quality of five knowledge synthesis (KS) approaches.

STUDY DESIGN AND SETTING

Retrospective analysis of a convenience sample of five published databases of KS approaches: overview of reviews (n = 74), scoping reviews (n = 494), rapid reviews (n = 84), systematic reviews (n = 300), and network meta-analyses (NMAs; n = 456). Data in the five published databases were abstracted by two reviewers independently, any missing data for this retrospective analysis were abstracted by one experienced reviewer. Methods were appraised using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) tool. Descriptive analysis was performed.

RESULTS

Reporting the use of a protocol ranged from 4% for rapid reviews to 32% for systematic reviews. The use of two reviewers for citation and full-text screening ranged from 20% for scoping reviews to 60% for NMAs. Data abstraction was performed in duplicate for 11% of rapid reviews and 54% of NMAs, and for risk of bias appraisal, this ranged from 6% for scoping reviews to 41% for NMAs. NMAs had the highest median percentage of maximum obtainable AMSTAR score (64%; Q1-Q3:45-73%), while scoping reviews had the lowest (25%; Q1-Q3:13-38%).

CONCLUSION

NMAs consistently scored the highest on the AMSTAR tool likely because the purpose is to estimate treatment effects statistically. Scoping reviews scored the lowest (even after adjusting the score for not relevant items) likely because the purpose is to characterize the literature.

Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction

BACKGROUND

Overviews of systematic reviews attempt to systematically retrieve and summarise the results of multiple systematic reviews. Methods for conducting, interpreting and reporting overviews are in their infancy. To date, there has been no evidence map of the methods used in overviews, thus making it difficult to determine the gaps and priorities for methods research. Our objectives were to develop and populate a comprehensive framework of methods for conducting, interpreting and reporting overviews (stage I) and to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II).

METHODS

We searched methods collections (e.g. Cochrane Methodology Register, Meth4ReSyn library, AHRQ Effective Health Care Program) to identify eligible studies for both stages of this research. In stage I, cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews were used to develop and populate the framework of methods. Drafts and multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified and described studies evaluating overview methods and mapped these evaluations to the framework.

RESULTS

In this paper, we present results for the four initial steps of conducting an overview: (a) specification of the purpose, objectives and scope, (b) specification of the eligibility criteria, (c) search methods and (d) data extraction. Twenty-nine studies mentioned or described methods relevant to one or more of these steps. In the developed framework, identified methods and approaches were grouped according to the steps an overview author would need to undertake. Fifteen studies evaluated identified methods, all of which mapped to the search methods step. These studies either reported the development and evaluation of a new search filter to retrieve systematic reviews or compared the performance of multiple filters.

CONCLUSION

Gaps in the evaluation of methods were found for the majority of steps in the framework. More empirical studies are needed to evaluate the methods outlined and provide a comprehensive evidence map. The framework is useful for planning these evaluations and for planning methods required to deal with challenges that arise when conducting an overview.

Effects of e-learning in a continuing education context on nursing care: a review of systematic qualitative, quantitative and mixed studies reviews (protocol)

INTRODUCTION

Continuing education (CE) is imperative to the future of professional nursing. The use of e-learning by registered nurses for CE is spreading. A review of systematic reviews will be conducted to develop a broad picture of the effects of e-learning in a CE context on nursing care.

METHODS AND ANALYSIS

Systematic qualitative, quantitative and mixed studies reviews published in English, French or Spanish from 1 January 2006 will be included. The outcomes of interest will be extracted and analysed inductively and deductively from the Nursing Care Performance Framework; some themes include nursing resources, nurses' practice environment, processes, professional satisfaction, and nursing sensitive outcomes. Three reviewers will independently screen first the title and abstract of the papers, and then the full texts in order to assess eligibility. Two teams of two reviewers will extract the selected reviews' characteristics and data. The results from various types of reviews will be integrated using a data-based convergent synthesis design. We will conduct a thematic synthesis and transform all quantitative and mixed data into qualitative data.

ETHICS AND DISSEMINATION

Ethics approval is not required for review of systematic reviews. We will summarise evidence concerning the negative, neutral and positive effects of various forms of e-learning on different aspects of nursing care. If we find gaps in the literature, we will highlight them and suggest ideas for further research. We will also focus on positive effects and present, if possible, the components and characteristics of e-learning interventions that were found to be successful. We will present this protocol and results in international conferences in nursing, medical, and health informatics domains. We will also submit the results of our work for peer-review publication in a journal indexed in the international bibliographic database of biomedical information.

Preferred reporting items for overviews of systematic reviews including harms checklist: a pilot tool to be used for balanced reporting of benefits and harms

OBJECTIVES

An overview of systematic reviews (OoSRs) is a study designed to synthesize multiple evidence from existing systematic reviews on a specific domain. The aim of this paper was to offer a pilot version checklist with Preferred Reporting Items for OoSRs (PRIO-harms) to promote a more balanced reporting of benefits and harms in OoSRs of health care interventions.

STUDY DESIGN AND SETTING

The included items were developed by combining key features from health care OoSRs designs with recommendations from statements of other relevant checklists and pertinent methodological review articles. Two raters independently used the PRIO-harms checklist to assess a sample of 20 OoSRs.

RESULTS

The PRIO-harms tool consists of a 27-item (56 (sub-)items in total) checklist and is accompanied by a five-stage process flow diagram (identification, screening, eligibility, inclusion, and separation of relevant studies). The mean interrater reliability (Gwet's AC1 statistic) between reviewers was 0.90 (95% confidence interval: 0.88, 0.92) indicating a very good agreement.

CONCLUSION

The PRIO-harms tool can be used in every OoSRs that addresses health care interventions. This instrument will assist overview authors to improve completeness and transparency of research reporting with emphasis on harms. However, it might benefit from critical review and further validation from experts and research teams that produce OoSRs.

Quality of Life and Breast Cancer: How Can Mind⁻Body Exercise Therapies Help? An Overview Study

Breast cancer survivors experience extensive treatments, threatening their quality of life. Complementary therapies used as a supplement to cancer treatment may control symptoms, enhance quality of life, and contribute to overall patient care. Mind–body exercise therapies might motivate cancer survivors to exercise, and assist them in regaining health. The purpose of this overview study is to study benefits from mind–body exercise of yoga, tai chi chuan and qigong upon quality of life in breast cancer populations. A systematic overview of reviews was applied. Literature search in five electronic databases and in reference lists was performed during April 2017. In addition, experts in the field were consulted. Of 38 identified titles, 11 review articles, including six meta-analyses were found eligible for review. Methodological quality was high for the majority of quality domains. Yoga, the most studied mind–body therapy, was found to benefit breast cancer patients’ psychological quality of life, while less support was established concerning physical quality of life elements. The evidence of improvements of quality of life from tai chi chuan and qigong remains unclear. Breast cancer survivors’ experiences of psychological and social well-being may be enhanced by practicing yoga.

Overviews of systematic reviews: great promise, greater challenge

The proliferation of systematic reviews and escalating demand from policy makers has driven a newer form of evidence synthesis-overviews of systematic reviews. Systematic Reviews are publishing a special thematic series on overviews and are encouraging submissions on the development and evaluation of methods for this review type. The authors' of this editorial introduce the series by considering challenges that arise when conducting an overview and what methods guidance is available. They emphasise the importance of evaluating overview methods to understand the trade-offs of using different approaches and propose that a more systematic and coordinated approach to methods development would be beneficial. Finally, they consider the potential for overviews to drive improvements in the conduct and reporting of systematic reviews.

Selecting and implementing overview methods: implications from five exemplar overviews

BACKGROUND

Overviews of systematic reviews are an increasingly popular method of evidence synthesis; there is a lack of clear guidance for completing overviews and a number of methodological challenges. At the UK Cochrane Symposium 2016, methodological challenges of five overviews were explored. Using data from these five overviews, practical implications to support methodological decision making of authors writing protocols for future overviews are proposed.

METHODS

Methods, and their justification, from the five exemplar overviews were tabulated and compared with areas of debate identified within current literature. Key methodological challenges and implications for development of overview protocols were generated and synthesised into a list, discussed and refined until there was consensus.

RESULTS

Methodological features of three Cochrane overviews, one overview of diagnostic test accuracy and one mixed methods overview have been summarised. Methods of selection of reviews and data extraction were similar. Either the AMSTAR or ROBIS tool was used to assess quality of included reviews. The GRADE approach was most commonly used to assess quality of evidence within the reviews. Eight key methodological challenges were identified from the exemplar overviews. There was good agreement between our findings and emerging areas of debate within a recent published synthesis. Implications for development of protocols for future overviews were identified.

CONCLUSIONS

Overviews are a relatively new methodological innovation, and there are currently substantial variations in the methodological approaches used within different overviews. There are considerable methodological challenges for which optimal solutions are not necessarily yet known. Lessons learnt from five exemplar overviews highlight a number of methodological decisions which may be beneficial to consider during the development of an overview protocol.

Extrapulmonary effects of continuous airway pressure on patients with obstructive sleep apnoea: protocol for an overview of systematic reviews

INTRODUCTION

Obstructive sleep apnoea (OSA) is associated with both pulmonary and extrapulmonary consequences. Continuous positive airway pressure (CPAP) is an effective treatment for pulmonary symptoms. However, it is not clear if CPAP modifies extrapulmonary symptoms, especially in the long term. To date, several studies addressing this issue have been performed, but the evidence is scattered across different reviews that address each outcome separately. Therefore, we will conduct an overview to assess and summarise all systematic reviews on the extrapulmonary effects of this intervention in patients with OSA.

METHODS

This is a protocol for a systematic overview of reviews that will follow the guidelines from the Cochrane Handbook of Systematic Reviews of Interventions and PRISMA-P. We will conduct comprehensive literature searches using MEDLINE, EMBASE, Cochrane, CINAHL, DARE and Epistemonikos. We will consider systematic reviews and meta-analyses of randomised controlled trials evaluating the extrapulmonary effects of CPAP on OSA. Two reviewers will independently screen articles for inclusion using criteria that are determined a priori, assessing the quality of included reviews and extracting relevant information. We will create matrices of evidence in the Epistemonikos database, including all systematic reviews addressing a similar question and all studies addressing the question in those reviews. We will try to explain why studies are not included in individual reviews using the inclusion criteria, and we will estimate the overlap between different reviews in the same matrix of evidence.

ETHICS AND DISSEMINATION

This review will be published in a peer-reviewed journal.

TRIAL REGISTRATION NUMBER

CRD42014013036.

Epidemiology and reporting characteristics of overviews of reviews of healthcare interventions published 2012-2016: protocol for a systematic review

BACKGROUND

Overviews of systematic reviews (overviews) attempt to systematically retrieve and summarize the results of multiple systematic reviews (SRs) for a given condition or public health problem. Two prior descriptive analyses of overviews found substantial variation in the methodological approaches used in overviews, and deficiencies in reporting of key methodological steps. Since then, new methods have been developed so it is timely to update the prior descriptive analyses. The objectives are to: (1) investigate the epidemiological, descriptive, and reporting characteristics of a random sample of 100 overviews published from 2012 to 2016 and (2) compare these recently published overviews (2012-2016) to those published prior to 2012 (based on the prior descriptive analyses).

METHODS

Medline, EMBASE, and CDSR will be searched for overviews published 2012-2016, using a validated search filter for overviews. Only overviews written in English will be included. All titles and abstracts will be screened by one review author; those deemed not relevant will be verified by a second person for exclusion. Full-texts will be assessed for inclusion by two reviewers independently. Of those deemed relevant, a random sample of 100 overviews will be selected for inclusion. Data extraction will be either performed by one reviewer with verification by a second reviewer or by one reviewer only depending on the complexity of the item. Discrepancies at any stage will be resolved by consensus or consulting a third person. Data will be extracted on the epidemiological, descriptive, and reporting characteristics of each overview. Data will be analyzed descriptively. When data are available for both time points (up to 2011 vs. 2012-2016), we will compare characteristics by calculating risk ratios or applying the Mann-Whitney test.

DISCUSSION

Overviews are becoming increasingly valuable evidence syntheses, and the number of published overviews is increasing. However, former analyses found limitations in the conduct and reporting of overviews. This update of a recent sample of overviews will inform whether this has changed, while also identifying areas for further improvement.

SYSTEMATIC REVIEW REGISTRATION

The review will not be registered in PROSPERO as it does not meet the eligibility criterion of dealing with health-related outcomes.

Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions

BACKGROUND

Overviews of reviews (overviews) compile information from multiple systematic reviews (SRs) to provide a single synthesis of relevant evidence for decision-making. It is recommended that authors assess and report the methodological quality of SRs in overviews-for example, using A MeaSurement Tool to Assess systematic Reviews (AMSTAR). Currently, there is variation in whether and how overview authors assess and report SR quality, and limited guidance is available. Our objectives were to: examine methodological considerations involved in using AMSTAR to assess the quality of Cochrane and non-Cochrane SRs in overviews of healthcare interventions; identify challenges (and develop potential decision rules) when using AMSTAR in overviews; and examine the potential impact of considering methodological quality when making inclusion decisions in overviews.

METHODS

We selected seven overviews of healthcare interventions and included all SRs meeting each overview's inclusion criteria. For each SR, two reviewers independently conducted AMSTAR assessments with consensus and discussed challenges encountered. We also examined the correlation between AMSTAR assessments and SR results/conclusions.

RESULTS

Ninety-five SRs were included (30 Cochrane, 65 non-Cochrane). Mean AMSTAR assessments (9.6/11 vs. 5.5/11; p < 0.001) and inter-rater reliability (AC1 statistic: 0.84 vs. 0.69; "almost perfect" vs. "substantial" using the Landis & Koch criteria) were higher for Cochrane compared to non-Cochrane SRs. Four challenges were identified when applying AMSTAR in overviews: the scope of the SRs and overviews often differed; SRs examining similar topics sometimes made different methodological decisions; reporting of non-Cochrane SRs was sometimes poor; and some non-Cochrane SRs included other SRs as well as primary studies. Decision rules were developed to address each challenge. We found no evidence that AMSTAR assessments were correlated with SR results/conclusions.

CONCLUSIONS

Results indicate that the AMSTAR tool can be used successfully in overviews that include Cochrane and non-Cochrane SRs, though decision rules may be useful to circumvent common challenges. Findings support existing recommendations that quality assessments of SRs in overviews be conducted independently, in duplicate, with a process for consensus. Results also suggest that using methodological quality to guide inclusion decisions (e.g., to exclude poorly conducted and reported SRs) may not introduce bias into the overview process.