Methodological approaches in conducting overviews: current state in HTA agencies

The reporting completeness and transparency of systematic reviews of prognostic prediction models for COVID-19 was poor: a methodological overview of systematic reviews

OBJECTIVES

To conduct a methodological overview of reviews to evaluate the reporting completeness and transparency of systematic reviews (SRs) of prognostic prediction models (PPMs) for COVID-19.

STUDY DESIGN AND SETTING

MEDLINE, Scopus, Cochrane Database of Systematic Reviews, and Epistemonikos (epistemonikos.org) were searched for SRs of PPMs for COVID-19 until December 31, 2022. The risk of bias in systematic reviews tool was used to assess the risk of bias. The protocol for this overview was uploaded in the Open Science Framework (https://osf.io/7y94c).

RESULTS

Ten SRs were retrieved; none of them synthesized the results in a meta-analysis. For most of the studies, there was absence of a predefined protocol and missing information on study selection, data collection process, and reporting of primary studies and models included, while only one SR had its data publicly available. In addition, for the majority of the SRs, the overall risk of bias was judged as being high. The overall corrected covered area was 6.3% showing a small amount of overlapping among the SRs.

CONCLUSION

The reporting completeness and transparency of SRs of PPMs for COVID-19 was poor. Guidance is urgently required, with increased awareness and education of minimum reporting standards and quality criteria. Specific focus is needed in predefined protocol, information on study selection and data collection process, and in the reporting of findings to improve the quality of SRs of PPMs for COVID-19.

Adaptive Health Technology Assessment: A Scoping Review of Methods

OBJECTIVES

Health technology assessment (HTA) is an established mechanism for explicit priority setting to support universal health coverage. However, full HTA requires significant time, data, and capacity for each intervention, which limits the number of decisions it can inform. Another approach systematically adapts full HTA methods by leveraging HTA evidence from other settings. We call this "adaptive" HTA (aHTA), although in settings where time is the main constraint, it is also called "rapid HTA."

METHODS

The objectives of this scoping review were to identify and map existing aHTA methods, and to assess their triggers, strengths, and weaknesses. This was done by searching HTA agencies' and networks' websites, and the published literature. Findings have been narratively synthesized.

RESULTS

This review identified 20 countries and 1 HTA network with aHTA methods in the Americas, Europe, Africa, and South-East Asia. These methods have been characterized into 5 types: rapid reviews, rapid cost-effectiveness analyses, rapid manufacturer submissions, transfers, and de facto HTA. Three characteristics "trigger" the use of aHTA instead of full HTA: urgency, certainty, and low budget impact. Sometimes, an iterative approach to selecting methods guides whether to do aHTA or full HTA. aHTA was found to be faster and more efficient, useful for decision makers, and to reduce duplication. Nevertheless, there is limited standardization, transparency, and measurement of uncertainty.

CONCLUSIONS

aHTA is used in many settings. It has potential to improve the efficiency of any priority-setting system, but needs to be better formalized to improve uptake, particularly for nascent HTA systems.

A methodological review finds mismatch between overall and pairwise overlap analysis in a sample of overviews

OBJECTIVES

Overlap of primary studies is a key methodological challenge for overviews. There are limited reports of methods used to address overlap, and there is no detailed assessment of the corrected covered area (CCA) of a representative sample of overviews. To describe the approaches used to address overlap, and to estimate the overall and pairwise CCA.

METHODS

We searched PubMed for overviews published in 2018. Two authors conducted the screening process. We described the strategy used for assessing overlap, and calculated overall and pairwise CCA for each overview.

RESULTS

We analyzed a random sample of 30 out of 89 eligible articles. Eleven did not address the overlap. Of the remainder, most frequent strategies were visual assessment and discussion of overlap as a limitation. Median overall CCA among the included overviews was 6.7%. The pairwise analysis showed that 52.8% of SR pairs had slight overlap, while 28.3% had very high overlap.

CONCLUSION

Reported strategies for addressing overlap vary considerably among overview authors. The pairwise approach for assessing the CCA revealed highly overlapped pairs of SRs in overviews with overall slight overlap and vice versa. We encourage authors to complement the overall CCA assessment with a pairwise approach.

Overviews of reviews in the cardiovascular field underreported critical methodological and transparency characteristics: a methodological study based on the Preferred Reporting Items for Overviews of Reviews (PRIOR) statement

OBJECTIVES

This study aimed to evaluate the epidemiology, reporting characteristics, and adherence to the Preferred Reporting Items for Overviews of Reviews (PRIOR) statement of overviews of reviews (overviews) of interventions in the cardiovascular field.

STUDY DESIGN AND SETTING

MEDLINE, Scopus, and the Cochrane Database of Systematic Reviews were searched from January 1, 2000, to October 15, 2020. An updated search was performed in MEDLINE, Epistemonikos, and Google Scholar up to August 25, 2022. Overviews of interventions published in English and primarily considering populations, interventions, and outcomes pertinent to the cardiovascular field were eligible. Study selection, data extraction, and PRIOR adherence assessment were performed by two authors independently.

RESULTS

We analyzed 96 overviews. Almost half (43/96 [45%]) were published between 2020 and 2022 and contained a median of 15 systematic reviews (SRs) (interquartile range, 9-28). The commonest title terminology was "overview of (systematic) reviews" (38/96 [40%]). Methods for handling SR overlap were reported in 24/96 (25%), methods for assessing primary study overlap in 18/96 (19%), handling of discrepant data in 11/96 (11%), and methods for methodological quality or risk of bias assessment of the primary studies within SRs in 23/96 (24%). Authors included data sharing statements in 28/96 (29%), complete funding disclosure in 43/96 (45%), protocol registration in 43/96 (45%), and conflict of interest statement in 82/96 (85%) overviews.

CONCLUSION

Insufficient reporting was identified in methodological characteristics unique in overviews' conduct and most transparency markers. Adoption of PRIOR from the research community could ameliorate overviews' reporting.

Interventions for ADHD in childhood and adolescence: A systematic umbrella review and meta-meta-analysis

There are several meta-analyses of treatment effects for children and adolescents with attention deficit hyperactivity disorder (ADHD). The conclusions of these meta-analyses vary considerably. Our aim was to synthesize the latest evidence of the effectiveness of psychological, pharmacological treatment options and their combination in a systematic overview and meta-meta-analyses. A systematic literature search until July 2022 to identify meta-analyses investigating effects of treatments for children and adolescents with ADHD and ADHD symptom severity as primary outcome (parent and teacher rated) yielded 16 meta-analyses for quantitative analyses. Meta-meta-analyses of pre-post data showed significant effects for pharmacological treatment options for parent (SMD = 0.67, 95% CI 0.60 to 0.74) and teacher ADHD symptom ratings (SMD = 0.68, 95% CI 0.54 to 0.82) as well as for psychological interventions for parent (SMD = 0.42, 95% CI 0.33 to 0.51) and teacher rated symptoms (SMD = 0.25, 95% CI 0.12 to 0.38). We were unable to calculate effect sizes for combined treatments due to the lack of meta-analyses. Our analyses revealed that there is a lack of research on combined treatments and for therapy options for adolescents. Finally, future research efforts should adhere to scientific standards as this allows comparison of effects across meta-analyses.

Guidance on review type selection for health technology assessments: key factors and considerations for deciding when to conduct a de novo systematic review, an update of a systematic review, or an overview of systematic reviews

BACKGROUND

A systematic review (SR) helps us make sense of a body of research while minimizing bias and is routinely conducted to evaluate intervention effects in a health technology assessment (HTA). In addition to the traditional de novo SR, which combines the results of multiple primary studies, there are alternative review types that use systematic methods and leverage existing SRs, namely updates of SRs and overviews of SRs. This paper shares guidance that can be used to select the most appropriate review type to conduct when evaluating intervention effects in an HTA, with a goal to leverage existing SRs and reduce research waste where possible.

PROCESS

We identified key factors and considerations that can inform the process of deciding to conduct one review type over the others to answer a research question and organized them into guidance comprising a summary and a corresponding flowchart. This work consisted of three steps. First, a guidance document was drafted by methodologists from two Canadian HTA agencies based on their experience. Next, the draft guidance was supplemented with a literature review. Lastly, broader feedback from HTA researchers across Canada was sought and incorporated into the final guidance.

INSIGHTS

Nine key factors and six considerations were identified to help reviewers select the most appropriate review type to conduct. These fell into one of two categories: the evidentiary needs of the planned review (i.e., to understand the scope, objective, and analytic approach required for the review) and the state of the existing literature (i.e., to know the available literature in terms of its relevance, quality, comprehensiveness, currency, and findings). The accompanying flowchart, which can be used as a decision tool, demonstrates the interdependency between many of the key factors and considerations and aims to balance the potential benefits and challenges of leveraging existing SRs instead of primary study reports.

CONCLUSIONS

Selecting the most appropriate review type to conduct when evaluating intervention effects in an HTA requires a myriad of factors to be considered. We hope this guidance adds clarity to the many competing considerations when deciding which review type to conduct and facilitates that decision-making process.

Umbrella reviews: A new tool to synthesize scientific evidence in surgery

Researchers and practitioners are faced with an exponential increase in the number of systematic reviews (SRs) (with or without meta-analysis), a so-called `secondary' research method that synthesizes data from primary research. This growing number, sometimes with discordant results on the same subject or with non-conclusions, has led to the introduction of the concept of reviews to synthesize SR in order to combine scientific knowledge useful to practitioners. These so-called ``umbrella reviews'' (UR) constitute a new tertiary research tool. Surgical research is no exception to this development but umbrella surgical reviews remain relatively rare. Any UR must be transparent and meet rigorous methodological criteria. The UR could thus provide answers to practical questions in the field of surgery, but only on condition that the bias of the included SRs is limited. Let us not forget that the base requirement of clinical surgical research remains the good methodological quality of clinical studies (primary research). Only thus can SRs or URs (secondary or tertiary research) be more useful and decisive.

Global mapping of overviews of systematic reviews in healthcare published between 2000 and 2020: a bibliometric analysis

OBJECTIVE

To conduct a bibliometric analysis using a large sample of overviews of systematic reviews (OoSRs) and reveal research trends and areas of interest about these studies.

STUDY DESIGN AND SETTING

We searched MEDLINE, Scopus and Cochrane Database of Systematic Reviews from 1/1/2000 to 15/10/2020. We used Scopus meta-data and two authors recorded supplementary information independently. We summarized the data using frequencies with percentages.

RESULTS

A total of 1558 studies were considered eligible for analysis. We found that the publications have been increasing yearly and their nomenclature was not uniform (the most frequent label in the title was "overview of systematic reviews"). The largest number of papers and the most cited ones were published by corresponding authors from the UK. The publications were distributed across 737 scholarly journals and many of them were published in the field of complementary/alternative medicine, psychiatry/psychology, nutrition/dietetics, and pediatrics. The co-authorship analysis revealed collaborations among countries. The most common clinical conditions were depression, diabetes, cancer, dementia, pain, cardiovascular disease, stroke, obesity, and schizophrenia.

CONCLUSION

OoSRs have recently become a popular approach of evidence synthesis. International collaborations between overview authors from countries with increased research productivity and countries with less research activity should be encouraged.

Pornography use and sexting amongst children and young people: a systematic overview of reviews

BACKGROUND

Young people's use of pornography and participation in sexting are commonly viewed as harmful behaviours. This paper reports findings from a 'review of reviews', which aimed to systematically identify and synthesise the evidence on pornography and sexting amongst young people. Here, we focus specifically on the evidence relating to young people's use of pornography; involvement in sexting; and their beliefs, attitudes, behaviours and wellbeing to better understand potential harms and benefits, and identify where future research is required.

METHODS

We searched five health and social science databases; searches for grey literature were also performed. Review quality was assessed and findings synthesised narratively.

RESULTS

Eleven reviews of quantitative and/or qualitative studies were included. A relationship was identified between pornography use and more permissive sexual attitudes. An association between pornography use and stronger gender-stereotypical sexual beliefs was also reported, but not consistently. Similarly, inconsistent evidence of an association between pornography use and sexting and sexual behaviour was identified. Pornography use has been associated with various forms of sexual violence, aggression and harassment, but the relationship appears complex. Girls, in particular, may experience coercion and pressure to engage in sexting and suffer more negative consequences than boys if sexts become public. Positive aspects to sexting were reported, particularly in relation to young people's personal relationships.

CONCLUSIONS

We identified evidence from reviews of varying quality that linked pornography use and sexting amongst young people to specific beliefs, attitudes and behaviours. However, evidence was often inconsistent and mostly derived from observational studies using a cross-sectional design, which precludes establishing any causal relationship. Other methodological limitations and evidence gaps were identified. More rigorous quantitative studies and greater use of qualitative methods are required.

Effectiveness and Safety of Herbal Medicine for Atopic Dermatitis: An Overview of Systematic Reviews

Objectives

Herbal medicine (HM) is attracting attention for treating atopic dermatitis (AD). This overview was conducted to summarize and critically evaluate the current systematic reviews (SRs) on HM for the treatment of AD.

Methods

Through comprehensive searches, all relevant SRs on HM for AD published until May 2020 were included. The quality of included SRs was assessed using the AMSTAR-2 tool. Moreover, original randomized controlled trials (RCTs) included in the SRs were resynthesized to investigate the efficacy and safety of oral HM for AD. The quality of evidence for the main findings was evaluated using the GRADE approach.

Results

Nine SRs were included in this overview. HM showed significantly better efficacy in terms of total effective rate (TER), itching and sleep symptom scores, quality of life, and the dose of topical treatment used compared with placebo. HM as a monotherapy and/or an adjunctive therapy to conventional medication (CM) showed significantly better results on the efficacy, symptom relief, and some laboratory parameters related to the inflammatory response. The methodological quality was generally low. When 58 original RCTs were reanalyzed, HM showed significantly lower SCORing Atopic Dermatitis (SCORAD) score and higher TER than the placebo or CM. In terms of the safety profile, HM was not significantly different from the placebo and was better than CM. The quality of evidence ranged from “moderate” to “very low.”

Conclusion

The results suggested that HM as a monotherapy or an adjunctive therapy is promising for the treatment of AD. However, due to low methodological quality and low quality of evidence, further rigorous, well-designed, high-quality SRs, and RCTs are needed to make clinical recommendations on HM use.

Improving the usefulness of evidence concerning the effectiveness of implementation strategies for knowledge products in primary healthcare: protocol for a series of systematic reviews

BACKGROUND

The literature on the implementation of knowledge products is extensive. However, this literature is still difficult to interpret for policymakers and other stakeholders when faced with choosing implementation strategies likely to bring about successful change in their health systems. This work has the particularity to examine the scope of this literature, and to clarify the effectiveness of implementation strategies for different knowledge products. Consequently, we aim to (1) determine the strengths and weaknesses of existing literature overviews; (2) produce a detailed portrait of the literature on implementation strategies for various knowledge products; and (3) assess the effectiveness of implementation strategies for each knowledge product identified and classify them.

METHODS

We will use a three-phase approach consisting of a critical analysis of existing literature overviews, a systematic review of systematic reviews, and a series of systematic reviews and meta-analyses. We will follow the Cochrane Methodology for each of the three phases. Our eligibility criteria are defined following a PICOS approach: Population, individuals or stakeholders participating in healthcare delivery, specifically, healthcare providers, caregivers, and end users; Intervention, any type of strategy aiming to implement a knowledge product including, but not limited to, a decision support tool, a clinical practice guideline, a policy brief, or a decision-making tool, a one-pager, or a health intervention; Comparison, any comparator will be considered; Outcomes, phases 1 and 2-any outcome related to implementation strategies including, but not limited to, the measures of adherence/fidelity to the use of knowledge products, their acceptability, adoption, appropriateness, feasibility, adaptability, implementation costs, penetration/reach and sustainability; phase 3-any additional outcome related to patients (psychosocial, health behavioral, and clinical outcomes) or healthcare professionals (behavioral and performance outcomes); Setting, primary healthcare has to be covered. We will search MEDLINE (Ovid), EMBASE, Web of Science, PsycINFO, CINAHL, and the Cochrane Library from their inception onwards. For each phase, two reviewers will independently perform the selection of studies, data extraction, and assess their methodological quality. We will analyze extracted data, and perform narrative syntheses, and meta-analyses when possible.

DISCUSSION

Our results could inform not only the overviews' methodology but also the development of an online platform for the implementation strategies of knowledge products. This platform could be useful for stakeholders in implementation science.

SYSTEMATIC REVIEW REGISTRATION

Protocol registered on Open Science Framework, https://osf.io/eb8w2/.

Silver diamine fluoride for managing carious lesions: an umbrella review

Background

This umbrella review comprehensively appraised evidence for silver diamine fluoride (SDF) to arrest and prevent root and coronal caries by summarizing systematic reviews. Adverse events were explored.

Methods

Following Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guidelines, PubMed, Embase, Cochrane Library, PROSPERO register and Joanna Briggs Institute Database of Systematic Reviews were searched for systematic reviews investigating SDF for caries prevention or arrest (1970–2018) without language restrictions. Systematic reviews were selected, data extracted, and risk of bias assessed using ROBIS by two independent reviewers, in duplicate. Corrected covered area was calculated to quantify studies’ overlap across reviews.

Results

Eleven systematic reviews were included; four focussing on SDF for root caries in adults and seven on coronal caries in children. These cited 30 studies (4 root caries; 26 coronal caries) appearing 63 times. Five systematic reviews were of “low”, one “unclear” and five “high” risk of bias. Overlap of studies was very high (50% root caries; 17% coronal caries). High overlap and heterogeneity, mainly comparators and outcome measures, precluded meta-analysis. Results were grouped by aim and outcomes to present an overview of direction and magnitude of effect. SDF had a positive effect on prevention and arrest of coronal and root caries, consistently outperforming comparators (fluoride varnish, Atraumatic Restorative Treatment, placebo). For root caries prevention, the prevented fraction (PF) was 25–71% higher for SDF compared to placebo (two systematic reviews with three studies) and PF = 100–725% for root caries arrest (one systematic review with two studies). For coronal caries prevention, PF = 70–78% (two systematic reviews with two studies) and PF = 55–96% for coronal caries arrest (one systematic review with two studies) with arrest rates of 65–91% (four systematic reviews with six studies). Eight systematic reviews reported adverse events, seven of which reported arrested lesions black staining.

Conclusion

Systematic reviews consistently supported SDF’s effectiveness for arresting coronal caries in the primary dentition and arresting and preventing root caries in older adults for all comparators. There is insufficient evidence to draw conclusions on SDF for prevention in primary teeth and prevention and arrest in permanent teeth in children. No serious adverse events were reported.

Electronic supplementary material

The online version of this article (10.1186/s12903-019-0830-5) contains supplementary material, which is available to authorized users.

Cochrane overviews: how can we optimize their impact on evidence-based rehabilitation?

INTRODUCTION

Overviews (i.e. reviews of multiple systematic reviews) comprise a relatively novel methodology to systematically synthesize research findings. Overviews aim for a beneficial impact on clinical practice, but their methods and pathways to impact have so far not been mapped. The aim of this paper was to inform recommendations for optimizing impact on rehabilitation practice and research by mapping methods and pathways to impact in Cochrane overviews relevant to rehabilitation.

EVIDENCE ACQUISITION

We systematically searched and identified published Cochrane overviews (to June 2018) relevant to rehabilitation. We extracted data and compared overviews on key characteristics, methods of evidence synthesis, statements about impact, and access metrics. We explored one overview in detail regarding beneficiaries, activities and outputs, mapped potential pathways to impact, and, using an iterative process, refined this into a generic map. Through exploration of all synthesized data, we propose further recommendations for planning, conducting and reporting of future overviews in order to optimize impact on rehabilitation.

EVIDENCE SYNTHESIS

We identified seven Cochrane overviews relevant to rehabilitation. Their focus and methods varied, but they were broadly related to rehabilitation interventions for populations of people with diverse long-term conditions. Overviews also varied regarding their intended impact; only 4 overviews identified specific beneficiaries. All overviews included multiple tables and figures, but only one synthesized key findings into a single figure. For five overviews, the Altmetric Attention Score (a weighted count of attention that an output receives based on a range of online sources) was in the top 5% of all research outputs scored by Altmetric. The overview within our worked example had four key impact goals, each with different beneficiaries and required actions; this example led to a generic map of potential pathways to impact for other overviews.

CONCLUSIONS

Cochrane overviews have the potential to play a key role in knowledge translation and therefore to be useful in supporting evidence-based rehabilitation practice. However, current overviews relating to rehabilitation differ in methods, approaches and intended impact, and sometimes fall short of promoting easy access to key information for beneficiaries. Future Cochrane overviews should address topics of importance to key beneficiaries and clearly outline potential pathways to impact in order to have a potential beneficial impact on evidence-based rehabilitation and to improve rehabilitation outcomes.

On estimands and the analysis of adverse events in the presence of varying follow-up times within the benefit assessment of therapies

The analysis of adverse events (AEs) is a key component in the assessment of a drug's safety profile. Inappropriate analysis methods may result in misleading conclusions about a therapy's safety and consequently its benefit-risk ratio. The statistical analysis of AEs is complicated by the fact that the follow-up times can vary between the patients included in a clinical trial. This paper takes as its focus the analysis of AE data in the presence of varying follow-up times within the benefit assessment of therapeutic interventions. Instead of approaching this issue directly and solely from an analysis point of view, we first discuss what should be estimated in the context of safety data, leading to the concept of estimands. Although the current discussion on estimands is mainly related to efficacy evaluation, the concept is applicable to safety endpoints as well. Within the framework of estimands, we present statistical methods for analysing AEs with the focus being on the time to the occurrence of the first AE of a specific type. We give recommendations which estimators should be used for the estimands described. Furthermore, we state practical implications of the analysis of AEs in clinical trials and give an overview of examples across different indications. We also provide a review of current practices of health technology assessment (HTA) agencies with respect to the evaluation of safety data. Finally, we describe problems with meta-analyses of AE data and sketch possible solutions.

Toward a comprehensive evidence map of overview of systematic review methods: paper 2-risk of bias assessment; synthesis, presentation and summary of the findings; and assessment of the certainty of the evidence

BACKGROUND

Overviews of systematic reviews (SRs) attempt to systematically retrieve and summarise the results of multiple systematic reviews. This is the second of two papers from a study aiming to develop a comprehensive evidence map of the methods used in overviews. Our objectives were to (a) develop a framework of methods for conducting, interpreting and reporting overviews (stage I)-the Methods for Overviews of Reviews (MOoR) framework-and (b) to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II). In the first paper, we reported findings for the four initial steps of an overview (specification of purpose, objectives and scope; eligibility criteria; search methods; data extraction). In this paper, we report the remaining steps: assessing risk of bias; synthesis, presentation and summary of the findings; and assessing certainty of the evidence arising from the overview.

METHODS

In stage I, we identified cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews. Based on these studies, we developed a framework of possible methods for overviews, categorised by the steps in conducting an overview. Multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified studies evaluating methods and mapped these evaluations to the framework.

RESULTS

Forty-two stage I studies described methods relevant to one or more of the latter steps of an overview. Six studies evaluating methods were included in stage II. These mapped to steps involving (i) the assessment of risk of bias (RoB) in SRs (two SRs and three primary studies, all reporting evaluation of RoB tools) and (ii) the synthesis, presentation and summary of the findings (one primary study evaluating methods for measuring overlap).

CONCLUSION

Many methods have been described for use in the latter steps in conducting an overview; however, evaluation and guidance for applying these methods is sparse. The exception is RoB assessment, for which a multitude of tools exist-several with sufficient evaluation and guidance to recommend their use. Evaluation of other methods is required to provide a comprehensive evidence map.

An introduction to overviews of reviews: planning a relevant research question and objective for an overview

BACKGROUND

Overviews of systematic reviews are a relatively new approach to synthesising evidence, and research methods and associated guidance are developing. Within this paper we aim to help readers understand key issues which are essential to consider when taking the first steps in planning an overview. These issues relate to the development of clear, relevant research questions and objectives prior to the development of an overview protocol.

METHODS

Initial discussions and key concepts for this paper were formed during a workshop on overview methods at the 2016 UK Cochrane Symposium, at which all members of this author group presented work and contributed to wider discussions. Detailed descriptions of the various key features of overviews and their different objectives were created by the author group based upon current evidence (Higgins J, Green S. Cochrane Handbook Syst Rev Interv. 2011;4:5, Pollock M, et al. Sys Rev. 2016;5:190-205, Pollock A, et al. Cochrane overviews of reviews: exploring the methods and challenges. UK and Ireland: Cochrane Symposium; 2016, Pieper D, et al. Res Syn Meth. 2014;5:187-99, Lunny C, et al. Sys Rev. 2016;5:4-12, Hartling L, et al. Comparing multiple treatments: an introduction to overviews of reviews. In 23rd Cochrane Colloquium; 2015, Hartling L, et al. Plos One. 2012;7:1-8, Ballard M, Montgomery P. Res Syn Meth. 2017;8:92-108) and author experiences conducting overviews.

RESULTS

Within this paper we introduce different types of overviews and suggest common research questions addressed by these overviews. We briefly reflect on the key features and objectives of the example overviews discussed.

CONCLUSIONS

Clear decisions relating to the research questions and objectives are a fundamental first step during the initial planning stages for an overview. Key stakeholders should be involved at the earliest opportunity to ensure that the planned overview is relevant and meaningful to the potential end users of the overview. Following best practice in common with other forms of systematic evidence synthesis, an overview protocol should be published, ensuring transparency and reducing opportunities for introduction of bias in the conduct of the overview.

Toward a comprehensive evidence map of overview of systematic review methods: paper 1-purpose, eligibility, search and data extraction

BACKGROUND

Overviews of systematic reviews attempt to systematically retrieve and summarise the results of multiple systematic reviews. Methods for conducting, interpreting and reporting overviews are in their infancy. To date, there has been no evidence map of the methods used in overviews, thus making it difficult to determine the gaps and priorities for methods research. Our objectives were to develop and populate a comprehensive framework of methods for conducting, interpreting and reporting overviews (stage I) and to create an evidence map by mapping studies that have evaluated overview methods to the framework (stage II).

METHODS

We searched methods collections (e.g. Cochrane Methodology Register, Meth4ReSyn library, AHRQ Effective Health Care Program) to identify eligible studies for both stages of this research. In stage I, cross-sectional studies, guidance documents and commentaries that described methods proposed for, or used in, overviews were used to develop and populate the framework of methods. Drafts and multiple iterations of the framework were discussed and refined by all authors. In stage II, we identified and described studies evaluating overview methods and mapped these evaluations to the framework.

RESULTS

In this paper, we present results for the four initial steps of conducting an overview: (a) specification of the purpose, objectives and scope, (b) specification of the eligibility criteria, (c) search methods and (d) data extraction. Twenty-nine studies mentioned or described methods relevant to one or more of these steps. In the developed framework, identified methods and approaches were grouped according to the steps an overview author would need to undertake. Fifteen studies evaluated identified methods, all of which mapped to the search methods step. These studies either reported the development and evaluation of a new search filter to retrieve systematic reviews or compared the performance of multiple filters.

CONCLUSION

Gaps in the evaluation of methods were found for the majority of steps in the framework. More empirical studies are needed to evaluate the methods outlined and provide a comprehensive evidence map. The framework is useful for planning these evaluations and for planning methods required to deal with challenges that arise when conducting an overview.

Preferred reporting items for overviews of systematic reviews including harms checklist: a pilot tool to be used for balanced reporting of benefits and harms

OBJECTIVES

An overview of systematic reviews (OoSRs) is a study designed to synthesize multiple evidence from existing systematic reviews on a specific domain. The aim of this paper was to offer a pilot version checklist with Preferred Reporting Items for OoSRs (PRIO-harms) to promote a more balanced reporting of benefits and harms in OoSRs of health care interventions.

STUDY DESIGN AND SETTING

The included items were developed by combining key features from health care OoSRs designs with recommendations from statements of other relevant checklists and pertinent methodological review articles. Two raters independently used the PRIO-harms checklist to assess a sample of 20 OoSRs.

RESULTS

The PRIO-harms tool consists of a 27-item (56 (sub-)items in total) checklist and is accompanied by a five-stage process flow diagram (identification, screening, eligibility, inclusion, and separation of relevant studies). The mean interrater reliability (Gwet's AC1 statistic) between reviewers was 0.90 (95% confidence interval: 0.88, 0.92) indicating a very good agreement.

CONCLUSION

The PRIO-harms tool can be used in every OoSRs that addresses health care interventions. This instrument will assist overview authors to improve completeness and transparency of research reporting with emphasis on harms. However, it might benefit from critical review and further validation from experts and research teams that produce OoSRs.

Overviews of systematic reviews: great promise, greater challenge

The proliferation of systematic reviews and escalating demand from policy makers has driven a newer form of evidence synthesis-overviews of systematic reviews. Systematic Reviews are publishing a special thematic series on overviews and are encouraging submissions on the development and evaluation of methods for this review type. The authors' of this editorial introduce the series by considering challenges that arise when conducting an overview and what methods guidance is available. They emphasise the importance of evaluating overview methods to understand the trade-offs of using different approaches and propose that a more systematic and coordinated approach to methods development would be beneficial. Finally, they consider the potential for overviews to drive improvements in the conduct and reporting of systematic reviews.

Selecting and implementing overview methods: implications from five exemplar overviews

BACKGROUND

Overviews of systematic reviews are an increasingly popular method of evidence synthesis; there is a lack of clear guidance for completing overviews and a number of methodological challenges. At the UK Cochrane Symposium 2016, methodological challenges of five overviews were explored. Using data from these five overviews, practical implications to support methodological decision making of authors writing protocols for future overviews are proposed.

METHODS

Methods, and their justification, from the five exemplar overviews were tabulated and compared with areas of debate identified within current literature. Key methodological challenges and implications for development of overview protocols were generated and synthesised into a list, discussed and refined until there was consensus.

RESULTS

Methodological features of three Cochrane overviews, one overview of diagnostic test accuracy and one mixed methods overview have been summarised. Methods of selection of reviews and data extraction were similar. Either the AMSTAR or ROBIS tool was used to assess quality of included reviews. The GRADE approach was most commonly used to assess quality of evidence within the reviews. Eight key methodological challenges were identified from the exemplar overviews. There was good agreement between our findings and emerging areas of debate within a recent published synthesis. Implications for development of protocols for future overviews were identified.

CONCLUSIONS

Overviews are a relatively new methodological innovation, and there are currently substantial variations in the methodological approaches used within different overviews. There are considerable methodological challenges for which optimal solutions are not necessarily yet known. Lessons learnt from five exemplar overviews highlight a number of methodological decisions which may be beneficial to consider during the development of an overview protocol.

The Healthcare Improvement Scotland evidence note rapid review process: providing timely, reliable evidence to inform imperative decisions on healthcare

AIM

Rapid review has become widely adopted by health technology assessment agencies in response to demand for evidence-based information to support imperative decisions. Concern about the credibility of rapid reviews and the reliability of their findings has prompted a call for wider publication of their methods. In publishing this overview of the accredited rapid review process developed by Healthcare Improvement Scotland, we aim to raise awareness of our methods and advance the discourse on best practice.

METHODS

Healthcare Improvement Scotland produces rapid reviews called evidence notes using a process that has achieved external accreditation through the National Institute for Health and Care Excellence. Key components include a structured approach to topic selection, initial scoping, considered stakeholder involvement, streamlined systematic review, internal quality assurance, external peer review and updating.

RESULTS

The process was introduced in 2010 and continues to be refined over time in response to user feedback and operational experience. Decision-makers value the responsiveness of the process and perceive it as being a credible source of unbiased evidence-based information supporting advice for NHSScotland.

CONCLUSION

Many agencies undertaking rapid reviews are striving to balance efficiency with methodological rigour. We agree that there is a need for methodological guidance and that it should be informed by better understanding of current approaches and the consequences of different approaches to streamlining systematic review methods. Greater transparency in the reporting of rapid review methods is essential to enable that to happen.

Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist

OBJECTIVE

To assess the conditions under which employing an overview of systematic reviews is likely to lead to a high risk of bias.

STUDY DESIGN

To synthesise existing guidance concerning overview practice, a scoping review was conducted. Four electronic databases were searched with a pre-specified strategy (PROSPERO 2015:CRD42015027592) ending October 2015. Included studies needed to describe or develop overview methodology. Data were narratively synthesised to delineate areas highlighted as outstanding challenges or where methodological recommendations conflict.

RESULTS

Twenty-four papers met the inclusion criteria. There is emerging debate regarding overlapping systematic reviews; systematic review scope; quality of included research; updating; and synthesizing and reporting results. While three functions for overviews have been proposed-identify gaps, explore heterogeneity, summarize evidence-overviews cannot perform the first; are unlikely to achieve the second and third simultaneously; and can only perform the third under specific circumstances. Namely, when identified systematic reviews meet the following four conditions: (1) include primary trials that do not substantially overlap, (2) match overview scope, (3) are of high methodological quality, and (4) are up-to-date.

CONCLUSION

Considering the intended function of proposed overviews with the corresponding methodological conditions may improve the quality of this burgeoning publication type. Copyright © 2017 John Wiley & Sons, Ltd.

Discharge interventions for older patients leaving hospital: protocol for a systematic meta-review

BACKGROUND

There is an increased need for additional care and support services for the elderly population. It is important to identify what support older people need once they are discharged from hospital and to ensure continuity of care. There is a large evidence base focusing on enhanced discharge services and their impact on patients. The services show some potential benefits, but there are inconsistent findings across reviews. Furthermore, it is unclear what elements of enhanced discharge interventions could be most beneficial to older people. This meta-review aims to identify existing systematic reviews of discharge interventions for older people, identify potentially effective elements of enhanced discharge services for this patient group and identify areas where further work may still be needed.

METHODS/DESIGN

The search will aim to identify English language systematic reviews that have assessed the effectiveness of discharge interventions for older people. The following databases will be searched: Medline, Embase, PsycINFO, HMIC, Social Policy and Practice, CINAHL, the Cochrane Library, ASSIA, Social Science Citation Index and the Grey Literature Report. The search strategy will comprise the keywords 'systematic reviews', 'older people' and 'discharge'. Discharge interventions must aim to support older patients before, during and/or after discharge from hospital. Outcomes of interest will include mortality, readmissions, length of hospital stay, patient health status, patient and carer satisfaction and staff views. Abstract, title and full text screening will be conducted independently by two reviewers. Data extracted from reviews will include review characteristics, patient population, review quality score, outcome measures and review findings, and a narrative synthesis will be conducted.

DISCUSSION

This review will identify existing reviews of discharge interventions and appraise how these interventions can impact outcomes in older people such as readmissions, health status, length of hospital stay and mortality. The review could inform practice and will help identify where further research is needed.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42015025737.