Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis

Identification of application and interpretation errors that can occur in pairwise meta-analyses in systematic reviews of interventions: a systematic review

OBJECTIVES

To generate a bank of items describing application and interpretation errors that can arise in pairwise meta-analyses in systematic reviews of interventions.

STUDY DESIGN AND SETTING

MEDLINE, Embase, and Scopus were searched to identify studies describing types of errors in meta-analyses. Descriptions of errors and supporting quotes were extracted by multiple authors. Errors were reviewed at team meetings to determine if they should be excluded, reworded, or combined with other errors, and were categorized into broad categories of errors and subcategories within.

RESULTS

Fifty articles met our inclusion criteria, leading to the identification of 139 errors. We identified 25 errors covering data extraction/manipulation, 74 covering statistical analyses, and 40 covering interpretation. Many of the statistical analysis errors related to the meta-analysis model (eg, using a two-stage strategy to determine whether to select a fixed or random-effects model) and statistical heterogeneity (eg, not undertaking an assessment for statistical heterogeneity).

CONCLUSION

We generated a comprehensive bank of possible errors that can arise in the application and interpretation of meta-analyses in systematic reviews of interventions. This item bank of errors provides the foundation for developing a checklist to help peer reviewers detect statistical errors.

Neutrophil-Enriched Biomarkers and Long-Term Prognosis in Acute Coronary Syndrome: a Systematic Review and Meta-analysis

Activated neutrophils release a range of inflammatory products that represent potential biomarkers, and there is interest in the prognostic value of these in acute coronary syndrome (ACS) patients. We conducted a systematic review to examine neutrophil-enriched biomarkers and the occurrence of major adverse cardiovascular events (MACE) in patients with ACS. We identified twenty-seven studies including 17,831 patients with ACS. The most studied biomarkers were neutrophil gelatinase-associated lipocalin (NGAL) and myeloperoxidase (MPO). Meta-analyses showed that elevated NGAL was associated with higher MACE rates (unadjusted risk ratio (RR) 1.52, 95% CI 1.12-2.06, p = 0.006) as were elevated MPO levels (unadjusted RR 1.61, 95% CI 1.22-2.13, p = 0.01). There was limited data suggesting that increased levels of calprotectin, proteinase-3 and double-stranded DNA were also associated with MACE. These results suggest that higher levels of neutrophil-enriched biomarkers may be predictive of MACE in patients with ACS, although higher-quality studies are needed to confirm these observations.

Comparison of statistical methods used to meta-analyse results from interrupted time series studies: an empirical study

BACKGROUND

The Interrupted Time Series (ITS) is a robust design for evaluating public health and policy interventions or exposures when randomisation may be infeasible. Several statistical methods are available for the analysis and meta-analysis of ITS studies. We sought to empirically compare available methods when applied to real-world ITS data.

METHODS

We sourced ITS data from published meta-analyses to create an online data repository. Each dataset was re-analysed using two ITS estimation methods. The level- and slope-change effect estimates (and standard errors) were calculated and combined using fixed-effect and four random-effects meta-analysis methods. We examined differences in meta-analytic level- and slope-change estimates, their 95% confidence intervals, p-values, and estimates of heterogeneity across the statistical methods.

RESULTS

Of 40 eligible meta-analyses, data from 17 meta-analyses including 282 ITS studies were obtained (predominantly investigating the effects of public health interruptions (88%)) and analysed. We found that on average, the meta-analytic effect estimates, their standard errors and between-study variances were not sensitive to meta-analysis method choice, irrespective of the ITS analysis method. However, across ITS analysis methods, for any given meta-analysis, there could be small to moderate differences in meta-analytic effect estimates, and important differences in the meta-analytic standard errors. Furthermore, the confidence interval widths and p-values for the meta-analytic effect estimates varied depending on the choice of confidence interval method and ITS analysis method.

CONCLUSIONS

Our empirical study showed that meta-analysis effect estimates, their standard errors, confidence interval widths and p-values can be affected by statistical method choice. These differences may importantly impact interpretations and conclusions of a meta-analysis and suggest that the statistical methods are not interchangeable in practice.

Statistical heterogeneity in meta-analysis of hypertension and exercise training: A meta-review

We aim to evaluate the prevalence of reporting and the extent of statistical heterogeneity of systematic reviews with meta-analyses (SRMAs) of exercise training for hypertension and to provide practical recommendations for dealing with statistical heterogeneity. We systematically searched on four databases (from 2002 to September of 2023) for SRMAs comparing exercise interventions vs. a non-exercise control on blood pressure (BP) control in adults with hypertension. Fifty-nine SRMAs, with a median of 14 studies, were analysed. Cochran's Q (41%), I2 (24%), forest plots (44%), and particularly τ2 (54%) and prediction intervals (96.6%) frequently were not reported for the hypertension subgroup. The recalculated prediction intervals were discrepant (i.e., crossed the null effect) of significant 95% confidence intervals of most meta-analyses (systolic BP: 65%; diastolic BP: 92%). This suggests substantial heterogeneity across studies, which was often not acknowledged by authors' conclusions (78%). Consequently, downgrading the certainty of the available evidence may be justified alone due to heterogeneity across studies. Finally, we illustrate areas for improving I2 interpretation and provide practical recommendations on how to address statistical heterogeneity across all stages of a SRMA.

Evaluation of statistical methods used to meta-analyse results from interrupted time series studies: A simulation study

Interrupted time series (ITS) are often meta-analysed to inform public health and policy decisions but examination of the statistical methods for ITS analysis and meta-analysis in this context is limited. We simulated meta-analyses of ITS studies with continuous outcome data, analysed the studies using segmented linear regression with two estimation methods [ordinary least squares (OLS) and restricted maximum likelihood (REML)], and meta-analysed the immediate level- and slope-change effect estimates using fixed-effect and (multiple) random-effects meta-analysis methods. Simulation design parameters included varying series length; magnitude of lag-1 autocorrelation; magnitude of level- and slope-changes; number of included studies; and, effect size heterogeneity. All meta-analysis methods yielded unbiased estimates of the interruption effects. All random effects meta-analysis methods yielded coverage close to the nominal level, irrespective of the ITS analysis method used and other design parameters. However, heterogeneity was frequently overestimated in scenarios where the ITS study standard errors were underestimated, which occurred for short series or when the ITS analysis method did not appropriately account for autocorrelation. The performance of meta-analysis methods depends on the design and analysis of the included ITS studies. Although all random effects methods performed well in terms of coverage, irrespective of the ITS analysis method, we recommend the use of effect estimates calculated from ITS methods that adjust for autocorrelation when possible. Doing so will likely to lead to more accurate estimates of the heterogeneity variance.

The problems with systematic reviews: a living systematic review

OBJECTIVES

Systematic reviews and meta-analyses are proliferating as they are an important building block to inform evidence-based guidelines and decision-making. Enforcement of best practice in clinical trials is firmly on the research agenda of good clinical practice, but there is less clarity as to how evidence syntheses that combine these studies can be influenced by bad practice. Our aim was to conduct a living systematic review of articles that highlight flaws in published systematic reviews to formally document and understand these problems.

STUDY DESIGN AND SETTING

We conducted a comprehensive assessment of all literature examining problems, which relate to published systematic reviews.

RESULTS

The first iteration of our living systematic review (https://systematicreviewlution.com/) has found 485 articles documenting 67 discrete problems relating to the conduct and reporting of systematic reviews which can potentially jeopardize their reliability or validity.

CONCLUSION

Many hundreds of articles highlight that there are many flaws in the conduct, methods, and reporting of published systematic reviews, despite the existence and frequent application of guidelines. Considering the pivotal role that systematic reviews have in medical decision-making due to having apparently transparent, objective, and replicable processes, a failure to appreciate and regulate problems with these highly cited research designs is a threat to credible science.

Introducing the Library of Guidance for Health Scientists (LIGHTS): A Living Database for Methods Guidance

IMPORTANCE

Improving methodological quality is a priority in the health research community. Finding appropriate methods guidance can be challenging due to heterogeneous terminology, poor indexing in medical databases, and variation in formats. The Library of Guidance for Health Scientists (LIGHTS) is a new searchable database for methods guidance articles.

OBSERVATIONS

Journal articles that aim to provide guidance for performing (including planning, design, conduct, analysis, and interpretation), reporting, and assessing the quality of health-related research involving humans or human populations (ie, excluding basic and animal research) are eligible for LIGHTS. A team of health researchers, information specialists, and methodologists continuously identifies and manually indexes eligible guidance documents. The search strategy includes focused searches of specific journals, specialized databases, and suggestions from researchers. A current limitation is that a keyword-based search of MEDLINE (and other general databases) and manual screening of records were not feasible because of the large number of hits (n = 915 523). As of September 20, 2022, LIGHTS included 1246 articles (336 reporting guidelines, 80 quality assessment tools, and 830 other methods guidance articles). The LIGHTS website provides a user-oriented search interface including filters for study type, specific methodological topic, research context, guidance type, and development process of the guidance. Automated matching of alternative methodological expressions (eg, enter loss to follow-up and find articles indexed with missing data) enhances search queries.

CONCLUSIONS AND RELEVANCE

LIGHTS is a peer-supported initiative that is intended to increase access to and use of methods guidance relevant to health researchers, statisticians, methods consultants, methods developers, ethics boards, peer reviewers, journal editors, and funding bodies.

[The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsDeclaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas]

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews.

Meta-analysis of prevalence: I2 statistic and how to deal with heterogeneity

Over the last decade there has been a ten-fold increase in the number of published systematic reviews of prevalence. In meta-analyses of prevalence, the summary estimate represents an average prevalence from included studies. This estimate is truly informative only if there is no substantial heterogeneity among the different contexts being pooled. In systematic reviews, heterogeneity is usually explored with I-squared statistic (I² ), but this statistic does not directly inform us about the distribution of effects and frequently systematic reviewers and readers misinterpret this result. In a sample of 134 meta-analyses of prevalence, the median I² was 96.9% (IQR 90.5 to 98.7). We observed larger I² in meta-analysis with higher number of studies and extreme pooled estimates (defined as <10% or > 90%). Studies with high I² values were more likely to have conducted a sensitivity analysis, including subgroup analysis. Only 3 (2%) systematic reviews reported prediction intervals. Meta-analyses of prevalence often present high I² values. However, the number of studies included in the meta-analysis and the point estimate can be associated with the I² value, and a high I² value is not always synonymous with high heterogeneity. In meta-analyses of prevalence, I² statistics may not be discriminative and should be interpreted with caution, avoiding arbitrary thresholds. To discuss heterogeneity, reviewers should focus on the description of the expected range of estimates, which can be done using prediction intervals and planned sensitivity analysis. This article is protected by copyright. All rights reserved.

Methodological systematic review recommends improvements to conduct and reporting when meta-analysing interrupted time series studies

OBJECTIVES

Interrupted Time Series (ITS) are a type of non-randomised design commonly used to evaluate public health policy interventions, and the impact of exposures, at the population level. Meta-analysis may be used to combine results from ITS across studies (in the context of systematic reviews) or across sites within the same study. We aimed to examine the statistical approaches, methods, and completeness of reporting in reviews that meta-analyse results from ITS.

STUDY DESIGN AND SETTINGS

Eight electronic databases were searched to identify reviews (published 2000-2019) that meta-analysed at least two ITS. Characteristics of the included reviews, the statistical methods used to analyse the ITS and meta-analyse their results, effect measures, and risk of bias assessment tools were extracted.

RESULTS

Of the 4213 identified records, 54 reviews were included. Nearly all reviews (94%) used two-stage meta-analysis, most commonly fitting a random effects model (69%). Among the 41 reviews that re-analysed the ITS, linear regression (39%) and ARIMA (20%) were most commonly used; 38% adjusted for autocorrelation. The most common effect measure meta-analysed was an immediate level-change (46/54). Reporting of the statistical methods and ITS characteristics was often incomplete.

CONCLUSION

Improvement is needed in the conduct and reporting of reviews that meta-analyse results from ITS.

The application of AMSTAR2 in 32 overviews of systematic reviews of interventions for mental and behavioural disorders: A cross-sectional study

'A measurement tool to assess systematic reviews, version 2' (AMSTAR2) is a 16-item tool to critically appraise systematic reviews (SRs) of healthcare interventions. This study aimed to assess the methods and outcomes of AMSTAR2 appraisals in overviews of SRs of interventions for mental and behavioural disorders. The cross-sectional study was conducted using 32 overviews of SRs selected from three electronic databases in January 2021. Data items included overview and SR characteristics and AMSTAR2 appraisal methods and outcomes. Data were extracted by two authors independently and narratively synthesised using descriptive statistics (means ± SD and relative frequencies). SR characteristics were compared based on AMSTAR2 appraisal outcomes using chi-square tests. The 32 overviews appraised SRs of predominantly non-pharmacological interventions for mental disorders. AMSTAR2 appraisals were reported as confidence ratings in 25/32 overviews or individual item scores in 24/32 overviews. Most SRs/overview were non-Cochrane (mean = 94%), included RCTs only (mean = 77%) and were published before AMSTAR2 release (mean = 79%). The confidence ratings derived in 25 overviews for 349 SRs were predominantly critically low (68%). Confidence ratings were similar for SRs with RCTs only versus RCTs+non-RCTs or SRs published before versus after AMSTAR2 release, while Cochrane SRs received more high+moderate than low+critically low confidence ratings (p < 0.01). Confidence ratings derived based on AMSTAR2 do not differentiate among SRs of healthcare interventions except for Cochrane SRs that fulfil the criteria for high confidence ratings. AMSTAR2 items should be consulted to avoid common weaknesses in future SRs.

The PRISMA 2020 statement: an updated guideline for reporting systematic reviews

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews. Full English text available from:www.revespcardiol.org/en.

Development of a checklist to detect errors in meta-analyses in systematic reviews of interventions: study protocol

Background : Systematic reviews underpin clinical practice and policies that guide healthcare decisions. A core component of many systematic reviews is meta-analysis, which is a statistical synthesis of results across studies. Errors in the conduct and interpretation of meta-analysis can lead to incorrect conclusions regarding the benefits and harms of interventions; and studies have shown that these errors are common. Enabling peer reviewers to better detect errors in meta-analysis through the use of a checklist provides an opportunity for these errors to be rectified before publication. To our knowledge, no such checklist exists. Objective : To develop and evaluate a checklist to detect errors in pairwise meta-analyses in systematic reviews of interventions. Methods : We will undertake a four-step process to develop the checklist. First, we will undertake a systematic review of studies that have evaluated errors in the conduct and interpretation of meta-analysis to generate a bank of items to consider for the checklist. Second, we will undertake a survey of systematic review methodologists and statisticians to seek their views on which items, of the bank of items generated in step 1, are most important to include in the checklist. Third, we will hold a virtual meeting to agree upon which items to include in the checklist. Fourth, before finalising the checklist, we will pilot with editors and peer reviewers of journals. Conclusion : The developed checklist is intended to help journal editors and peer reviewers identify errors in the application and interpretation of meta-analyses in systematic reviews. Fewer errors in the conduct and improved interpretation will lead to more accurate review findings and conclusions to inform clinical practice.

Prognostic models for knee osteoarthritis: a protocol for systematic review, critical appraisal, and meta-analysis

BACKGROUND

Osteoarthritis is the most common degenerative joint disease. It is associated with significant socioeconomic burden and poor quality of life, mainly due to knee osteoarthritis (KOA), and related total knee arthroplasty (TKA). Since early detection method and disease-modifying drug is lacking, the key of KOA treatment is shifting to disease prevention and progression slowing. The prognostic prediction models are called for to guide clinical decision-making. The aim of our review is to identify and characterize reported multivariable prognostic models for KOA about three clinical concerns: (1) the risk of developing KOA in the general population, (2) the risk of receiving TKA in KOA patients, and (3) the outcome of TKA in KOA patients who plan to receive TKA.

METHODS

The electronic datasets (PubMed, Embase, the Cochrane Library, Web of Science, Scopus, SportDiscus, and CINAHL) and gray literature sources (OpenGrey, British Library Inside, ProQuest Dissertations & Theses Global, and BIOSIS preview) will be searched from their inception onwards. Title and abstract screening and full-text review will be accomplished by two independent reviewers. The multivariable prognostic models that concern on (1) the risk of developing KOA in the general population, (2) the risk of receiving TKA in KOA patients, and (3) the outcome of TKA in KOA patients who plan to receive TKA will be included. Data extraction instrument and critical appraisal instrument will be developed before formal assessment and will be modified during a training phase in advance. Study reporting transparency, methodological quality, and risk of bias will be assessed according to the TRIPOD statement, CHARMS checklist, and PROBAST tool, respectively. Prognostic prediction models will be summarized qualitatively. Quantitative metrics on the predictive performance of these models will be synthesized with meta-analyses if appropriate.

DISCUSSION

Our systematic review will collate evidence from prognostic prediction models that can be used through the whole process of KOA. The review may identify models which are capable of allowing personalized preventative and therapeutic interventions to be precisely targeted at those individuals who are at the highest risk. To accomplish the prediction models to cross the translational gaps between an exploratory research method and a valued addition to precision medicine workflows, research recommendations relating to model development, validation, or impact assessment will be made.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42020203543.

Successful incorporation of single reviewer assessments during systematic review screening: development and validation of sensitivity and work-saved of an algorithm that considers exclusion criteria and count

BACKGROUND

Accepted systematic review (SR) methodology requires citation screening by two reviewers to maximise retrieval of eligible studies. We hypothesized that records could be excluded by a single reviewer without loss of sensitivity in two conditions; the record was ineligible for multiple reasons, or the record was ineligible for one or more specific reasons that could be reliably assessed.

METHODS

Twenty-four SRs performed at CHEO, a pediatric health care and research centre in Ottawa, Canada, were divided into derivation and validation sets. Exclusion criteria during abstract screening were sorted into 11 specific categories, with loss in sensitivity determined by individual category and by number of exclusion criteria endorsed. Five single reviewer algorithms that combined individual categories and multiple exclusion criteria were then tested on the derivation and validation sets, with success defined a priori as less than 5% loss of sensitivity.

RESULTS

The 24 SRs included 930 eligible and 27390 ineligible citations. The reviews were mostly focused on pediatrics (70.8%, N=17/24), but covered various specialties. Using a single reviewer to exclude any citation led to an average loss of sensitivity of 8.6% (95%CI, 6.0-12.1%). Excluding citations with ≥2 exclusion criteria led to 1.2% average loss of sensitivity (95%CI, 0.5-3.1%). Five specific exclusion criteria performed with perfect sensitivity: conference abstract, ineligible age group, case report/series, not human research, and review article. In the derivation set, the five algorithms achieved a loss of sensitivity ranging from 0.0 to 1.9% and work-saved ranging from 14.8 to 39.1%. In the validation set, the loss of sensitivity for all 5 algorithms remained below 2.6%, with work-saved between 10.5% and 48.2%.

CONCLUSIONS

Findings suggest that targeted application of single-reviewer screening, considering both type and number of exclusion criteria, could retain sensitivity and significantly decrease workload. Further research is required to investigate the potential for combining this approach with crowdsourcing or machine learning methodologies.

The PRISMA 2020 statement: An updated guideline for reporting systematic reviews

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews.

PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews

The methods and results of systematic reviews should be reported in sufficient detail to allow users to assess the trustworthiness and applicability of the review findings. The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement was developed to facilitate transparent and complete reporting of systematic reviews and has been updated (to PRISMA 2020) to reflect recent advances in systematic review methodology and terminology. Here, we present the explanation and elaboration paper for PRISMA 2020, where we explain why reporting of each item is recommended, present bullet points that detail the reporting recommendations, and present examples from published reviews. We hope that changes to the content and structure of PRISMA 2020 will facilitate uptake of the guideline and lead to more transparent, complete, and accurate reporting of systematic reviews.

The PRISMA 2020 statement: an updated guideline for reporting systematic reviews

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews.