An appraisal of meta-analysis guidelines: how do they relate to safety outcomes?

Conducting separate reviews of benefits and harms could improve systematic reviews and meta-analyses

Guidance for systematic reviews of interventions recommends both benefits and harms be included. Systematic reviews may reach conclusions about harms (or lack of harms) that are not true when reviews include only some relevant studies, rely on incomplete data from eligible studies, use inappropriate methods for synthesizing data, and report results selectively. Separate reviews about harms could address some of these problems, and we argue that conducting separate reviews of harms is a feasible alternative to current standards and practices. Systematic reviews of potential benefits could be organized around the use of interventions for specific health problems. Systematic reviews of potential harms could be broader, including more diverse study designs and including all people at risk of harms (who might use the same intervention to treat different health problems). Multiple reviews about benefits could refer to a single review of harms. This approach could improve the reliability, completeness, and efficiency of systematic reviews.

Harms in Systematic Reviews Paper 2: Methods used to assess harms are neglected in systematic reviews of gabapentin

OBJECTIVE

We compared methods used with current recommendations for synthesizing harms in systematic reviews and meta-analyses (SRMAs) of gabapentin.

STUDY DESIGN & SETTING

We followed recommended systematic review practices. We selected reliable SRMAs of gabapentin (i.e., met a pre-defined list of methodological criteria) that assessed at least one harm. We extracted and compared methods in four areas: pre-specification, searching, analysis, and reporting. Whereas our focus in this paper is on the methods used, Part 2 examines the results for harms across reviews.

RESULTS

We screened 4320 records and identified 157 SRMAs of gabapentin, 70 of which were reliable. Most reliable reviews (51/70; 73%) reported following a general guideline for SRMA conduct or reporting, but none reported following recommendations specifically for synthesizing harms. Across all domains assessed, review methods were designed to address questions of benefit and rarely included the additional methods that are recommended for evaluating harms.

CONCLUSION

Approaches to assessing harms in SRMAs we examined are tokenistic and unlikely to produce valid summaries of harms to guide decisions. A paradigm shift is needed. At a minimal, reviewers should describe any limitations to their assessment of harms and provide clearer descriptions of methods for synthesizing harms.

Harms in Systematic Reviews Paper 1: An introduction to research on harms

OBJECTIVE

Most systematic reviews of interventions focus on potential benefits. Common methods and assumptions that are appropriate for assessing benefits can be inappropriate for harms. This paper provides a primer on researching harms, particularly in systematic reviews.

STUDY DESIGN AND SETTING

Commentary describing challenges with assessing harm.

RESULTS

Investigators should be familiar with various terminologies used to describe, classify, and group harms. Published reports of clinical trials include limited information about harms, so systematic reviewers should not depend on these studies and journal articles to reach conclusions about harms. Visualizations might improve communication of multiple dimensions of harms such as severity, relatedness, and timing.

CONCLUSION

The terminology, classification, detection, collection, and reporting of harms create unique challenges that take time, expertise, and resources to navigate in both primary studies and evidence syntheses. Systematic reviewers might reach incorrect conclusions if they focus on evidence about harms found in published reports of randomized trials of a particular health problem. Systematic reviews could be improved through better identification and reporting of harms in primary studies and through better training and uptake of appropriate methods for synthesizing evidence about harms.

Matching-Adjusted Indirect Comparison of Crisaborole Ointment 2% vs. Topical Calcineurin Inhibitors in the Treatment of Patients with Mild-to-Moderate Atopic Dermatitis

Introduction

Crisaborole topical ointment, 2%, is a nonsteroidal, topical anti-inflammatory phosphodiesterase-4 (PDE4) inhibitor that is approved for the treatment of mild-to-moderate atopic dermatitis (AD). The objective of the current analysis was to compare the efficacy of crisaborole 2% relative to pimecrolimus 1%, tacrolimus 0.03% and tacrolimus 0.1% in patients aged ≥ 2 years with mild-to-moderate AD by comparing improvement in Investigator’s Static Global Assessment scores ( (ISGA scores of 0/1 indicating “clear or almost clear”). ISGA was selected as the primary efficacy outcome given the US Food and Drug Administration’s recommendations on the use of ISGA for assessment of global severity in AD and to align with efficacy measurements in the crisaborole registration trials. Safety endpoints could not be analyzed due to differences in outcome definitions across studies.

Methods

Efficacy of crisaborole was evaluated using individual patient data (IPD) from two pivotal phase III randomized controlled trials (RCTs), and efficacy of comparators was evaluated using published RCTs included in a previous network meta-analysis. Vehicle controls were not comparable due to differences in ingredients and population imbalance and, therefore, an unanchored matching-adjusted indirect comparison (MAIC) was used, which reweighted IPD for crisaborole to estimate absolute response in comparator populations.

Results

The odds of achieving an improvement in ISGA score was higher with crisaborole 2% versus pimecrolimus 1% (odds ratio [OR] 2.03; 95% confidence interval [CI] 1.45–2.85; effective sample size =  627, reduced from 1021; p value < 0.001) and for crisaborole 2% versus tacrolimus 0.03% (OR 1.50; 95% CI 1.09–2.05; effective sample size = 311, reduced from 1021; p = 0.012).

Conclusion

The unanchored MAIC suggests that the odds of achieving an improvement in ISGA score is greater with crisaborole 2% than with pimecrolimus 1% or tacrolimus 0.03% in patients aged ≥ 2 years with mild-to-moderate AD. These results are consistent with findings from the previously published network meta-analysis, which used a different methodology for performing indirect treatment comparisons.

Supplementary Information

The online version contains supplementary material available at 10.1007/s13555-021-00646-1.

Meta-analysis of rare adverse events in randomized clinical trials: Bayesian and frequentist methods

BACKGROUND/AIMS

Regulatory approval of a drug or device involves an assessment of not only the benefits but also the risks of adverse events associated with the therapeutic agent. Although randomized controlled trials (RCTs) are the gold standard for evaluating effectiveness, the number of treated patients in a single RCT may not be enough to detect a rare but serious side effect of the treatment. Meta-analysis plays an important role in the evaluation of the safety of medical products and has advantage over analyzing a single RCT when estimating the rate of adverse events.

METHODS

In this article, we compare 15 widely used meta-analysis models under both Bayesian and frequentist frameworks when outcomes are extremely infrequent or rare. We present extensive simulation study results and then apply these methods to a real meta-analysis that considers RCTs investigating the effect of rosiglitazone on the risks of myocardial infarction and of death from cardiovascular causes.

RESULTS

Our simulation studies suggest that the beta hyperprior method modeling treatment group-specific parameters and accounting for heterogeneity performs the best. Most models ignoring between-study heterogeneity give poor coverage probability when such heterogeneity exists. In the data analysis, different methods provide a wide range of log odds ratio estimates between rosiglitazone and control treatments with a mixed conclusion on their statistical significance based on 95% confidence (or credible) intervals.

CONCLUSION

In the rare event setting, treatment effect estimates obtained from traditional meta-analytic methods may be biased and provide poor coverage probability. This trend worsens when the data have large between-study heterogeneity. In general, we recommend methods that first estimate the summaries of treatment-specific risks across studies and then relative treatment effects based on the summaries when appropriate. Furthermore, we recommend fitting various methods, comparing the results and model performance, and investigating any significant discrepancies among them.

Assessment of the course of acute pancreatitis in the light of aetiology: a systematic review and meta-analysis

The main causes of acute pancreatitis (AP) are biliary disease, alcohol consumption, hypertriglyceridaemia (HTG) and endoscopic retrograde cholangiopancreatography (ERCP). The aim of this meta-analysis was to evaluate the effects of these aetiological factors on the severity and outcome of AP. Pubmed and Embase were searched between 01/01/2012 and 31/05/2020. Included articles involved adult alcoholic, biliary, HTG- or post-ERCP AP (PAP) patients. Primary outcome was severity, secondary outcomes were organ failures, intensive care unit admission, recurrence rate, pancreatic necrosis, mortality, length of hospital stay, pseudocyst, fluid collection and systematic inflammatory response syndrome. Data were analysed from 127 eligible studies. The risk for non-mild (moderately severe and severe) condition was the highest in HTG-induced AP (HTG-AP) followed by alcoholic AP (AAP), biliary AP (BAP) and PAP. Recurrence rate was significantly lower among BAP vs. HTG-AP or AAP patients (OR = 2.69 and 2.98, 95% CI 1.55–4.65 and 2.22–4.01, respectively). Mortality rate was significantly greater in HTG-AP vs. AAP or BAP (OR = 1.72 and 1.50, 95% CI 1.04–2.84 and 0.96–2.35, respectively), pancreatic necrosis occurred more frequently in AAP than BAP patients (OR = 1.58, 95% CI 1.08–2.30). Overall, there is a potential association between aetiology and the development and course of AP. HTG-AP is associated with the highest number of complications. Furthermore, AAP is likely to be more severe than BAP or PAP. Greater emphasis should be placed on determining aetiology on admission.

Assessing the quality of meta-analyses in systematic reviews in pharmaceutical research in Iran by 2016: A systematic review

Background: Meta-analyses, like all other studies, may be poorly designed and implemented. This study was designed to determine the quality of meta-analyses in systematic reviews in the field of pharmaceutical research in Iran. Methods: Web of Science Core Collection, EMBASE, Ovid Medline, CINAHL, Scopus, and PubMed were systematically searched on June 4, 2017. The search was limited to the researches in the field of pharmaceutical studies. Based on inclusion criteria, 104 systematic reviews with meta-analysis (SRMA) were selected and assessed using quality assessment tools introduced by Higgins. Results: Participants, experimental interventions, and outcomes were reported in all the articles. Comparator intervention and study design were correctly reported in 103 (99.04%) and 101 (97.12%) articles, respectively. The comprehensive search strategy was available only in 48 articles (46.16%), and there was no evidence of a comprehensive search in 56 articles (53.84%). Risk of bias was investigated in 78 articles (75%). Also, funnel plots were the most commonly used method for reporting the bias in 64 articles (46.42%). Conclusion: In many of the meta-analyses, several items of the tool that represented a high-quality meta-analysis were absent. According to the findings, the comprehensive search and quality assessment were not at an appropriate level. Thus, the importance of reproducibility of information and quality assessment of included studies should be emphasized.

Prothrombin complex concentrate versus placebo, no intervention, or other interventions in critically bleeding patients associated with oral anticoagulant administration: a protocol for a systematic review of randomised clinical trials with meta-analysis and trial sequential analysis

BACKGROUND

Acute critical bleeding is one of the most feared complications during treatment with oral anticoagulating agents. As more patients undergo treatment with anticoagulating agents, critically bleeding episodes in patients with vitamin K antagonists, thrombin inhibitor, or factor Xa inhibitor-inducted coagulopathy will be encountered frequently by physicians. Hence, an effective treatment capable of reversing the iatrogenic coagulopathy in the acute setting is needed. In randomised clinical trials and observational studies, prothrombin complex concentrate has been reported to be superior to other acute interventions, and many guidelines recommend prothrombin complex concentrate in treatment of critically bleeding patients. The aim of this systematic review is to synthesise the evidence of the effects of prothrombin complex concentrate compared with placebo, no intervention, or other treatment options in critically bleeding patients treated with oral anticoagulants.

METHODS/DESIGN

A comprehensive search for relevant published literature will be undertaken in Cochrane Central Register of Controlled Trials, MEDLINE, Embase, WHO International Clinical Trials Registry Platform, Science Citation Index, regulatory databases, and trial registers. We will include randomised clinical trials comparing prothrombin complex concentrate versus placebo, no intervention, or other interventions in critically bleeding patients with oral anticoagulant-induced coagulopathy. Data extraction and risk of bias assessment will be handled by two independent review authors. Meta-analysis will be performed as recommended by Cochrane Handbook for Systematic Reviews of Interventions, bias will be assessed with domains, and trial sequential analysis will be conducted to control random errors. Certainty will be assessed by GRADE.

DISCUSSION

As critical bleeding in patients treated with oral anticoagulants is an increasing problem, an up-to-date systematic review evaluating the benefits and harms of prothrombin complex concentrate is urgently needed. It is the hope that this review will be able to guide best practice in treatment and clinical research of these critically bleeding patients.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42018084371.