Interval estimation of the overall treatment effect in a meta-analysis of a few small studies with zero events

The impact of correction methods on rare-event meta-analysis

Meta-analyses have become the gold standard for synthesizing evidence from multiple clinical trials, and they are especially useful when outcomes are rare or adverse since individual trials often lack sufficient power to detect a treatment effect. However, when zero events are observed in one or both treatment arms in a trial, commonly used meta-analysis methods can perform poorly. Continuity corrections (CCs), and numerical adjustments to the data to make computations feasible, have been proposed to ameliorate this issue. While the impact of various CCs on meta-analyses with rare events has been explored, how this impact varies based on the choice of pooling method and heterogeneity variance estimator is not widely understood. We compare several correction methods via a simulation study with a variety of commonly used meta-analysis methods. We consider how these method combinations impact important meta-analysis results, such as the estimated overall treatment effect, 95% confidence interval coverage, and Type I error rate. We also provide a website application of these results to aid researchers in selecting meta-analysis methods for rare-event data sets. Overall, no one-method combination can be consistently recommended, but some general trends are evident. For example, when there is no heterogeneity variance, we find that all pooling methods can perform well when paired with a specific correction method. Additionally, removing studies with zero events can work very well when there is no heterogeneity variance, while excluding single-zero studies results in poorer method performance when there is non-negligible heterogeneity variance and is not recommended.

Bayesian estimation and testing in random-effects meta-analysis of rare binary events allowing for flexible group variability

Rare binary events data arise frequently in medical research. Due to lack of statistical power in individual studies involving such data, meta-analysis has become an increasingly important tool for combining results from multiple independent studies. However, traditional meta-analysis methods often report severely biased estimates in such rare-event settings. Moreover, many rely on models assuming a pre-specified direction for variability between control and treatment groups for mathematical convenience, which may be violated in practice. Based on a flexible random-effects model that removes the assumption about the direction, we propose new Bayesian procedures for estimating and testing the overall treatment effect and inter-study heterogeneity. Our Markov chain Monte Carlo algorithm employs Pólya-Gamma augmentation so that all conditionals are known distributions, greatly facilitating computational efficiency. Our simulation shows that the proposed approach generally reports less biased and more stable estimates compared to existing methods. We further illustrate our approach using two real examples, one using rosiglitazone data from 56 studies and the other using stomach ulcers data from 41 studies.

Random-effects meta-analysis models for the odds ratio in the case of rare events under different data-generating models: A simulation study

Meta-analysis of binary data is challenging when the event under investigation is rare, and standard models for random-effects meta-analysis perform poorly in such settings. In this simulation study, we investigate the performance of different random-effects meta-analysis models in terms of point and interval estimation of the pooled log odds ratio in rare events meta-analysis. First and foremost, we evaluate the performance of a hypergeometric-normal model from the family of generalized linear mixed models (GLMMs), which has been recommended, but has not yet been thoroughly investigated for rare events meta-analysis. Performance of this model is compared to performance of the beta-binomial model, which yielded favorable results in previous simulation studies, and to the performance of models that are frequently used in rare events meta-analysis, such as the inverse variance model and the Mantel-Haenszel method. In addition to considering a large number of simulation parameters inspired by real-world data settings, we study the comparative performance of the meta-analytic models under two different data-generating models (DGMs) that have been used in past simulation studies. The results of this study show that the hypergeometric-normal GLMM is useful for meta-analysis of rare events when moderate to large heterogeneity is present. In addition, our study reveals important insights with regard to the performance of the beta-binomial model under different DGMs from the binomial-normal family. In particular, we demonstrate that although misalignment of the beta-binomial model with the DGM affects its performance, it shows more robustness to the DGM than its competitors.

Comparison of Demographic, Clinical, Biochemical, and Imaging Findings in Hypertrophic Cardiomyopathy Prognosis: A Network Meta-Analysis

BACKGROUND

Despite hypertrophic cardiomyopathy (HCM) being the most common inherited heart disease and conferring increased risk for heart failure (HF) and sudden cardiac death (SCD), risk assessment in HCM patients is still largely unresolved.

OBJECTIVES

This study aims to synthesize and compare the prognostic impact of demographic, clinical, biochemical, and imaging findings in patients with HCM.

METHODS

The authors searched PubMed, Embase, and Cochrane Library for studies published from 1955 to November 2020, and the endpoints were: 1) all-cause death; 2) an arrhythmic endpoint including SCD, sustained ventricular tachycardia, ventricular fibrillation, or aborted SCD; and 3) a composite endpoint including (1) or (2) plus hospitalization for HF or cardiac transplantation. The authors performed a pairwise meta-analysis obtaining the pooled estimate separately for the association between baseline variables and study endpoints. A random-effects network meta-analysis was subsequently used to comparatively assess the prognostic value of outcome associates.

RESULTS

A total of 112 studies with 58,732 HCM patients were included. Among others, increased brain natriuretic peptide/N-terminal pro-B-type natriuretic peptide, late gadolinium enhancement (LGE), positive genotype, impaired global longitudinal strain, and presence of apical aneurysm conferred increased risk for the composite endpoint. At network meta-analysis, LGE showed the highest prognostic value for all endpoints and was superior to all other associates except New York Heart Association functional class >class II. A multiparametric imaging-based model was superior in predicting the composite endpoint compared to a prespecified model based on conventional risk factors.

CONCLUSIONS

This network meta-analysis supports the development of multiparametric risk prediction algorithms, including advanced imaging markers additively to conventional risk factors, for refined risk stratification in HCM. (Long-term prognosis of hypertrophic cardiomyopathy according to genetic, clinical, biochemical and imaging findings: a systemic review and meta-analysis; CRD42020185219).

Bayesian heterogeneity in a meta-analysis with two studies and binary data

The meta-analysis of two trials is valuable in many practical situations, such as studies of rare and/or orphan diseases focussed on a single intervention. In this context, additional concerns, like small sample size and/or heterogeneity in the results obtained, might make standard frequentist and Bayesian techniques inappropriate. In a meta-analysis, moreover, the presence of between-sample heterogeneity adds model uncertainty, which must be taken into consideration when drawing inferences. We suggest that the most appropriate way to measure this heterogeneity is by clustering the samples and then determining the posterior probability of the cluster models. The meta-inference is obtained as a mixture of all the meta-inferences for the cluster models, where the mixing distribution is the posterior model probability. We present a simple two-component form of Bayesian model averaging that is unaffected by characteristics such as small study size or zero-cell counts, and which is capable of incorporating uncertainties into the estimation process. Illustrative examples are given and analysed, using real sparse binomial data.

A permutation-based approach for heterogeneous meta-analyses of rare events

The increasingly widespread use of meta-analysis has led to growing interest in meta-analytic methods for rare events and sparse data. Conventional approaches tend to perform very poorly in such settings. Recent work in this area has provided options for sparse data, but these are still often hampered when heterogeneity across the available studies differs based on treatment group. We propose a permutation-based approach based on conditional logistic regression that accommodates this common contingency, providing more reliable statistical tests when such patterns of heterogeneity are observed. We find that commonly used methods can yield highly inflated Type I error rates, low confidence interval coverage, and bias when events are rare and non-negligible heterogeneity is present. Our method often produces much lower Type I error rates and higher confidence interval coverage than traditional methods in these circumstances. We illustrate the utility of our method by comparing it to several other methods via a simulation study and analyzing an example data set, which assess the use of antibiotics to prevent acute rheumatic fever.

A comparison of confidence distribution approaches for rare event meta-analysis

Meta-analysis of rare event data has recently received increasing attention due to the challenging issues rare events pose to traditional meta-analytic methods. One specific way to combine information and analyze rare event meta-analysis data utilizes confidence distributions (CDs). While several CD methods exist, no comparisons have been made to determine which method is best suited for homogeneous or heterogeneous meta-analyses with rare events. In this article, we review several CD methods: Fisher's classic P-value combination method, one that combines P-value functions, another that combines confidence intervals, and one that combines confidence log-likelihood functions. We compare these CD approaches, and we propose and compare variations of these methods to determine which method produces reliable results for homogeneous or heterogeneous rare event meta-analyses. We find that for homogeneous rare event data, most CD methods perform very well. On the other hand, for heterogeneous rare event data, there is a clear split in performance between some CD methods, with some performing very poorly and others performing reasonably well.

Combined assessment of early and late-phase outcomes in orphan drug development

In drug development programs, proof‐of‐concept Phase II clinical trials typically have a biomarker as a primary outcome, or an outcome that can be observed with relatively short follow‐up. Subsequently, the Phase III clinical trials aim to demonstrate the treatment effect based on a clinical outcome that often needs a longer follow‐up to be assessed. Early‐phase outcomes or biomarkers are typically associated with late‐phase outcomes and they are often included in Phase III trials. The decision to proceed to Phase III development is based on analysis of the early‐Phase II outcome data. In rare diseases, it is likely that only one Phase II trial and one Phase III trial are available. In such cases and before drug marketing authorization requests, positive results of the early‐phase outcome of Phase II trials are then likely seen as supporting (or even replicating) positive Phase III results on the late‐phase outcome, without a formal retrospective combined assessment and without accounting for between‐study differences. We used double‐regression modeling applied to the Phase II and Phase III results to numerically mimic this informal retrospective assessment. We provide an analytical solution for the bias and mean square error of the overall effect that leads to a corrected double‐regression. We further propose a flexible Bayesian double‐regression approach that minimizes the bias by accounting for between‐study differences via discounting the Phase II early‐phase outcome when they are not in line with the Phase III biomarker outcome results. We illustrate all methods with an orphan drug example for Fabry disease.

Bayesian Meta-Analysis for Binary Data and Prior Distribution on Models

In meta-analysis, the structure of the between-sample heterogeneity plays a crucial role in estimating the meta-parameter. A Bayesian meta-analysis for binary data has recently been proposed that measures this heterogeneity by clustering the samples and then determining the posterior probability of the cluster models through model selection. The meta-parameter is then estimated using Bayesian model averaging techniques. Although an objective Bayesian meta-analysis is proposed for each type of heterogeneity, we concentrate the attention of this paper on priors over the models. We consider four alternative priors which are motivated by reasonable but different assumptions. A frequentist validation with simulated data has been carried out to analyze the properties of each prior distribution for a set of different number of studies and sample sizes. The results show the importance of choosing an adequate model prior as the posterior probabilities for the models are very sensitive to it. The hierarchical Poisson prior and the hierarchical uniform prior show a good performance when the real model is the homogeneity, or when the sample sizes are high enough. However, the uniform prior can detect the true model when it is an intermediate model (neither homogeneity nor heterogeneity) even for small sample sizes and few studies. An illustrative example with real data is also given, showing the sensitivity of the estimation of the meta-parameter to the model prior.

Meta-Analysis with Few Studies and Binary Data: A Bayesian Model Averaging Approach

In meta-analysis, the existence of between-sample heterogeneity introduces model uncertainty, which must be incorporated into the inference. We argue that an alternative way to measure this heterogeneity is by clustering the samples and then determining the posterior probability of the cluster models. The meta-inference is obtained as a mixture of all the meta-inferences for the cluster models, where the mixing distribution is the posterior model probabilities. When there are few studies, the number of cluster configurations is manageable, and the meta-inferences can be drawn with BMA techniques. Although this topic has been relatively neglected in the meta-analysis literature, the inference thus obtained accurately reflects the cluster structure of the samples used. In this paper, illustrative examples are given and analysed, using real binary data.

Prior distributions for variance parameters in a sparse-event meta-analysis of a few small trials

In rare diseases, typically only a small number of patients are available for a randomized clinical trial. Nevertheless, it is not uncommon that more than one study is performed to evaluate a (new) treatment. Scarcity of available evidence makes it particularly valuable to pool the data in a meta-analysis. When the primary outcome is binary, the small sample sizes increase the chance of observing zero events. The frequentist random-effects model is known to induce bias and to result in improper interval estimation of the overall treatment effect in a meta-analysis with zero events. Bayesian hierarchical modeling could be a promising alternative. Bayesian models are known for being sensitive to the choice of prior distributions for between-study variance (heterogeneity) in sparse settings. In a rare disease setting, only limited data will be available to base the prior on, therefore, robustness of estimation is desirable. We performed an extensive and diverse simulation study, aiming to provide practitioners with advice on the choice of a sufficiently robust prior distribution shape for the heterogeneity parameter. Our results show that priors that place some concentrated mass on small τ values but do not restrict the density for example, the Uniform(-10, 10) heterogeneity prior on the log(τ² ) scale, show robust 95% coverage combined with less overestimation of the overall treatment effect, across varying degrees of heterogeneity. We illustrate the results with meta-analyzes of a few small trials.

Controversies in the application of corticosteroids for pediatric septic shock treatment: a preferred reporting items for systematic reviews and meta-analysis-compliant updated meta-analysis

OBJECTIVES

Septic shock is the major cause of childhood mortality. However, the application of corticosteroids remains controversial. This work aimed to analyze the source of controversy based on existing data and recent randomized controlled trials by meta-analysis and to assess whether it can avoid these factors to guide clinical treatment.

METHODS

We searched the public databases up to 8 June 2019 and included only randomized controlled trials. The primary outcome was mortality. Sensitivity analysis, subgroup analysis, and dose-response meta-analysis were performed in this work.

RESULTS

We included twelve studies consisting of 701 children in the meta-analysis. For primary outcome, the fixed-effect model showed steroids could significantly reduce the mortality compared to the control (Odds Ratio: 0.67; 95% confidence interval: 0.46-0.98; P = .041). However, the random-effect model showed a negative result (Odds Ratio: 0.69; 95% confidence interval: 0.32-1.51; P = .252). None of the subgroup results rejected the null hypothesis that the overall effect equaled zero. Dose-response effect analysis showed that increased dosage at a low dosage might reduce the mortality, while at a high dosage, increasing the dose might increase the mortality. Moreover, the grading of recommendations assessment, development, and evaluation level of evidence is low for mortality.

CONCLUSIONS

Corticosteroid application is not recommended for septic shock children under current medical conditions.

Antithrombotic treatment in patients with stroke and supracardiac atherosclerosis

OBJECTIVE

To compare the efficacy and safety of oral anticoagulants vs antiplatelets in patients with stroke and atherosclerotic plaques in the aortic arch or cervical or intracranial arteries, collectively described as supracardiac atherosclerosis.

METHODS

We searched PubMed and Scopus until August 28, 2019, for randomized trials comparing oral anticoagulants vs antiplatelets in patients with stroke and supracardiac atherosclerosis using the terms "anticoagulant or anticoagulation" and "antiplatelet or aspirin" and "randomized controlled trial or RCT" and "stroke or cerebral ischemia" and "aortic or carotid or vertebrobasilar or intracranial or atherosclerosis or stenosis or arterial." Four outcomes were assessed: recurrent ischemic stroke, major ischemic event or death, major bleeding, and intracranial bleeding. Treatment effects (relative risk [RR] and 95% confidence interval [CI]) were estimated by meta-analysis using random-effects models.

RESULTS

Among 1,117 articles identified in the literature search, results from 10 randomized controlled trials involving 6,068 patients with stroke/TIA with supracardiac atherosclerosis were included in the meta-analysis. Recurrent ischemic stroke rates were 2.94 per 100 patient-years in the anticoagulant-assigned patients vs 3.30 per 100 patient-years in the antiplatelet-assigned patients (RR, 0.91; 95% CI, 0.70-1.18 for the SJ estimator, I² = 26%). Major ischemic event or death rates were 4.39 per 100 patient-years in anticoagulant-assigned patients vs 4.32 in antiplatelet-assigned patients (RR, 1.03; 95% CI, 0.79-1.35; I² = 54.5%). Major bleeding rates were 2.88 per 100 patient-years in anticoagulant-assigned patients vs 0.82 in antiplatelet-assigned patients (RR, 3.21; 95% CI, 1.96-5.24; I² = 46%).

CONCLUSION

This systematic review and meta-analysis showed that anticoagulant-assigned patients with stroke and supracardiac atherosclerosis were not at different risk of ischemic stroke recurrence and increased risk of major bleeding compared to antiplatelet-assigned patients.

Comparison of random-effects meta-analysis models for the relative risk in the case of rare events: A simulation study

Pooling the relative risk (RR) across studies investigating rare events, for example, adverse events, via meta-analytical methods still presents a challenge to researchers. The main reason for this is the high probability of observing no events in treatment or control group or both, resulting in an undefined log RR (the basis of standard meta-analysis). Other technical challenges ensue, for example, the violation of normality assumptions, or bias due to exclusion of studies and application of continuity corrections, leading to poor performance of standard approaches. In the present simulation study, we compared three recently proposed alternative models (random-effects [RE] Poisson regression, RE zero-inflated Poisson [ZIP] regression, binomial regression) to the standard methods in conjunction with different continuity corrections and to different versions of beta-binomial regression. Based on our investigation of the models' performance in 162 different simulation settings informed by meta-analyses from the Cochrane database and distinguished by different underlying true effects, degrees of between-study heterogeneity, numbers of primary studies, group size ratios, and baseline risks, we recommend the use of the RE Poisson regression model. The beta-binomial model recommended by Kuss (2015) also performed well. Decent performance was also exhibited by the ZIP models, but they also had considerable convergence issues. We stress that these recommendations are only valid for meta-analyses with larger numbers of primary studies. All models are applied to data from two Cochrane reviews to illustrate differences between and issues of the models. Limitations as well as practical implications and recommendations are discussed; a flowchart summarizing recommendations is provided.

ECMO, ARDS and meta-analyses: Bayes to the rescue?

PURPOSE

A recent meta-analysis by Munshi et al. (Lancet Respiratory Medicine, 2019) claimed mortality treatment efficacy for extra corporeal membrane oxygenation (ECMO) in the acute respitratory syndrome (ARDS) despite very low meta-analytic study numbers (n = 2 (RCTs), risk-ratio (RR) 0·73 (95%CI: 0·58-0·92); n = 5 (2 RCT, 3 observational), RR 0·69 (95%CI: 0·50-0·95)). We explore this efficacy claim by a comprehensive re-analysis of the data.

METHODS

Data were sourced from the two- and five-study meta-analyses, conducted using the Der-Simonian & Laird (DSL) method. A variety of frequentist (DSL, restricted maximum likelihood (REML), Paul-Mandel (PM), with/without Hartung-Knapp-Sidik-Jonkman variance correction), a beta-binomial model (BBN)) and Bayesian models (2 finite-mixture and several Markov-Chain-Monte-Carlo) were used to estimate treatment effects. Fragility-indices, the minimum patients changing mortality outcome needed to induce a conclusion change were also applied.

RESULTS

For the 2-study and 5-study meta-analysis only the uncorrected frequentist estimators (DSL, REML, PM) demonstrated significant RR. Except for the BBN model, which was significant for the 2-study meta-analysis, intervals for all other models included the null. Both meta-analyses demonstrated fragility.

CONCLUSIONS

Having canvassed the conduct of both meta-analyses presented by Munshi et al. and proffered alternative methods, we find no certainty regarding the efficacy of ECMO in ARDS.

Applicability and added value of novel methods to improve drug development in rare diseases

BACKGROUND

The ASTERIX project developed a number of novel methods suited to study small populations. The objective of this exercise was to evaluate the applicability and added value of novel methods to improve drug development in small populations, using real world drug development programmes as reported in European Public Assessment Reports.

METHODS

The applicability and added value of thirteen novel methods developed within ASTERIX were evaluated using data from 26 European Public Assessment Reports (EPARs) for orphan medicinal products, representative of rare medical conditions as predefined through six clusters. The novel methods included were 'innovative trial designs' (six methods), 'level of evidence' (one method), 'study endpoints and statistical analysis' (four methods), and 'meta-analysis' (two methods) and they were selected from the methods developed within ASTERIX based on their novelty; methods that discussed already available and applied strategies were not included for the purpose of this validation exercise. Pre-requisites for application in a study were systematized for each method, and for each main study in the selected EPARs it was assessed if all pre-requisites were met. This direct applicability using the actual study design was firstly assessed. Secondary, applicability and added value were explored allowing changes to study objectives and design, but without deviating from the context of the drug development plan. We evaluated whether differences in applicability and added value could be observed between the six predefined condition clusters.

RESULTS AND DISCUSSION

Direct applicability of novel methods appeared to be limited to specific selected cases. The applicability and added value of novel methods increased substantially when changes to the study setting within the context of drug development were allowed. In this setting, novel methods for extrapolation, sample size re-assessment, multi-armed trials, optimal sequential design for small sample sizes, Bayesian sample size re-estimation, dynamic borrowing through power priors and fall-back tests for co-primary endpoints showed most promise - applicable in more than 40% of evaluated EPARs in all clusters. Most of the novel methods were applicable to conditions in the cluster of chronic and progressive conditions, involving multiple systems/organs. Relatively fewer methods were applicable to acute conditions with single episodes. For the chronic clusters, Goal Attainment Scaling was found to be particularly applicable as opposed to other (non-chronic) clusters.

CONCLUSION

Novel methods as developed in ASTERIX can improve drug development programs. Achieving optimal added value of these novel methods often requires consideration of the entire drug development program, rather than reconsideration of methods for a specific trial. The novel methods tested were mostly applicable in chronic conditions, and acute conditions with recurrent episodes.