Exclusion of studies with no events in both arms in meta-analysis impacted the conclusions

Assessment of inverse publication bias in safety outcomes: an empirical analysis

BACKGROUND

The aims of this study were to assess the presence of inverse publication bias (IPB) in adverse events, evaluate the performance of visual examination, and explore the impact of considering effect direction in statistical tests for such assessments.

METHODS

We conducted a cross-sectional study using the SMART Safety, the largest dataset for evidence synthesis of adverse events. The visual assessment was performed using contour-enhanced funnel plots, trim-and-fill funnel plots, and sample-size-based funnel plots. Two authors conducted visual assessments of these plots independently, and their agreements were quantified by the kappa statistics. Additionally, IPB was quantitatively assessed using both the one- and two-sided Egger's and Peters' tests.

RESULTS

In the SMART Safety dataset, we identified 277 main meta-analyses of safety outcomes with at least 10 individual estimates after dropping missing data. We found that about 13.7-16.2% of meta-analyses exhibited IPB according to the one-sided test results. The kappa statistics for the visual assessments roughly ranged from 0.3 to 0.5, indicating fair to moderate agreement. Using the one-sided Egger's test, 57 out of 72 (79.2%) meta-analyses that initially showed significant IPB in the two-sided test changed to non-significant, while the remaining 15 (20.8%) meta-analyses changed from non-significant to significant.

CONCLUSIONS

Our findings provide supporting evidence of IPB in the SMART Safety dataset of adverse events. They also suggest the importance of researchers carefully accounting for the direction of statistical tests for IPB, as well as the challenges of assessing IPB using statistical methods, especially considering that the number of studies is typically small. Qualitative assessments may be a necessary supplement to gain a more comprehensive understanding of IPB.

Methods for assessing inverse publication bias of adverse events

In medical research, publication bias (PB) poses great challenges to the conclusions from systematic reviews and meta-analyses. The majority of efforts in methodological research related to classic PB have focused on examining the potential suppression of studies reporting effects close to the null or statistically non-significant results. Such suppression is common, particularly when the study outcome concerns the effectiveness of a new intervention. On the other hand, attention has recently been drawn to the so-called inverse publication bias (IPB) within the evidence synthesis community. It can occur when assessing adverse events because researchers may favor evidence showing a similar safety profile regarding an adverse event between a new intervention and a control group. In comparison to the classic PB, IPB is much less recognized in the current literature; methods designed for classic PB may be inaccurately applied to address IPB, potentially leading to entirely incorrect conclusions. This article aims to provide a collection of accessible methods to assess IPB for adverse events. Specifically, we discuss the relevance and differences between classic PB and IPB. We also demonstrate visual assessment through contour-enhanced funnel plots tailored to adverse events and popular quantitative methods, including Egger's regression test, Peters' regression test, and the trim-and-fill method for such cases. Three real-world examples are presented to illustrate the bias in various scenarios, and the implementations are illustrated with statistical code. We hope this article offers valuable insights for evaluating IPB in future systematic reviews of adverse events.

ZIBGLMM: Zero-Inflated Bivariate Generalized Linear Mixed Model for Meta-Analysis with Double-Zero-Event Studies

Double-zero-event studies (DZS) pose a challenge for accurately estimating the overall treatment effect in meta-analysis. Current approaches, such as continuity correction or omission of DZS, are commonly employed, yet these ad hoc methods can yield biased conclusions. Although the standard bivariate generalized linear mixed model can accommodate DZS, it fails to address the potential systemic differences between DZS and other studies. In this paper, we propose a zero-inflated bivariate generalized linear mixed model (ZIBGLMM) to tackle this issue. This two-component finite mixture model includes zero-inflation for a subpopulation with negligible or extremely low risk. We develop both frequentist and Bayesian versions of ZIBGLMM and examine its performance in estimating risk ratios (RRs) against the bivariate generalized linear mixed model and conventional two-stage meta-analysis that excludes DZS. Through extensive simulation studies and real-world meta-analysis case studies, we demonstrate that ZIBGLMM outperforms the bivariate generalized linear mixed model and conventional two-stage meta-analysis that excludes DZS in estimating the true effect size with substantially less bias and comparable coverage probability.

Comparing clinical and echocardiographic outcomes following valve-sparing versus transannular patch repair of tetralogy of Fallot: a systematic review and meta-analysis

OBJECTIVES

Transannular patch (TAP) repair of tetralogy of Fallot (ToF)relieves right ventricular tract obstruction but may lead to pulmonary regurgitation. Valve-sparing (VS) procedures can avoid this situation, but there is a potential for residual pulmonary stenosis. Our goal was to evaluate clinical and echocardiographic outcomes of TAP and VS repair for ToF.

METHODS

A systematic search of the PubMed, Embase, Scopus, Cochrane Central Register of Controlled Trials and Web of Science databases was carried out to identify articles comparing conventional TAP repair and VS repair for ToF. Random-effects models were used to perform meta-analyses of the clinical and echocardiographic outcomes.

RESULTS

Forty studies were included in this meta-analysis with data on 11 723 participants (TAP: 6171; VS: 5045). Participants who underwent a VS procedure experienced a significantly lower cardiopulmonary bypass time [mean difference (MD): -14.97; 95% confidence interval (CI): -22.54, -7.41], shorter ventilation duration (MD: -15.33; 95% CI: -30.20, -0.46) and shorter lengths of both intensive care unit (ICU) (MD: -0.67; 95% CI: -1.29, -0.06) and hospital stays (MD: -2.30; 95% CI: [-4.08, -0.52). There was also a lower risk of mortality [risk ratio: 0.40; 95% CI: (0.27, 0.60) and pulmonary regurgitation [risk ratio: 0.35; 95% CI: (0.26, 0.46)] associated with the VS group. Most other clinical and echocardiographic outcomes were comparable in the 2 groups.

CONCLUSIONS

This meta-analysis confirms the well-established increased risk of pulmonary insufficiency following TAP repair while also demonstrating that VS repairs are associated with several improved clinical outcomes. Continued research can identify the criteria for adopting a VS approach as opposed to a traditional TAP repair.

Predictors for Long-Term Survival After Resection of Pancreatic Ductal Adenocarcinoma: A Systematic Review and Meta-Analysis

BACKGROUND

Improved systemic therapy has made long term (≥ 5 years) overall survival (LTS) after resection of pancreatic ductal adenocarcinoma (PDAC) increasingly common. However, a systematic review on predictors of LTS following resection of PDAC is lacking.

METHODS

The PubMed, Embase, Scopus, and Cochrane CENTRAL databases were systematically searched from inception until March 2023. Studies reporting actual survival data (based on follow-up and not survival analysis estimates) on factors associated with LTS were included. Meta-analyses were conducted by using a random effects model, and study quality was gauged by using the Newcastle-Ottawa Scale (NOS).

RESULTS

Twenty-five studies with 27,091 patients (LTS: 2,132, non-LTS: 24,959) who underwent surgical resection for PDAC were meta-analyzed. The median proportion of LTS patients was 18.32% (IQR 12.97-21.18%) based on 20 studies. Predictors for LTS included sex, body mass index (BMI), preoperative levels of CA19-9, CEA, and albumin, neutrophil-lymphocyte ratio, tumor grade, AJCC stage, lymphovascular and perineural invasion, pathologic T-stage, nodal disease, metastatic disease, margin status, adjuvant therapy, vascular resection, operative time, operative blood loss, and perioperative blood transfusion. Most articles received a "good" NOS assessment, indicating an acceptable risk of bias.

CONCLUSIONS

Our meta-analysis pools all true follow up data in the literature to quantify associations between prognostic factors and LTS after resection of PDAC. While there appears to be evidence of a complex interplay between risk, tumor biology, patient characteristics, and management related factors, no single parameter can predict LTS after the resection of PDAC.

Does the use of tibial stem extensions reduce the risk of aseptic loosening in obese patients undergoing primary total knee arthroplasty: A systematic review and meta-analysis

BACKGROUND

This study aimed to compare the risk of revision for aseptic loosening in obese (body mass index >30 kg/m2) patients with stemmed (ST) versus non-stemmed (NST) tibial implants in primary total knee arthroplasty (TKA).

METHODS

A systematic review and meta-analysis were conducted following PRISMA and MOOSE guidelines. Studies reporting a direct comparison between ST and NST tibial implants in obese patients were included. The primary outcome of interest was revision for aseptic loosening. Outcomes were analysed using meta-analysis of relative risk. Risk of bias assessment was performed using the Newcastle-Ottawa Scale for observational studies and the RoB-2 Cochrane tool for randomised studies.

RESULTS

Seven studies met the selection criteria, consisting of four cohort studies and three randomised controlled trials. Mean follow up time for the eligible cohort was 62.6 months. Meta-analysis demonstrated a statistically significant reduction in the risk of aseptic revision in the ST group compared with the NST group (risk ratio 0.25, 95% confidence interval 0.07 to 0.92). After removal of all zero-event studies, the results remained in favour of the ST group (risk ratio 0.15, 95% confidence interval 0.03 to 0.64).

CONCLUSIONS

This study found that obese patients undergoing TKA with stemmed tibial implants may have a lower risk of aseptic revision compared with those with non-stemmed tibial implants. However, due to the lack of high-quality literature available, our study is unable to draw a definitive conclusion on this matter. We suggest that this topic should be re-evaluated using higher-quality study methods, particularly national joint registries studies and randomised controlled trials.

Methods for meta-analysis and meta-regression of binomial data: concepts and tutorial with Stata command metapreg

BACKGROUND

Despite the widespread interest in meta-analysis of proportions, its rationale, certain theoretical and methodological concepts are poorly understood. The generalized linear models framework is well-established and provides a natural and optimal model for meta-analysis, network meta-analysis, and meta-regression of proportions. Nonetheless, generic methods for meta-analysis of proportions based on the approximation to the normal distribution continue to dominate.

METHODS

We developed metapreg, a tool with advanced statistical procedures to perform a meta-analysis, network meta-analysis, and meta-regression of binomial proportions in Stata using binomial, logistic and logistic-normal models. First, we explain the rationale and concepts essential in understanding statistical methods for meta-analysis of binomial proportions and describe the models implemented in metapreg. We then describe and demonstrate the models in metapreg using data from seven published meta-analyses. We also conducted a simulation study to compare the performance of metapreg estimators with the existing estimators of the population-averaged proportion in metaprop and metan under a broad range of conditions including, high over-dispersion and small meta-analysis.

CONCLUSION

metapreg is a flexible, robust and user-friendly tool employing a rigorous approach to evidence synthesis of binomial data that makes the most efficient use of all available data and does not require ad-hoc continuity correction or data imputation. We expect its use to yield higher-quality meta-analysis of binomial proportions.

Comparison of statistical methods for integrating real-world evidence in a rare events meta-analysis of randomized controlled trials

Rare events meta-analyses of randomized controlled trials (RCTs) are often underpowered because the outcomes are infrequent. Real-world evidence (RWE) from non-randomized studies may provide valuable complementary evidence about the effects of rare events, and there is growing interest in including such evidence in the decision-making process. Several methods for combining RCTs and RWE studies have been proposed, but the comparative performance of these methods is not well understood. We describe a simulation study that aims to evaluate an array of alternative Bayesian methods for including RWE in rare events meta-analysis of RCTs: the naïve data synthesis, the design-adjusted synthesis, the use of RWE as prior information, the three-level hierarchical models, and the bias-corrected meta-analysis model. The percentage bias, root-mean-square-error, mean 95% credible interval width, coverage probability, and power are used to measure performance. The various methods are illustrated using a systematic review to evaluate the risk of diabetic ketoacidosis among patients using sodium/glucose co-transporter 2 inhibitors as compared with active-comparators. Our simulations show that the bias-corrected meta-analysis model is comparable to or better than the other methods in terms of all evaluated performance measures and simulation scenarios. Our results also demonstrate that data solely from RCTs may not be sufficiently reliable for assessing the effects of rare events. In summary, the inclusion of RWE could increase the certainty and comprehensiveness of the body of evidence of rare events from RCTs, and the bias-corrected meta-analysis model may be preferable.

Infective Endocarditis Risk with Melody versus Sapien Valves Following Transcatheter Pulmonary Valve Implantation: A Systematic Review and Meta-Analysis of Prospective Cohort Studies

BACKGROUND

Transcatheter pulmonary valve implantation (TPVI) is an effective non-surgical treatment method for patients with right ventricle outflow tract dysfunction. The Medtronic Melody and the Edwards Sapien are the two valves approved for use in TPVI. Since TPVI patients are typically younger, even a modest annual incidence of infective endocarditis (IE) is significant. Several previous studies have shown a growing risk of IE after TPVI. There is uncertainty regarding the overall incidence of IE and differences in the risk of IE between the valves.

METHODS

A systematic search was conducted in the MEDLINE, EMBASE, PubMed, and Cochrane databases from inception to 1 January 2023 using the search terms 'pulmonary valve implantation', 'TPVI', or 'PPVI'. The primary outcome was the pooled incidence of IE following TPVI in Melody and Sapien valves and the difference in incidence between Sapien and Melody valves. Fixed effect and random effect models were used depending on the valve. Meta-regression with random effects was conducted to test the difference in the incidence of IE between the two valves.

RESULTS

A total of 22 studies (including 10 Melody valve studies, 8 Sapien valve studies, and 4 studies that included both valves (572 patients that used the Sapien valve and 1395 patients that used the Melody valve)) were used for the final analysis. Zero IE incidence following TPVI was reported by eight studies (66.7%) that utilized Sapien valves compared to two studies (14.3%) that utilized Melody valves. The pooled incidence of IE following TPVI with Sapien valves was 2.1% (95% CI: 0.9% to 5.13%) compared to 8.5% (95% CI: 4.8% to 15.2%) following TPVI with Melody valves. Results of meta-regression indicated that the Sapien valve had a 79.6% (95% CI: 24.2% to 94.4%, p = 0.019; R2 = 34.4) lower risk of IE incidence compared to the Melody valve.

CONCLUSIONS

The risk of IE following TPVI differs significantly. A prudent valve choice in favor of Sapien valves to lower the risk of post-TPVI endocarditis may be beneficial.

Effect of important modifiers on harmful effects in evidence synthesis practice of adverse events were insufficiently investigated: an empirical investigation

BACKGROUND

Safety is important in the assessment of health interventions, while the results of adverse events are often susceptive to potential effect modifiers since the event risk tends to be rare. In this study, we investigated whether the potential impact of the important effect modifiers on harmful effects was analyzed in meta-analyses of adverse events.

METHODS

Systematic reviews of healthcare interventions, had adverse events as the exclusive outcomes, had at least one meta-analysis, and published between 1st January 2015, and 1st January 2020 were collected. An adverse event was defined as any untoward medical occurrence in a patient or subject in healthcare practice. Six effect modifiers that are the most important for harmful effects were identified by a group discussion. The proportions of eligible systematic reviews that investigated the potential impact of the six effect modifiers on harmful effects were summarized.

RESULTS

We identified 279 systematic reviews eligible for this study. Except for the modifier of interventions/controls (70.61%, 197/279), most of the systematic reviews failed to investigate the potential impact of treatment duration (21.15%, 59/279), dosage (24.73%, 69/279), age (11.47%, 32/279), risk of bias (6.45%, 18/279), and source of funding (1.08%, 3/279) on harmful effects. Systematic reviews with meta-analyses containing more studies were more likely to investigate the potential impacts of these modifiers on the effects, but the proportion was still low (2.3% to 33.3%). Systematic reviews that developed a protocol were significantly more likely to investigate the potential impact of all these effect modifiers (e.g. treatment duration: odds ratio = 5.08, 95% CI: 2.76 to 9.35) on the results.

CONCLUSIONS

Current systematic reviews rarely investigated the potential impact of the important effect modifiers on harmful effects. Methodological guidelines for meta-analysis of adverse events should consider "effect modifier" as one of the domains to help systematic review authors better investigate harmful effects.

Adherence to the World Health Organization's physical activity recommendation in preschool-aged children: a systematic review and meta-analysis of accelerometer studies

BACKGROUND

The World Health Organization (WHO) recommend that preschool-aged children should engage in 180 min of total physical activity (TPA) including 60 min of moderate-to-vigorous physical activity (MVPA) each day. No systematic reviews or meta-analyses have pooled adherence to the recommendation across multiple studies. This study aimed to estimate the prevalence of preschool-aged children achieving the WHO's physical activity recommendation for young children, and determine if the prevalence differed between boys and girls.

METHODS

Primary literature searches were conducted on six online databases and a machine learning assisted systematic review was used to identify relevant studies. Studies written in English reporting on the prevalence of children aged 3-5 years achieving overall WHO physical activity recommendation or the individual TPA or MVPA aspects of the recommendation measured using accelerometers were eligible for inclusion. Random effects meta-analysis was used to determine the prevalence of preschools achieving the overall WHO recommendation and the individual TPA and MVPA aspect of the recommendation, and to determine difference in prevalence between boys and girls.

RESULTS

Forty-eight studies reporting on 20,078 preschool-aged children met the inclusion criteria. Based on the most commonly employed accelerometer cut-points across all aspects of the recommendation, 60% (95% Confidence Interval [CI] = 37%, 79%) of preschool-aged children adhered to the overall physical activity recommendation, 78% (95% CI = 38%, 95%) adhered to the TPA aspect of the recommendation, and 90% (95% CI = 81%, 95%) adhered to the MVPA aspect of the recommendation. There was substantial variability is prevalence estimates between different accelerometer cut-points. Girls were significantly less likely to achieve the overall recommendation and the MVPA aspect of the recommendation than boys were.

CONCLUSIONS

Although there was substantial variability in estimated prevalence of preschool-aged children adhering the WHO physical activity recommendation between various accelerometer cut-points, the weight of available evidence suggests that the majority of young children are adhering to the overall recommendation and the individual TPA and MVPA aspects of the recommendation. Large-scale, intercontinental surveillance studies are needed to further strengthen the evidence regarding the prevalence of preschool-aged children achieving physical activity recommendation globally.

Conducting separate reviews of benefits and harms could improve systematic reviews and meta-analyses

Guidance for systematic reviews of interventions recommends both benefits and harms be included. Systematic reviews may reach conclusions about harms (or lack of harms) that are not true when reviews include only some relevant studies, rely on incomplete data from eligible studies, use inappropriate methods for synthesizing data, and report results selectively. Separate reviews about harms could address some of these problems, and we argue that conducting separate reviews of harms is a feasible alternative to current standards and practices. Systematic reviews of potential benefits could be organized around the use of interventions for specific health problems. Systematic reviews of potential harms could be broader, including more diverse study designs and including all people at risk of harms (who might use the same intervention to treat different health problems). Multiple reviews about benefits could refer to a single review of harms. This approach could improve the reliability, completeness, and efficiency of systematic reviews.

Measuring the impact of zero-cases studies in evidence synthesis practice using the harms index and benefits index (Hi-Bi)

OBJECTIVES

In evidence synthesis practice, dealing with studies with no cases in both arms has been a tough problem, for which there is no consensus in the research community. In this study, we propose a method to measure the potential impact of studies with no cases for meta-analysis results which we define as harms index (Hi) and benefits index (Bi) as an alternative solution for deciding how to deal with such studies.

METHODS

Hi and Bi are defined by the minimal number of cases added to the treatment arm (Hi) or control arm (Bi) of studies with no cases in a meta-analysis that lead to a change of the direction of the estimates or its statistical significance. Both exact and approximating methods are available to calculate Hi and Bi. We developed the "hibi" module in Stata so that researchers can easily implement the method. A real-world investigation of meta-analyses from Cochrane reviews was employed to evaluate the proposed method.

RESULTS

Based on Hi and Bi, our results suggested that 21.53% (Hi) to 26.55% (Bi) of Cochrane meta-analyses may be potentially impacted by studies with no cases, for which studies with no cases could not be excluded from the synthesis. The approximating method shows excellent specificity (100%) for both Hi and Bi, moderate sensitivity (68.25%) for Bi, and high sensitivity (80.61%) for Hi compared to the exact method.

CONCLUSIONS

The proposed method is practical and useful for systematic reviewers to measure whether studies with no cases impact the results of meta-analyses and may act as an alternative solution for review authors to decide whether to include studies with no events for the synthesis or not.

A Bayesian model for combining standardized mean differences and odds ratios in the same meta-analysis

In meta-analysis practice, researchers frequently face studies that report the same outcome differently, such as a continuous variable (e.g., scores for rating depression) or a binary variable (e.g., counts of patients with depression dichotomized by certain latent and unreported depression scores). For combining these two types of studies in the same analysis, a simple conversion method has been widely used to handle standardized mean differences (SMDs) and odds ratios (ORs). This conventional method uses a linear function connecting the SMD and log OR; it assumes logistic distributions for (latent) continuous measures. However, the normality assumption is more commonly used for continuous measures, and the conventional method may be inaccurate when effect sizes are large or cutoff values for dichotomizing binary events are extreme (leading to rare events). This article proposes a Bayesian hierarchical model to synthesize SMDs and ORs without using the conventional conversion method. This model assumes exact likelihoods for continuous and binary outcome measures, which account for full uncertainties in the synthesized results. We performed simulation studies to compare the performance of the conventional and Bayesian methods in various settings. The Bayesian method generally produced less biased results with smaller mean squared errors and higher coverage probabilities than the conventional method in most cases. Nevertheless, this superior performance depended on the normality assumption for continuous measures; the Bayesian method could lead to nonignorable biases for non-normal data. In addition, we used two case studies to illustrate the proposed Bayesian method in real-world settings.

Methods for the Inclusion of Real-World Evidence in a Rare Events Meta-Analysis of Randomized Controlled Trials

BACKGROUND

Many rare events meta-analyses of randomized controlled trials (RCTs) have lower statistical power, and real-world evidence (RWE) is becoming widely recognized as a valuable source of evidence. The purpose of this study is to investigate methods for including RWE in a rare events meta-analysis of RCTs and the impact on the level of uncertainty around the estimates.

METHODS

Four methods for the inclusion of RWE in evidence synthesis were investigated by applying them to two previously published rare events meta-analyses: the naïve data synthesis (NDS), the design-adjusted synthesis (DAS), the use of RWE as prior information (RPI), and the three-level hierarchical models (THMs). We gauged the effect of the inclusion of RWE by varying the degree of confidence placed in RWE.

RESULTS

This study showed that the inclusion of RWE in a rare events meta-analysis of RCTs could increase the precision of the estimates, but this depended on the method of inclusion and the level of confidence placed in RWE. NDS cannot consider the bias of RWE, and its results may be misleading. DAS resulted in stable estimates for the two examples, regardless of whether we placed high- or low-level confidence in RWE. The results of the RPI approach were sensitive to the confidence level placed in RWE. The THM was effective in allowing for accommodating differences between study types, while it had a conservative result compared with other methods.

CONCLUSION

The inclusion of RWE in a rare events meta-analysis of RCTs could increase the level of certainty of the estimates and enhance the decision-making process. DAS might be appropriate for inclusion of RWE in a rare event meta-analysis of RCTs, but further evaluation in different scenarios of empirical or simulation studies is still warranted.

Meta-analysis methods for risk difference: A comparison of different models

Risk difference is a frequently-used effect measure for binary outcomes. In a meta-analysis, commonly-used methods to synthesize risk differences include: (1) the two-step methods that estimate study-specific risk differences first, then followed by the univariate common-effect model, fixed-effects model, or random-effects models; and (2) the one-step methods using bivariate random-effects models to estimate the summary risk difference from study-specific risks. These methods are expected to have similar performance when the number of studies is large and the event rate is not rare. However, studies with zero events are common in meta-analyses, and bias may occur with the conventional two-step methods from excluding zero-event studies or using an artificial continuity correction to zero events. In contrast, zero-event studies can be included and modeled by bivariate random-effects models in a single step. This article compares various methods to estimate risk differences in meta-analyses. Specifically, we present two case studies and three simulation studies to compare the performance of conventional two-step methods and bivariate random-effects models in the presence or absence of zero-event studies. In conclusion, we recommend researchers using bivariate random-effects models to estimate risk differences in meta-analyses, particularly in the presence of zero events.

An improved score-type confidence interval for stratified risk differences involving rare events

A stratified analysis of the differences in proportions has been widely employed in epidemiological research, social sciences, and drug development. It provides a useful framework for combining data across strata to produce a common effect. However, for rare events with incidence rates close to zero, popular confidence intervals for risk differences in a stratified analysis may not have appropriate coverage probabilities that approach the nominal confidence levels and the algorithms may fail to produce a valid confidence interval because of zero events in both the arms of a stratum. The main objective of this study is to evaluate the performance of certain methods commonly employed to construct confidence intervals for stratified risk differences when the response probabilities are close to a boundary value of zero or one. Additionally, we propose an improved stratified Miettinen-Nurminen confidence interval that exhibits a superior performance over standard methods while avoiding computational difficulties involving rare events. The proposed method can also be employed when the response probabilities are close to one.

Network meta-analysis of rare events using penalized likelihood regression

Network meta-analysis (NMA) of rare events has attracted little attention in the literature. Until recently, networks of interventions with rare events were analyzed using the inverse-variance NMA approach. However, when events are rare the normal approximations made by this model can be poor and effect estimates are potentially biased. Other methods for the synthesis of such data are the recent extension of the Mantel-Haenszel approach to NMA or the use of the noncentral hypergeometric distribution. In this article, we suggest a new common-effect NMA approach that can be applied even in networks of interventions with extremely low or even zero number of events without requiring study exclusion or arbitrary imputations. Our method is based on the implementation of the penalized likelihood function proposed by Firth for bias reduction of the maximum likelihood estimate to the logistic expression of the NMA model. A limitation of our method is that heterogeneity cannot be taken into account as an additive parameter as in most meta-analytical models. However, we account for heterogeneity by incorporating a multiplicative overdispersion term using a two-stage approach. We show through simulation that our method performs consistently well across all tested scenarios and most often results in smaller bias than other available methods. We also illustrate the use of our method through two clinical examples. We conclude that our "penalized likelihood NMA" approach is promising for the analysis of binary outcomes with rare events especially for networks with very few studies per comparison and very low control group risks.

Should studies with no events in both arms be excluded in evidence synthesis?

OBJECTIVES

In safety assessment, studies with no events are a frequent occurrence when conducting meta-analyses. The current approach in meta-analysis is to exclude double-zero studies from the synthesis. In this study, we compared the performance of excluding and including double-zero studies.

METHODS

A simulation with 5000 iterations was conducted based on the real-world dataset from Cochrane reviews. The true distribution of the rare events rather than normal distribution for the effects were used in the data generating mechanism to simulate aggregate meta-analysis data. We used Doi's inverse variance heterogeneity (IVhet) model for the meta-analyses with continuity correction (of 0.5) to include double-zero studies and used the odds ratio effect size. The performance of including versus excluding double-zero studies were then compared.

RESULTS

Generally, there was much larger mean squared error when double zero studies were excluded than when double-zero studies were included. The coverage when studies were excluded rapidly deteriorates as heterogeneity increased, while remained at or above the nominal level when double-zero studies were included. When there were very few double-zero studies, the performances was almost the same when including or excluding these studies. Subgroup analysis showed that, even for meta-analyses with unbalanced sample size across the two arms, including double-zero studies improved performance compared to when they were excluded.

CONCLUSIONS

Including double-zero studies in meta-analysis improved performance substantively when compared to excluding them, especially when the proportion of double-zero studies was large. Continuity correction with use of the IVhet model is therefore a good solution to deal with double-zero studies and should be considered in future meta-analyses.

Efficacy of secondary vs primary closure techniques for the prevention of postoperative complications after impacted mandibular third molar extractions: A systematic review update and meta-analysis

BACKGROUND

The aim of this systematic review was to determine whether secondary closure (SC) or primary closure (PC) is better at preventing postoperative complications after impacted mandibular third-molar extraction.

TYPES OF STUDIES REVIEWED

The authors sought randomized controlled trials comparing the effects of PC and SC on pain, swelling, trismus, infection, and bleeding after impacted mandibular third-molar extraction. Screening, data extraction, and risk of bias assessments were conducted independently and in duplicate. The reviewers pooled results across studies using a random-effects meta-analysis and assessed the certainty of the evidence using the Grading of Recommendations Assessment, Development and Evaluation approach.

RESULTS

This review identified 785 unique citations and included 40 trials. Compared with PC, SC was found to have trivial benefits for pain at day 7 and trismus within 1 week (moderate certainty). The incidence of infection and bleeding did not differ importantly between techniques (moderate certainty). However, SC is probably associated with less swelling on day 1 (standardized mean difference, -0.98; 95% CI, -1.22 to -0.73; moderate certainty) and day 3 (standardized mean difference, -0.87; 95% CI, -1.16 to -0.59; moderate certainty). There was very low certainty evidence for pain on days 1 and 3 and low certainty evidence for swelling on day 7.

CONCLUSIONS AND PRACTICAL IMPLICATIONS

Clinicians choosing between closure techniques should be aware that SC probably imparts an important benefit only for swelling at days 1 and 3. There seems to be a trivial difference between the techniques in other outcomes.

Synthesizing evidence from the earliest studies to support decision-making: To what extent could the evidence be reliable?

In evidence-based practice, new topics generally only have a few studies available for synthesis. As a result, the evidence of such meta-analyses raised substantial concerns. We investigated the robustness of the evidence from these earliest studies. Real-world data from the Cochrane Database of Systematic Reviews (CDSR) were collected. We emulated meta-analyses with the earliest 1 to 10 studies through cumulative meta-analysis from eligible meta-analyses. The magnitude and the direction of meta-analyses with the earliest few studies were compared to the full meta-analyses. From the CDSR, we identified 20,227 meta-analyses of binary outcomes and 7683 meta-analyses of continuous outcomes. Under the tolerable difference of 20% on the magnitude of the effects, the convergence proportion ranged from 24.24% (earliest 1 study) to 77.45% (earliest 10 studies) for meta-analyses of few earliest studies with binary outcomes. For meta-analyses of continuous outcomes, the convergence proportion ranged from 13.86% to 56.52%. In terms of the direction of the effects, even when only three studies were available at the earliest stage, the majority had the same direction as full meta-analyses; Only 19% for binary outcomes and 12% for continuous outcomes changed the direction as further evidence accumulated. Synthesizing evidence from the earliest studies is feasible to support urgent decision-making, and in most cases, the decisions would be reasonable. Considering the potential uncertainties, it is essential to evaluate the confidence of the evidence of these meta-analyses and update the evidence when necessary.

Methodological quality for systematic reviews of adverse events with surgical interventions: a cross-sectional survey

BACKGROUND

An increasing number of systematic reviews assessed the safety of surgical interventions over time. How well these systematic reviews were designed and conducted determines the reliability of evidence. In this study, we aimed to assess the methodological quality of systematic reviews on the safety of surgical interventions.

METHODS

We searched PubMed for systematic reviews of surgical interventions with safety as the exclusive outcome from 1st-Jan, 2015 to 1st-Jan, 2020. The methodological quality of eligible systematic reviews was evaluated according to the AMSTAR 2.0 instrument. The primary outcomes were the number of methodological weaknesses and the global methodological quality. The proportion of each methodological weakness among eligible systematic reviews was compared by three pre-defined stratification variables. The absolute difference of the proportion (PD) was used as the effect estimator, with the two-tailed z-test for the significance.

RESULTS

We identified 127 systematic reviews from 18,636 records. None (n = 0, 0.00%) of them could be rated as "high" in terms of the global methodological quality; in contrast, they were either rated as "low" (n = 18, 14.17%) or as "critically low" (n = 109, 85.83%). The median number of methodological weaknesses of these systematic reviews was 8 (interquartile range, IQR: 6 to 9), in which 4 (IQR: 2 to 4) were critical weaknesses. Systematic reviews that used any reporting guideline (e.g., domain 13, PD = -0.22, 95% CI: - 0.39, - 0.06; p = 0.01) and developed a protocol in advance (e.g., domain 6, PD = -0.20, 95% CI: - 0.39, - 0.01; p = 0.04) were less likely to have methodological weakness in some domains but not for the rest (e.g., domain 8, PD = 0.04, 95% CI: - 0.14, 0.21; p = 0.68; with protocol vs. without).

CONCLUSIONS

The methodological quality of current systematic reviews of adverse events with surgical interventions was poor. Further efforts, for example, encouraging researchers to develop a protocol in advance, are needed to enhance the methodological quality of these systematic reviews.

Rapid evidence synthesis approach for limits on the search date: How rapid could it be?

Rapid reviews have been widely employed to support timely decision-making, and limiting the search date is the most popular approach in published rapid reviews. We assessed the accuracy and workload of search date limits on the meta-analytical results to determine the best rapid strategy. The meta-analyses data were collected from the Cochrane Database of Systematic Reviews (CDSR). We emulated the rapid reviews by limiting the search date of the original CDSR to the recent 40, 35, 30, 25, 20, 15, 10, 7, 5, and 3 years, and their results were compared to the full meta-analyses. A random sample of 10% was drawn to repeat the literature search by the same timeframe limits to measure the relative workload reduction (RWR). The relationship between accuracy and RWR was established. We identified 21,363 meta-analyses of binary outcomes and 7683 meta-analyses of continuous outcomes from 2693 CDSRs. Our results suggested that under a maximum tolerance of 5% and 10% on the bias of magnitude, a limit on the recent 20 years can achieve good accuracy and at the same time save the most workload. Under the tolerance of 15% and 20% on the bias, a limit on the recent 10 years and 15 years could be considered. Limiting the search date is a valid rapid method to produce credible evidence for timely decisions. When conducting rapid reviews, researchers should consider both the accuracy and workload to make an appropriate decision.

Meta-computational techniques' for managing spare data: An application in off-pump heart surgery

BACKGROUND AND OBJECTIVES

This research looked at the key considerations to remember when selecting a model for working with sparse data. In the presence of sparse evidence, it proposes ideal conditions for conducting meta-analysis.

METHODS

Monte Carlo simulations were used to produce study results, and three forms of continuity correction were used in the research. Besides, meta-analytical approaches were used to measure the cumulative effect of treatment and estimate each method's efficiency. A clinical trial in off-pump surgery met the main objectives of this research. Meta-analysis methods were used to determine the outcome of postoperative risk results. After that, with a total population of 3030, Monte Carlo simulations were used to produce research data to run fixed and random-effect models with three continuity correction forms. The type of consistency adjustment used, group imbalances, statistical analysis used, and variance values between studies all affect meta-analytical methods' results.

RESULTS

MSE values for balanced groups are normally zero. While the Arc-sine variation approach does a decent job of coping with inconsistent results on the effect of treatment, it has concerns with boundary estimates of variance between tests. Furthermore, using continuity correction methods introduces bias and imprecise medication outcome calculations. The spectrum of statistical analysis, such as fixed effects and random effects, can be inferred as completely based on data in samples. The sensitivity analysis of correction decisions could increase the reliability of meta-analysis approaches by enabling researchers to analyze various effect estimation findings.

CONCLUSION

This research study can be expanded upon by identifying alternative approaches to continuity correction methods and resolving boundary estimate problems. The range of statistical analysis, such as fixed effects and random effects, can be entirely dependent on the samples' type of data. The sensitivity analysis of correction decisions could improve the efficiency of meta-analysis methods by allowing researchers to investigate a wide range of effect estimation results.

Synthesis of evidence from zero-events studies: A comparison of one-stage framework methods

In evidence synthesis, dealing with zero-events studies is an important and complicated task that has generated broad discussion. Numerous methods provide valid solutions to synthesizing data from studies with zero-events, either based on a frequentist or a Bayesian framework. Among frequentist frameworks, the one-stage methods have their unique advantages to deal with zero-events studies, especially for double-arm-zero-events. In this article, we give a concise overview of the one-stage frequentist methods. We conducted simulation studies to compare the statistical properties of these methods to the two-stage frequentist method (continuity correction) for meta-analysis with zero-events studies when double-zero-events studies were included. Our simulation studies demonstrated that the generalized estimating equation with unstructured correlation and beta-binomial method had the best performance among the one-stage methods. The random intercepts generalized linear mixed model showed good performance in the absence of obvious between-study variance. Our results also showed that the continuity correction with inverse-variance heterogeneous (IVhet) analytic model based on the two-stage framework had good performance when the between-study variance was obvious and the group size was balanced for included studies. In summary, the one-stage framework has unique advantages to deal with studies with zero events and is not susceptive to group size ratio. It should be considered in future meta-analyses whenever possible.

Meta-analysis with zero-event studies: a comparative study with application to COVID-19 data

BACKGROUND

Meta-analysis is a statistical method to synthesize evidence from a number of independent studies, including those from clinical studies with binary outcomes. In practice, when there are zero events in one or both groups, it may cause statistical problems in the subsequent analysis.

METHODS

In this paper, by considering the relative risk as the effect size, we conduct a comparative study that consists of four continuity correction methods and another state-of-the-art method without the continuity correction, namely the generalized linear mixed models (GLMMs). To further advance the literature, we also introduce a new method of the continuity correction for estimating the relative risk.

RESULTS

From the simulation studies, the new method performs well in terms of mean squared error when there are few studies. In contrast, the generalized linear mixed model performs the best when the number of studies is large. In addition, by reanalyzing recent coronavirus disease 2019 (COVID-19) data, it is evident that the double-zero-event studies impact the estimate of the mean effect size.

CONCLUSIONS

We recommend the new method to handle the zero-event studies when there are few studies in a meta-analysis, or instead use the GLMM when the number of studies is large. The double-zero-event studies may be informative, and so we suggest not excluding them.

Comparison of immunogenicity and safety of licensed Japanese encephalitis vaccines: A systematic review and network meta-analysis

INTRODUCTION

Annually more than 100,000 Japanese encephalitis (JE) cases and 25,000 deaths worldwide are caused by JE virus infection. More than 15 JE vaccines are currently in use worldwide. It is unknown whether any of the vaccines is superior to the others in terms of immunogenicity and safety.

METHODS

Four databases were systematically searched for randomised controlled trials that compared two or more types of JE vaccines. Vaccines were classified into four classes: inactivated mouse brain-derived (oldest class), inactivated Vero cell, live chimeric, and live attenuated. Network meta-analysis was used to generate mixed effect estimates against inactivated mouse brain-derived vaccines for seroconversion, and against placebo for adverse event (AE) and severe adverse event (SAE).

RESULTS

23 studies (38,496 participants) were included. All newer vaccine classes had better immunogenicity, the difference was statistically significant for inactivated Vero cell (OR = 2.98; 95 %CI: 1.02-8.65) and live chimeric (OR = 5.93; 95 %CI: 1.73-20.32) vaccines. Inactivated mouse-derived vaccines had the highest odds for AEs (OR = 2.27; 95 %CI: 1.59-3.23), the odds of AE of newer vaccines was not different to placebo. There was no difference in SAEs across vaccine classes.

CONCLUSIONS

All newer JE vaccines have comparable safety profiles, live chimeric and inactivated Vero cell vaccines are the most immunogenic among the newer vaccine classes.

Utilization of the evidence from studies with no events in meta-analyses of adverse events: an empirical investigation

BACKGROUNDS

Zero-events studies frequently occur in systematic reviews of adverse events, which consist of an important source of evidence. We aimed to examine how evidence of zero-events studies was utilized in the meta-analyses of systematic reviews of adverse events.

METHODS

We conducted a survey of systematic reviews published in two periods: January 1, 2015, to January 1, 2020, and January 1, 2008, to April 25, 2011. Databases were searched for systematic reviews that conducted at least one meta-analysis of any healthcare intervention and used adverse events as the exclusive outcome. An adverse event was defined as any untoward medical occurrence in a patient or subject in healthcare practice. We summarized the frequency of occurrence of zero-events studies in eligible systematic reviews and how these studies were dealt with in the meta-analyses of these systematic reviews.

RESULTS

We included 640 eligible systematic reviews. There were 406 (63.45%) systematic reviews involving zero-events studies in their meta-analyses, among which 389 (95.11%) involved single-arm-zero-events studies and 223 (54.93%) involved double-arm-zero-events studies. The majority (98.71%) of these systematic reviews incorporated single-arm-zero-events studies into the meta-analyses. On the other hand, the majority (76.23%) of them excluded double-arm-zero-events studies from the meta-analyses, of which the majority (87.06%) did not discuss the potential impact of excluding such studies. Systematic reviews published at present (2015-2020) tended to incorporate zero-events studies in meta-analyses than those published in the past (2008-2011), but the difference was not significant (proportion difference=-0.09, 95% CI -0.21 to 0.03, p = 0.12).

CONCLUSION

Systematic review authors routinely treated studies with zero-events in both arms as "non-informative" carriers and excluded them from their reviews. Whether studies with no events are "informative" or not largely depends on the methods and assumptions applied, thus sensitivity analyses using different methods should be considered in future meta-analyses.

Association between a soy-based infant diet and the onset of puberty: A systematic review and meta-analysis

The objective of this systematic review was to evaluate the association between a soy-based infant diet and the onset of puberty. We included studies in which children were fed a soy-based diet, and we compared them with those who were not. The primary outcomes were the onset of puberty in girls (thelarche, pubarche, and menarche age), boys (pubarche, voice change, testicular and penis enlargement age), and both (risk of delayed and precocious puberty [PP]). Search strategies were performed in PubMed, Embase, LILACS, and CENTRAL databases. Two reviewers selected eligible studies, assessed the risk of bias, and extracted data from the included studies. The odds ratio (OR) and mean difference (MD) were calculated with a 95% confidence interval (CI) as a measure of the association between soy consumption and outcomes. We used a random-effects model to pool results across studies and the Grading of Recommendations Assessment, Development, and Evaluation to evaluate the certainty of evidence. We included eight studies in which 598 children consumed a soy-based diet but 2957 did not. The primary outcomes that could be plotted in the meta-analysis were the risk of PP and age at menarche. There was no statistical difference between groups for PP (OR: 0.51, 95% CI: 0.09 to 2.94, 3 studies, 206 participants, low certainty of evidence). No between-group difference was observed in menarche age (MD 0.14 years, 95% CI -0.16 to 0.45, 3 studies, 605 children, low certainty of evidence). One study presented this outcome in terms of median and interquartile range, and although the onset of menarche was marginally increased in girls who received a soy-based diet, the reported age was within the normal age range for menarche. We did not find any association between a soy-based infant diet and the onset of puberty in boys or girls.

Trial Registration: PROSPERO registration: CRD42018088902.

A proposed framework to guide evidence synthesis practice for meta-analysis with zero-events studies

OBJECTIVE

In evidence synthesis practice, researchers often face the problem of how to deal with zero-events. Inappropriately dealing with zero-events studies may lead to research waste and mislead healthcare practice. We propose a framework to guide researchers to better deal with zero-events in meta-analysis.

STUDY DESIGN AND SETTING

We used two dimensions, one with respect to the total events count across all studies in the comparative arms in a meta-analysis, and a second with respect to whether included studies have single or both arms with zero-events, to establish the framework for the classification of meta-analysis with zero-events studies. A dataset from Cochrane systematic reviews was used to evaluate the classification.

RESULTS

The proposed framework classifies meta-analysis with zero-events studies into six subtypes. The classification matched well to the large real-world dataset. The applicability of existing methods for zero-events were then presented under each meta-analysis subtype based on this framework, with a 5-step principle to help researchers in evidence synthesis practice.

CONCLUSIONS

The proposed framework should be considered by researchers when making decisions on the selection of the synthesis methods in a meta-analysis. It also provides a reasonable basis for the development of methodological guidelines to deal with zero-events in meta-analysis.

Many meta-analyses of rare events in the Cochrane Database of Systematic Reviews were underpowered

BACKGROUND AND OBJECTIVE

Meta-analysis is a statistical method with the ability to increase the power for statistical inference, while it may still face the problem of being underpowered. In this study, we investigated the power to detect certain true effects for published meta-analyses of rare events.

METHODS

We extracted data from the Cochrane Database of Systematic Reviews for meta-analyses of rare events from January 2003 to May 2018. We retrospectively estimated the power to detect a 10-50% relative risk reduction (RRR) of eligible meta-analyses. The proportion of meta-analyses achieved a sufficient power (≥0.8) were estimated.

RESULTS

We identified 4,177 meta-analyses. The median power to detect 10%, 30%, and 50% RRR were 0.06 (interquartile range [IQR]: 0.05 to 0.06), 0.08 (IQR: 0.06 to 0.15), and 0.17 (IQR: 0.10 to 0.42), respectively); the corresponding proportion of meta-analyses that reached sufficient power were 0.32%, 3.68%, and 11.81%. Meta-analyses incorporating data from more studies had higher probability to achieve a sufficient power (rate ratio = 2.49, 95% CI: 1.76, 3.52, P < 0.001).

CONCLUSION

Most of the meta-analyses of rare events in Cochrane systematic reviews were underpowered. Future meta-analysis of rare events should report the power of the results to support informative conclusions.

Arcsine-based transformations for meta-analysis of proportions: Pros, cons, and alternatives

Meta-analyses have been increasingly used to synthesize proportions (eg, disease prevalence) from multiple studies in recent years. Arcsine-based transformations, especially the Freeman-Tukey double-arcsine transformation, are popular tools for stabilizing the variance of each study's proportion in two-step meta-analysis methods. Although they offer some benefits over the conventional logit transformation, they also suffer from several important limitations (eg, lack of interpretability) and may lead to misleading conclusions. Generalized linear mixed models and Bayesian models are intuitive one-step alternative approaches, and can be readily implemented via many software programs. This article explains various pros and cons of the arcsine-based transformations, and discusses the alternatives that may be generally superior to the currently popular practice.

Double-zero-event studies matter: a re-evaluation of physical distancing, face masks, and eye protection for preventing person-to-person transmission of COVID-19 and its policy impact

OBJECTIVES

High-quality meta-analyses on COVID-19 are in urgent demand for evidence-based decision making. However, conventional approaches exclude double-zero-event studies (DZS) from meta-analyses. We assessed whether including such studies impacts the conclusions in a recent systematic urgent review on prevention measures for preventing person-to-person transmission of COVID-19. Study designs and settings: We extracted data for meta-analyses containing DZS from a recent review that assessed the effects of physical distancing, face masks, and eye protection for preventing person-to-person transmission. A bivariate generalized linear mixed model was used to re-do the meta-analyses with DZS included. We compared the synthesized relative risks (RRs) of the three prevention measures, their 95% confidence intervals (CI), and significance tests (at the level of 0.05) including and excluding DZS.

RESULTS

The re-analyzed COVID-19 data containing DZS involved a total of 1,784 participants who were not considered in the original review. Including DZS noticeably changed the synthesized RRs and 95% CIs of several interventions. For the meta-analysis of the effect of physical distancing, the RR of COVID-19 decreased from 0.15 (95% CI, 0.03 to 0.73) to 0.07 (95% CI, 0.01 to 0.98). For several meta-analyses, the statistical significance of the synthesized RR was changed. The RR of eye protection with a physical distance of 2 m and the RR of physical distancing when using N95 respirators were no longer statistically significant after including DZS.

CONCLUSIONS

DZS may contain useful information. Sensitivity analyses that include DZS in meta-analysis are recommended.

Laplace approximation, penalized quasi-likelihood, and adaptive Gauss-Hermite quadrature for generalized linear mixed models: towards meta-analysis of binary outcome with sparse data

BACKGROUND

In meta-analyses of a binary outcome, double zero events in some studies cause a critical methodology problem. The generalized linear mixed model (GLMM) has been proposed as a valid statistical tool for pooling such data. Three parameter estimation methods, including the Laplace approximation (LA), penalized quasi-likelihood (PQL) and adaptive Gauss-Hermite quadrature (AGHQ) were frequently used in the GLMM. However, the performance of GLMM via these estimation methods is unclear in meta-analysis with zero events.

METHODS

A simulation study was conducted to compare the performance. We fitted five random-effects GLMMs and estimated the results through the LA, PQL and AGHQ methods, respectively. Each scenario conducted 20,000 simulation iterations. The data from Cochrane Database of Systematic Reviews were collected to form the simulation settings. The estimation methods were compared in terms of the convergence rate, bias, mean square error, and coverage probability.

RESULTS

Our results suggested that when the total events were insufficient in either of the arms, the GLMMs did not show good point estimation to pool studies of rare events. The AGHQ method did not show better properties than the LA estimation in terms of convergence rate, bias, coverage, and possibility to produce very large odds ratios. In addition, although the PQL had some advantages, it was not the preferred option due to its low convergence rate in some situations, and the suboptimal point and variance estimation compared to the LA.

CONCLUSION

The GLMM is an alternative for meta-analysis of rare events and is especially useful in the presence of zero-events studies, while at least 10 total events in both arms is recommended when employing GLMM for meta-analysis. The penalized quasi-likelihood and adaptive Gauss-Hermite quadrature are not superior to the Laplace approximation for rare events and thus they are not recommended.