LFK index does not reliably detect small-study effects in meta-analysis: A simulation study

The LFK index has been promoted as an improved method to detect bias in meta-analysis. Putatively, its performance does not depend on the number of studies in the meta-analysis. We conducted a simulation study, comparing the LFK index test to three standard tests for funnel plot asymmetry in settings with smaller or larger group sample sizes. In general, false positive rates of the LFK index test markedly depended on the number and size of studies as well as the between-study heterogeneity with values between 0% and almost 30%. Egger's test adhered well to the pre-specified significance level of 5% under homogeneity, but was too liberal (smaller groups) or conservative (larger groups) under heterogeneity. The rank test was too conservative for most simulation scenarios. The Thompson-Sharp test was too conservative under homogeneity, but adhered well to the significance level in case of heterogeneity. The true positive rate of the LFK index test was only larger compared with classic tests if the false positive rate was inflated. The power of classic tests was similar or larger than the LFK index test if the false positive rate of the LFK index test was used as significance level for the classic tests. Under ideal conditions, the false positive rate of the LFK index test markedly and unpredictably depends on the number and sample size of studies as well as the extent of between-study heterogeneity. The LFK index test in its current implementation should not be used to assess funnel plot asymmetry in meta-analysis.

Benchmarking survival outcomes: A funnel plot for survival data

Benchmarking is commonly used in many healthcare settings to monitor clinical performance, with the aim of increasing cost-effectiveness and safe care of patients. The funnel plot is a popular tool in visualizing the performance of a healthcare center in relation to other centers and to a target, taking into account statistical uncertainty. In this paper, we develop a methodology for constructing funnel plots for survival data. The method takes into account censoring and can deal with differences in censoring distributions across centers. Practical issues in implementing the methodology are discussed, particularly in the setting of benchmarking clinical outcomes for hematopoietic stem cell transplantation. A simulation study is performed to assess the performance of the funnel plots under several scenarios. Our methodology is illustrated using data from the European Society for Blood and Marrow Transplantation benchmarking project.

Process Design of Continuous Powder Blending Using Residence Time Distribution and Feeding Models

The present paper reports a thorough continuous powder blending process design of acetylsalicylic acid (ASA) and microcrystalline cellulose (MCC) based on the Process Analytical Technology (PAT) guideline. A NIR-based method was applied using multivariate data analysis to achieve in-line process monitoring. The process dynamics were described with residence time distribution (RTD) models to achieve deep process understanding. The RTD was determined using the active pharmaceutical ingredient (API) as a tracer with multiple designs of experiment (DoE) studies to determine the effect of critical process parameters (CPPs) on the process dynamics. To achieve quality control through material diversion from feeding data, soft sensor-based process control tools were designed using the RTD model. The operation block model of the system was designed to select feasible experimental setups using the RTD model, and feeder characterizations as digital twins, therefore visualizing the output of theoretical setups. The concept significantly reduces the material and instrumental costs of process design and implementation.

Graphical augmentations to sample-size-based funnel plot in meta-analysis

Assessing publication bias is a critical procedure in meta-analyses for rating the synthesized overall evidence. Because statistical tests for publication bias are usually not powerful and only give P values that inform either the presence or absence of the bias, examining the asymmetry of funnel plots has been popular to investigate potentially missing studies and the direction of the bias. Most funnel plots present treatment effects against their standard errors, and the contours depicting studies' significance levels have been used in the plots to distinguish publication bias from other factors (such as heterogeneity and subgroup effects) that may cause the plots' asymmetry. However, treatment effects and their standard errors are frequently associated even if no publication bias exists (eg, both variables depend on the four data cells in a 2 × 2 table for the odds ratio), so standard-error-based funnel plots may lead to false positive conclusions when such association may not be negligible. In addition, the missingness of studies may relate to their sample sizes besides P values (which are partly determined by standard errors); studies with more samples are more likely published. Therefore, funnel plots based on sample sizes can be an alternative tool. However, the contours for standard-error-based funnel plots cannot be directly applied to sample-size-based ones. This article introduces contours for sample-size-based funnel plots of various effect sizes, which may help meta-analysts properly interpret such plots' asymmetry. We provide five examples to illustrate the use of the proposed contours.

Empirical Comparison of Publication Bias Tests in Meta-Analysis

BACKGROUND

Decision makers rely on meta-analytic estimates to trade off benefits and harms. Publication bias impairs the validity and generalizability of such estimates. The performance of various statistical tests for publication bias has been largely compared using simulation studies and has not been systematically evaluated in empirical data.

METHODS

This study compares seven commonly used publication bias tests (i.e., Begg's rank test, trim-and-fill, Egger's, Tang's, Macaskill's, Deeks', and Peters' regression tests) based on 28,655 meta-analyses available in the Cochrane Library.

RESULTS

Egger's regression test detected publication bias more frequently than other tests (15.7% in meta-analyses of binary outcomes and 13.5% in meta-analyses of non-binary outcomes). The proportion of statistically significant publication bias tests was greater for larger meta-analyses, especially for Begg's rank test and the trim-and-fill method. The agreement among Tang's, Macaskill's, Deeks', and Peters' regression tests for binary outcomes was moderately strong (most κ's were around 0.6). Tang's and Deeks' tests had fairly similar performance (κ > 0.9). The agreement among Begg's rank test, the trim-and-fill method, and Egger's regression test was weak or moderate (κ < 0.5).

CONCLUSIONS

Given the relatively low agreement between many publication bias tests, meta-analysts should not rely on a single test and may apply multiple tests with various assumptions. Non-statistical approaches to evaluating publication bias (e.g., searching clinical trials registries, records of drug approving agencies, and scientific conference proceedings) remain essential.

The use of funnel plots with regression as a tool to visually compare HIV treatment outcomes between centres adjusting for patient characteristics and size: a UK Collaborative HIV Cohort study

Objectives

A measure used for assessing the effectiveness of HIV care and comparing clinical centres is the proportion of people starting antiretroviral therapy (ART) with viral suppression (VS) after 1 year. We propose a method that adjusts for patients’ demographic characteristics, and visually compares this measure between different sites accounting for centre size.

Methods

We analysed viral load measurements for UK Collaborative HIV Cohort (UK CHIC) patients starting ART between 2006 and 2013. We used logistic regression to estimate the proportion with VS after 1 year of ART adjusted for patient mix (in terms of age and a combined gender/ethnicity/acquisition mode variable) and calendar year. We compared outcomes between centres using funnel plots which account for centre size.

Results

The overall proportion of the cohort with VS 1 year after starting ART was 90% and increased from 83% to 93% between 2006 and 2013. VS was lower in younger individuals. White men who have sex with men (MSM) had the highest (94%), and black African (81%) and white (82%) heterosexual women the lowest proportions achieving VS. Comparing the unadjusted funnel plot with the adjusted, there were movements of some centres from outside to inside the 95% contour limits, which was largely explained by the patient mix of these centres.

Conclusions

VS 1 year after ART start was associated with demographic characteristics and centre size; therefore, to compare the performances of centres, adjustment for these factors is required. Adjusted funnel plot is an effective tool which accounts for both the demographic characteristics and the centre size. Social factors, rather than treatment decisions within the control of the centres, may drive differences in outcomes.

Detecting small-study effects and funnel plot asymmetry in meta-analysis of survival data: A comparison of new and existing tests

Small‐study effects are a common threat in systematic reviews and may indicate publication bias. Their existence is often verified by visual inspection of the funnel plot. Formal tests to assess the presence of funnel plot asymmetry typically estimate the association between the reported effect size and their standard error, the total sample size, or the inverse of the total sample size. In this paper, we demonstrate that the application of these tests may be less appropriate in meta‐analysis of survival data, where censoring influences statistical significance of the hazard ratio. We subsequently propose 2 new tests that are based on the total number of observed events and adopt a multiplicative variance component. We compare the performance of the various funnel plot asymmetry tests in an extensive simulation study where we varied the true hazard ratio (0.5 to 1), the number of published trials (N=10 to 100), the degree of censoring within trials (0% to 90%), and the mechanism leading to participant dropout (noninformative versus informative). Results demonstrate that previous well‐known tests for detecting funnel plot asymmetry suffer from low power or excessive type‐I error rates in meta‐analysis of survival data, particularly when trials are affected by participant dropout. Because our novel test (adopting estimates of the asymptotic precision as study weights) yields reasonable power and maintains appropriate type‐I error rates, we recommend its use to evaluate funnel plot asymmetry in meta‐analysis of survival data. The use of funnel plot asymmetry tests should, however, be avoided when there are few trials available for any meta‐analysis.

Standardized mean differences cause funnel plot distortion in publication bias assessments

Meta-analyses are increasingly used for synthesis of evidence from biomedical research, and often include an assessment of publication bias based on visual or analytical detection of asymmetry in funnel plots. We studied the influence of different normalisation approaches, sample size and intervention effects on funnel plot asymmetry, using empirical datasets and illustrative simulations. We found that funnel plots of the Standardized Mean Difference (SMD) plotted against the standard error (SE) are susceptible to distortion, leading to overestimation of the existence and extent of publication bias. Distortion was more severe when the primary studies had a small sample size and when an intervention effect was present. We show that using the Normalised Mean Difference measure as effect size (when possible), or plotting the SMD against a sample size-based precision estimate, are more reliable alternatives. We conclude that funnel plots using the SMD in combination with the SE are unsuitable for publication bias assessments and can lead to false-positive results.

A clinical prediction model for prolonged air leak after pulmonary resection

OBJECTIVE

Prolonged air leak increases costs and worsens outcomes after pulmonary resection. We aimed to develop a clinical prediction tool for prolonged air leak using pretreatment and intraoperative variables.

METHODS

Patients who underwent pulmonary resection for lung cancer/nodules (from January 2009 to June 2014) were stratified by prolonged parenchymal air leak (>5 days). Using backward stepwise logistic regression with bootstrap resampling for internal validation, candidate variables were identified and a nomogram risk calculator was developed.

RESULTS

A total of 2317 patients underwent pulmonary resection for lung cancer/nodules. Prolonged air leak (8.6%, n = 200) was associated with significantly longer hospital stay (median 10 vs 4 days; P < .001). Final model variables associated with increased risk included low percent forced expiratory volume in 1 second, smoking history, bilobectomy, higher annual surgeon caseload, previous chest surgery, Zubrod score >2, and interaction terms for right-sided thoracotomy and wedge resection by thoracotomy. Wedge resection, higher body mass index, and unmeasured percent forced expiratory volume in 1 second were protective. Derived nomogram discriminatory accuracy was 76% (95% confidence interval [CI], 0.72-0.79) and facilitated patient stratification into low-, intermediate- and high-risk groups with monotonic increase in observed prolonged air leaks (2.0%, 8.9%, and 19.2%, respectively; P < .001). Patients at intermediate and high risk were 4.80 times (95% CI, 2.86-8.07) and 11.86 times (95% CI, 7.21-19.52) more likely to have prolonged air leak compared with patients at low risk.

CONCLUSIONS

Using readily available candidate variables, our nomogram predicts increasing risk of prolonged air leak with good discriminatory ability. Risk stratification can support surgical decision making, and help initiate proactive, patient-specific surgical management.

Community-acquired pneumonia (CAP) hospitalizations and deaths: is there a role for quality improvement through inter-hospital comparisons?

OBJECTIVE

To assess between-hospital variations in standardized in-hospital mortality ratios of community-acquired pneumonia (CAP), and identify possible leads for quality improvement.

DESIGN

We used an administrative database to estimate standardized in-hospital mortality ratios for 111 Belgian hospitals, by carrying out a set of hierarchical logistic regression models, intended to disentangle therapeutic attitudes and biases. To facilitate the detection of false-negative/positive results, we added an inconclusive zone to the funnel plots, derived from the results of the study. Data quality was validated by comparison with (i) alternative data from the largest Belgian Sickness Fund, (ii) published German hospital data and (iii) the results of an on-site audit.

SETTING

All Belgian hospital discharge records from 2004 to 2007.

STUDY PARTICIPANTS

A total of 111 776 adult patients were admitted for CAP.

MAIN OUTCOME MEASURE

Risk-adjusted standardized in-hospital mortality ratios.

RESULTS

Out of the 111 hospitals, we identified five and six outlying hospitals, with standardized mortality ratios of CAP consistently on the extremes of the distribution, as providing possibly better or worse care, respectively, and 18 other hospitals as having possible quality weaknesses/strengths. At the individuals' level of the analysis, adjusted odds ratios showed the paramount importance of old age, comorbidity and mechanical ventilation. The data compared well with the different validation sources.

CONCLUSIONS

Despite the limitations inherent to administrative data, it seemed possible to establish inter-hospital differences in standardized in-hospital mortality ratios of CAP and to identify leads for quality improvement. Monitoring is needed to assess progress in quality.

Surgeon-level reporting presented by funnel plot is understood by doctors but inaccurately interpreted by members of the public

INTRODUCTION

Risk-adjusted outcome data for general surgeons practicing in the United Kingdom were published for the first time in 2013 with the aim of increasing transparency, improving standards, and providing the public with information to aid decision making. Most specialties used funnel plots to present their data. We assess the ability of members of the public (MoP), medical students, nonsurgical doctors (NSD), and surgeons to understand risk-adjusted surgical outcome data.

MATERIAL AND METHODS

A fictitious outcome dataset was created and presented in the form of a funnel plot to 10 participants from each of the aforementioned group. Standard explanatory text was provided. Each participant was given 5 minutes to review the funnel plot and complete a questionnaire. For each question, there was only 1 correct answer.

RESULTS

Completion rate was 100% (n = 40). No difference existed between NSD and surgeons. A significant difference for identification of the "worst performing surgeon" was noted between surgeons and MoP (p < 0.01) and between NSD and MoP (p < 0.01). Half of medical students and MoP claimed they would use this information to aid decision making compared with 80% of doctors. MoP reported the funnel plot significantly "more difficult" to interpret than surgeons did (p < 0.01) and NSD (p < 0.01).

CONCLUSIONS

MoP found these data significantly more "difficult to understand" and were less likely to both spot "outliers" and use this data to inform decisions than doctors. Surgeons should be aware that outcome data may require an alternative method of presentation to be understood by MoP.

Level-adjusted funnel plots based on predicted marginal expectations: an application to prophylactic antibiotics in gallstone surgery

Funnel plots are widely used to visualize grouped data, for example, in institutional comparison. This paper extends the concept to a multi-level setting, displaying one level at a time, adjusted for the other levels, as well as for covariates at all levels. These level-adjusted funnel plots are based on a Markov chain Monte Carlo fit of a random effects model, translating the estimated model parameters to predicted marginal expectations. Working within the estimation framework, we accommodate outlying institutions using heavy-tailed random effects distributions. We also develop computer-efficient methods to compute predicted probabilities in the case of dichotomous outcome data and various random effect distributions. We apply the method to a data set on prophylactic antibiotics in gallstone surgery.

Funnel plot control limits to identify poorly performing healthcare providers when there is uncertainty in the value of the benchmark

There is an increasing use of statistical methods, such as funnel plots, to identify poorly performing healthcare providers. Funnel plots comprise the construction of control limits around a benchmark and providers with outcomes falling outside the limits are investigated as potential outliers. The benchmark is usually estimated from observed data but uncertainty in this estimate is usually ignored when constructing control limits. In this paper, the use of funnel plots in the presence of uncertainty in the value of the benchmark is reviewed for outcomes from a Binomial distribution. Two methods to derive the control limits are shown: (i) prediction intervals; (ii) tolerance intervals Tolerance intervals formally include the uncertainty in the value of the benchmark while prediction intervals do not. The probability properties of 95% control limits derived using each method were investigated through hypothesised scenarios. Neither prediction intervals nor tolerance intervals produce funnel plot control limits that satisfy the nominal probability characteristics when there is uncertainty in the value of the benchmark. This is not necessarily to say that funnel plots have no role to play in healthcare, but that without the development of intervals satisfying the nominal probability characteristics they must be interpreted with care.

Funnel plots for population-based cancer survival: principles, methods and applications

Funnel plots are graphical tools designed to detect excessive variation in performance indicators by simple visual inspection of the data. Their main use in the biomedical domain so far has been to detect publication bias in meta-analyses, but they have also been recommended as the most appropriate way to display performance indicators for a vast range of health-related outcomes. Here, we extend the use of funnel plots to population-based cancer survival and several related measures. We present three applications to familiarise the reader with their interpretation. We propose funnel plots for various cancer survival measures, as well as age-standardised survival, trends in survival and excess hazard ratios. We describe the components of a funnel plot and the formulae for the construction of the control limits for each of these survival measures. We include three transformations to construct the control limits for the survival function: complementary log-log, logit and logarithmic transformations. We present applications of funnel plots to explore the following: (i) small-area and temporal variation in cancer survival; (ii) racial and geographical variation in cancer survival; and (iii) geographical variation in the excess hazard of death. Funnel plots provide a simple and informative graphical tool to display geographical variation and trend in a range of cancer survival measures. We recommend their use as a routine instrument for cancer survival comparisons, to inform health policy makers in planning and assessing cancer policies. We advocate the use of the complementary log-log or logit transformation to construct the control limits for the survival function.

The probability of being identified as an outlier with commonly used funnel plot control limits for the standardised mortality ratio

BACKGROUND

Emphasis is increasingly being placed on the monitoring of clinical outcomes for health care providers. Funnel plots have become an increasingly popular graphical methodology used to identify potential outliers. It is assumed that a provider only displaying expected random variation (i.e. 'in-control') will fall outside a control limit with a known probability. In reality, the discrete count nature of these data, and the differing methods, can lead to true probabilities quite different from the nominal value. This paper investigates the true probability of an 'in control' provider falling outside control limits for the Standardised Mortality Ratio (SMR).

METHODS

The true probabilities of an 'in control' provider falling outside control limits for the SMR were calculated and compared for three commonly used limits: Wald confidence interval; 'exact' confidence interval; probability-based prediction interval.

RESULTS

The probability of falling above the upper limit, or below the lower limit, often varied greatly from the nominal value. This was particularly apparent when there were a small number of expected events: for expected events ≤ 50 the median probability of an 'in-control' provider falling above the upper 95% limit was 0.0301 (Wald), 0.0121 ('exact'), 0.0201 (prediction).

CONCLUSIONS

It is important to understand the properties and probability of being identified as an outlier by each of these different methods to aid the correct identification of poorly performing health care providers. The limits obtained using probability-based prediction limits have the most intuitive interpretation and their properties can be defined a priori. Funnel plot control limits for the SMR should not be based on confidence intervals.

Using network meta-analysis to evaluate the existence of small-study effects in a network of interventions

Suggested methods for exploring the presence of small-study effects in a meta-analysis and the possibility of publication bias are associated with important limitations. When a meta-analysis comprises only a few studies, funnel plots are difficult to interpret, and regression-based approaches to test and account for small-study effects have low power. Assuming that the cause of funnel plot asymmetry is likely to affect an entire research field rather than only a particular comparison of interventions, we suggest that network meta-regression is employed to account for small-study effects in a set of related meta-analyses. We present several possible models for the direction and distribution of small-study effects and we describe the methods by re-analysing two published networks. Copyright © 2012 John Wiley & Sons, Ltd.

Graphical displays for meta-analysis: An overview with suggestions for practice

Meta-analyses are fundamental tools for collating and synthesizing large amounts of information, and graphical displays have become the principal tool for presenting the results of multiple studies of the same research question. We review standard and proposed graphical displays for presentation of meta-analytic data, and offer our recommendations on how they might be presented to provide the most useful and user-friendly illustrations. We concentrate on graphs that specifically aim to present similar sorts of univariate results from multiple studies. We start with forest plots and funnel plots, and proceed to Galbraith (or radial) plots, L'Abbé (and related) plots, further plots useful for investigating heterogeneity, plots useful for model diagnostics and plots for illustrating likelihoods and Bayesian meta-analyses. Copyright © 2010 John Wiley & Sons, Ltd.

Dealing with publication bias in translational stroke research

Publication bias has been around for about 50 years. It has become a concern for almost 20 years in the medical research community. This review briefly summarizes the current status of publication bias, potential sources where bias may arise from, and its common evaluation methods. In the field of translational stroke research, publication bias has long been suspected; however, it has not been addressed with sufficient efforts. Its status has remained the same during the last decade. The author emphasizes the important role that publishers might play in addressing publication bias.