Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study

The performance of different propensity score methods for estimating marginal hazard ratios

Propensity score methods are increasingly being used to reduce or minimize the effects of confounding when estimating the effects of treatments, exposures, or interventions when using observational or non-randomized data. Under the assumption of no unmeasured confounders, previous research has shown that propensity score methods allow for unbiased estimation of linear treatment effects (e.g., differences in means or proportions). However, in biomedical research, time-to-event outcomes occur frequently. There is a paucity of research into the performance of different propensity score methods for estimating the effect of treatment on time-to-event outcomes. Furthermore, propensity score methods allow for the estimation of marginal or population-average treatment effects. We conducted an extensive series of Monte Carlo simulations to examine the performance of propensity score matching (1:1 greedy nearest-neighbor matching within propensity score calipers), stratification on the propensity score, inverse probability of treatment weighting (IPTW) using the propensity score, and covariate adjustment using the propensity score to estimate marginal hazard ratios. We found that both propensity score matching and IPTW using the propensity score allow for the estimation of marginal hazard ratios with minimal bias. Of these two approaches, IPTW using the propensity score resulted in estimates with lower mean squared error when estimating the effect of treatment in the treated. Stratification on the propensity score and covariate adjustment using the propensity score result in biased estimation of both marginal and conditional hazard ratios. Applied researchers are encouraged to use propensity score matching and IPTW using the propensity score when estimating the relative effect of treatment on time-to-event outcomes.

Treatment of patent ductus arteriosus and neonatal mortality/morbidities: adjustment for treatment selection bias

OBJECTIVE

To examine the association between treatment for patent ductus arteriosus (PDA) and neonatal outcomes in preterm infants, after adjustment for treatment selection bias.

STUDY DESIGN

Secondary analyses were conducted using data collected by the Canadian Neonatal Network for neonates born at a gestational age ≤ 32 weeks and admitted to neonatal intensive care units in Canada between 2004 and 2008. Infants who had PDA and survived beyond 72 hours were included in multivariable logistic regression analyses that compared mortality or any severe neonatal morbidity (intraventricular hemorrhage grades ≥ 3, retinopathy of prematurity stages ≥ 3, bronchopulmonary dysplasia, or necrotizing enterocolitis stages ≥ 2) between treatment groups (conservative management, indomethacin only, surgical ligation only, or both indomethacin and ligation). Propensity scores (PS) were estimated for each pair of treatment comparisons, and used in PS-adjusted and PS-matched analyses.

RESULTS

Among 3556 eligible infants with a diagnosis of PDA, 577 (16%) were conservatively managed, 2026 (57%) received indomethacin only, 327 (9%) underwent ligation only, and 626 (18%) were treated with both indomethacin and ligation. All multivariable and PS-based analyses detected significantly higher mortality/morbidities for surgically ligated infants, irrespective of prior indomethacin treatment (OR ranged from 1.25-2.35) compared with infants managed conservatively or those who received only indomethacin. No significant differences were detected between infants treated with only indomethacin and those managed conservatively.

CONCLUSIONS

Surgical ligation of PDA in preterm neonates was associated with increased neonatal mortality/morbidity in all analyses adjusted for measured confounders that attempt to account for treatment selection bias.

Do observational studies using propensity score methods agree with randomized trials? A systematic comparison of studies on acute coronary syndromes

AIMS

Randomized controlled trials (RCTs) are the gold standard for assessing the efficacy of therapeutic interventions because randomization protects from biases inherent in observational studies. Propensity score (PS) methods, proposed as a potential solution to confounding of the treatment-outcome association, are widely used in observational studies of therapeutic interventions for acute coronary syndromes (ACS). We aimed to systematically assess agreement between observational studies using PS methods and RCTs on therapeutic interventions for ACS.

METHODS AND RESULTS

We searched for observational studies of interventions for ACS that used PS methods to estimate treatment effects on short- or long-term mortality. Using a standardized algorithm, we matched observational studies to RCTs based on patients' characteristics, interventions, and outcomes ('topics'), and we compared estimates of treatment effect between the two designs. When multiple observational studies or RCTs were identified for the same topic, we performed a meta-analysis and used the summary relative risk for comparisons. We matched 21 observational studies investigating 17 distinct clinical topics to 63 RCTs (median = 3 RCTs per observational study) for short-term (7 topics) and long-term (10 topics) mortality. Estimates from PS analyses differed statistically significantly from randomized evidence in two instances; however, observational studies reported more extreme beneficial treatment effects compared with RCTs in 13 of 17 instances (P = 0.049). Sensitivity analyses limited to large RCTs, and using alternative meta-analysis models yielded similar results.

CONCLUSION

For the treatment of ACS, observational studies using PS methods produce treatment effect estimates that are of more extreme magnitude compared with those from RCTs, although the differences are rarely statistically significant.

Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies

In a study comparing the effects of two treatments, the propensity score is the probability of assignment to one treatment conditional on a subject's measured baseline covariates. Propensity-score matching is increasingly being used to estimate the effects of exposures using observational data. In the most common implementation of propensity-score matching, pairs of treated and untreated subjects are formed whose propensity scores differ by at most a pre-specified amount (the caliper width). There has been a little research into the optimal caliper width. We conducted an extensive series of Monte Carlo simulations to determine the optimal caliper width for estimating differences in means (for continuous outcomes) and risk differences (for binary outcomes). When estimating differences in means or risk differences, we recommend that researchers match on the logit of the propensity score using calipers of width equal to 0.2 of the standard deviation of the logit of the propensity score. When at least some of the covariates were continuous, then either this value, or one close to it, minimized the mean square error of the resultant estimated treatment effect. It also eliminated at least 98% of the bias in the crude estimator, and it resulted in confidence intervals with approximately the correct coverage rates. Furthermore, the empirical type I error rate was approximately correct. When all of the covariates were binary, then the choice of caliper width had a much smaller impact on the performance of estimation of risk differences and differences in means. Copyright © 2010 John Wiley & Sons, Ltd.

Evaluation of the propensity score methods for estimating marginal odds ratios in case of small sample size

BACKGROUND

Propensity score (PS) methods are increasingly used, even when sample sizes are small or treatments are seldom used. However, the relative performance of the two mainly recommended PS methods, namely PS-matching or inverse probability of treatment weighting (IPTW), have not been studied in the context of small sample sizes.

METHODS

We conducted a series of Monte Carlo simulations to evaluate the influence of sample size, prevalence of treatment exposure, and strength of the association between the variables and the outcome and/or the treatment exposure, on the performance of these two methods.

RESULTS

Decreasing the sample size from 1,000 to 40 subjects did not substantially alter the Type I error rate, and led to relative biases below 10%. The IPTW method performed better than the PS-matching down to 60 subjects. When N was set at 40, the PS matching estimators were either similarly or even less biased than the IPTW estimators. Including variables unrelated to the exposure but related to the outcome in the PS model decreased the bias and the variance as compared to models omitting such variables. Excluding the true confounder from the PS model resulted, whatever the method used, in a significantly biased estimation of treatment effect. These results were illustrated in a real dataset.

CONCLUSION

Even in case of small study samples or low prevalence of treatment, PS-matching and IPTW can yield correct estimations of treatment effect unless the true confounders and the variables related only to the outcome are not included in the PS model.

Propensity score applied to survival data analysis through proportional hazards models: a Monte Carlo study

Propensity score methods are increasingly used in medical literature to estimate treatment effect using data from observational studies. Despite many papers on propensity score analysis, few have focused on the analysis of survival data. Even within the framework of the popular proportional hazard model, the choice among marginal, stratified or adjusted models remains unclear. A Monte Carlo simulation study was used to compare the performance of several survival models to estimate both marginal and conditional treatment effects. The impact of accounting or not for pairing when analysing propensity-score-matched survival data was assessed. In addition, the influence of unmeasured confounders was investigated. After matching on the propensity score, both marginal and conditional treatment effects could be reliably estimated. Ignoring the paired structure of the data led to an increased test size due to an overestimated variance of the treatment effect. Among the various survival models considered, stratified models systematically showed poorer performance. Omitting a covariate in the propensity score model led to a biased estimation of treatment effect, but replacement of the unmeasured confounder by a correlated one allowed a marked decrease in this bias. Our study showed that propensity scores applied to survival data can lead to unbiased estimation of both marginal and conditional treatment effect, when marginal and adjusted Cox models are used. In all cases, it is necessary to account for pairing when analysing propensity-score-matched data, using a robust estimator of the variance.

Survival after liver transplantation using hepatitis C virus-positive donor allografts: case-controlled analysis of the UNOS database

BACKGROUND

Numerous reports have documented reduced graft and patient survival after use of hepatitis C (HCV) seropositive allografts in liver transplantation (OLT). We aimed to examine if the use of a HCV+ liver allograft affects patient and graft survivals compared to HCV- donor allografts in a case-controlled analysis of the united network for organ sharing (UNOS) database.

METHODS

We examined 63,149 liver transplants (61,905 donors HCV-; 1,244 donors HCV+) from the UNOS standard transplant analysis and research (STAR) file from 1987 to 2007. Donor and recipient demographics and outcomes were collected in which donor HCV serology was complete. A case-controlled cohort from 11 donor and recipient variables comparing donor HCV- and HCV+ allografts (n=540 in each group) was created using propensity scores with a matching algorithm. Graft and patient survival was estimated using Kaplan-Meier survival curves.

RESULTS

Significant differences were evident in the unadjusted cohort between recipients who received HCV+ and HCV- allografts, including HCV+ recipients, donor and recipient age, and model for end-stage liver disease (MELD) exception cases. Use of HCV+ allograft resulted in significantly lower graft survival (8.1 vs. 10.6 years; P=0.001) and patient survival (10.2 vs. 12.3 years; P=0.01) after OLT. In the matched cohort, HCV seropositivity had no detrimental effect on the graft (P=0.57) or patient (P=0.78) survival after OLT.

CONCLUSIONS

This is the first population-based analysis to show that after adjusting for donor and recipient characteristics there was no difference in graft or patient survival with the use of HCV+ donor liver allografts compared to HCV- donor liver allografts.

Balance measures for propensity score methods: a clinical example on beta-agonist use and the risk of myocardial infarction

PURPOSE

Propensity score (PS) methods aim to control for confounding by balancing confounders between exposed and unexposed subjects with the same PS. PS balance measures have been compared in simulated data but limited in empirical data. Our objective was to compare balance measures in clinical data and assessed the association between long-acting inhalation beta-agonist (LABA) use and myocardial infarction.

METHODS

We estimated the relationship between LABA use and myocardial infarction in a cohort of adults with a diagnosis of asthma or chronic obstructive pulmonary disorder from the Utrecht General Practitioner Research Network database. More than two thousand PS models, including information on the observed confounders age, sex, diabetes, cardiovascular disease and chronic obstructive pulmonary disorder status, were applied. The balance of these confounders was assessed using the standardised difference (SD), Kolmogorov-Smirnov (KS) distance and overlapping coefficient. Correlations between these balance measures were calculated. In addition, simulation studies were performed to assess the correlation between balance measures and bias.

RESULTS

LABA use was not related to myocardial infarction after conditioning on the PS (median heart rate = 1.14, 95%CI = 0.47-2.75). When using the different balance measures for selecting a PS model, similar associations were obtained. In our empirical data, SD and KS distance were highly correlated balance measures (r = 0.92). In simulations, SD, KS distance and overlapping coefficient were similarly correlated to bias (e.g. r = 0.55, r = 0.52 and r = -0.57, respectively, when conditioning on the PS).

CONCLUSIONS

We recommend using the SD or the KS distance to quantify the balance of confounder distributions when applying PS methods.

Performance of disease risk scores, propensity scores, and traditional multivariable outcome regression in the presence of multiple confounders

Propensity scores are widely used in cohort studies to improve performance of regression models when considering large numbers of covariates. Another type of summary score, the disease risk score (DRS), which estimates disease probability conditional on nonexposure, has also been suggested. However, little is known about how it compares with propensity scores. Monte Carlo simulations were conducted comparing regression models using the DRS and the propensity score with models that directly adjust for all of the individual covariates. The DRS was calculated in 2 ways: from the unexposed population and from the full cohort. Compared with traditional multivariable outcome regression models, all 3 summary scores had comparable performance for moderate correlation between exposure and covariates and, for strong correlation, the full-cohort DRS and propensity score had comparable performance. When traditional methods had model misspecification, propensity scores and the full-cohort DRS had superior performance. All 4 models were affected by the number of events per covariate, with propensity scores and traditional multivariable outcome regression least affected. These data suggest that, for cohort studies for which covariates are not highly correlated with exposure, the DRS, particularly that calculated from the full cohort, is a useful tool.

Measuring balance and model selection in propensity score methods

PURPOSE

Propensity score (PS) methods focus on balancing confounders between groups to estimate an unbiased treatment or exposure effect. However, there is lack of attention in actually measuring, reporting and using the information on balance, for instance for model selection. We propose to use a measure for balance in PS methods and describe several of such measures: the overlapping coefficient, the Kolmogorov-Smirnov distance, and the Lévy distance.

METHODS

We performed simulation studies to estimate the association between these three and several mean based measures for balance and bias (i.e., discrepancy between the true and the estimated treatment effect).

RESULTS

For large sample sizes (n = 2000) the average Pearson's correlation coefficients between bias and Kolmogorov-Smirnov distance (r = 0.89), the Lévy distance (r = 0.89) and the absolute standardized mean difference (r = 0.90) were similar, whereas this was lower for the overlapping coefficient (r = -0.42). When sample size decreased to 400, mean based measures of balance had stronger correlations with bias. Models including all confounding variables, their squares and interaction terms resulted in smaller bias than models that included only main terms for confounding variables.

CONCLUSIONS

We conclude that measures for balance are useful for reporting the amount of balance reached in propensity score analysis and can be helpful in selecting the final PS model.

ESTIMATING TREATMENT EFFECTS ON HEALTHCARE COSTS UNDER EXOGENEITY: IS THERE A 'MAGIC BULLET'?

Methods for estimating average treatment effects, under the assumption of no unmeasured confounders, include regression models; propensity score adjustments using stratification, weighting, or matching; and doubly robust estimators (a combination of both). Researchers continue to debate about the best estimator for outcomes such as health care cost data, as they are usually characterized by an asymmetric distribution and heterogeneous treatment effects,. Challenges in finding the right specifications for regression models are well documented in the literature. Propensity score estimators are proposed as alternatives to overcoming these challenges. Using simulations, we find that in moderate size samples (n= 5000), balancing on propensity scores that are estimated from saturated specifications can balance the covariate means across treatment arms but fails to balance higher-order moments and covariances amongst covariates. Therefore, unlike regression model, even if a formal model for outcomes is not required, propensity score estimators can be inefficient at best and biased at worst for health care cost data. Our simulation study, designed to take a 'proof by contradiction' approach, proves that no one estimator can be considered the best under all data generating processes for outcomes such as costs. The inverse-propensity weighted estimator is most likely to be unbiased under alternate data generating processes but is prone to bias under misspecification of the propensity score model and is inefficient compared to an unbiased regression estimator. Our results show that there are no 'magic bullets' when it comes to estimating treatment effects in health care costs. Care should be taken before naively applying any one estimator to estimate average treatment effects in these data. We illustrate the performance of alternative methods in a cost dataset on breast cancer treatment.

Comparison of right lobe donor hepatectomy with elective right hepatectomy for other causes in New York

INTRODUCTION

Right lobe donor hepatectomy (RLDH) is a potential source of liver allografts given the ongoing shortage of deceased donor organs available. Since there is no live donor registry in the United States, a population-based, unsolicited state-wide analysis has yet to be reported.

METHODS

The New York (NY) State Inpatient Database was used to query 1,524 elective liver lobectomies performed from 2001 to 2006. RLDH were identified in this cohort (n = 195; 13%). Most common indications for elective right lobe hepatectomy (ERH) were metastatic colon cancer (50%) and hepatocellular carcinoma (HCC) (34%). Primary outcomes were mortality, perioperative resources and major postoperative complications.

RESULTS

After a dramatic drop in 2002, there was a slow increase in RLDH from 2003 to 2006 in New York. Donors were younger (median age 36 vs. 60 years, P < 0.0001) and healthier (75% with no comorbidities vs. 18%, P < 0.0001) than patients undergoing ERH for other causes. Median length of hospital stay was 7 days in both groups. Donors were less likely to require blood transfusion (22.6 vs. 62.8%, P < 0.0001) and received less blood (mean 0.10 units vs. 2.4 units). Major post-operative complications based on the Clavien classification occurred in only 2.6% of donor cases compared to 13.8% in non-donors (P < 0.0001). There was one RLDH in-hospital mortality (0.5%) in New York compared to 4.3% after ERH (P = 0.003).

CONCLUSIONS

This study represents one of the first unsolicited regional analyses of donor morbidity and resource utilization for RLDH and further emphasizes the need and utility of a live donor registry.

An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies

The propensity score is the probability of treatment assignment conditional on observed baseline characteristics. The propensity score allows one to design and analyze an observational (nonrandomized) study so that it mimics some of the particular characteristics of a randomized controlled trial. In particular, the propensity score is a balancing score: conditional on the propensity score, the distribution of observed baseline covariates will be similar between treated and untreated subjects. I describe 4 different propensity score methods: matching on the propensity score, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, and covariate adjustment using the propensity score. I describe balance diagnostics for examining whether the propensity score model has been adequately specified. Furthermore, I discuss differences between regression-based methods and propensity score-based methods for the analysis of observational data. I describe different causal average treatment effects and their relationship with propensity score analyses.

Comparing paired vs non-paired statistical methods of analyses when making inferences about absolute risk reductions in propensity-score matched samples

Propensity-score matching allows one to reduce the effects of treatment-selection bias or confounding when estimating the effects of treatments when using observational data. Some authors have suggested that methods of inference appropriate for independent samples can be used for assessing the statistical significance of treatment effects when using propensity-score matching. Indeed, many authors in the applied medical literature use methods for independent samples when making inferences about treatment effects using propensity-score matched samples. Dichotomous outcomes are common in healthcare research. In this study, we used Monte Carlo simulations to examine the effect on inferences about risk differences (or absolute risk reductions) when statistical methods for independent samples are used compared with when statistical methods for paired samples are used in propensity-score matched samples. We found that compared with using methods for independent samples, the use of methods for paired samples resulted in: (i) empirical type I error rates that were closer to the advertised rate; (ii) empirical coverage rates of 95 per cent confidence intervals that were closer to the advertised rate; (iii) narrower 95 per cent confidence intervals; and (iv) estimated standard errors that more closely reflected the sampling variability of the estimated risk difference. Differences between the empirical and advertised performance of methods for independent samples were greater when the treatment-selection process was stronger compared with when treatment-selection process was weaker. We recommend using statistical methods for paired samples when using propensity-score matched samples for making inferences on the effect of treatment on the reduction in the probability of an event occurring.

Relapse according to antipsychotic treatment in schizophrenic patients: a propensity-adjusted analysis

OBJECTIVE

To compare the rate of relapse as a function of antipsychotic treatment (monotherapy vs. polypharmacy) in schizophrenic patients over a 2-year period.

METHODS

Using data from a multicenter cohort study conducted in France, we performed a propensity-adjusted analysis to examine the association between the rate of relapse over a 2-year period and antipsychotic treatment (monotherapy vs. polypharmacy).

RESULTS

Our sample consisted in 183 patients; 50 patients (27.3%) had at least one period of relapse and 133 had no relapse (72.7%). Thirty-eight (37.7) percent of the patients received polypharmacy. The most severely ill patients were given polypharmacy: the age at onset of illness was lower in the polypharmacy group (p = 0.03). Patients that received polypharmacy also presented a higher general psychopathology PANSS subscore (p = 0.04) but no statistically significant difference was found in the PANSS total score or the PANSS positive or negative subscales. These patients were more likely to be given prescriptions for sedative drugs (p < 0.01) and antidepressant medications (p = 0.03). Relapse was found in 23.7% of patients given monotherapy and 33.3% given polypharmacy (p = 0.16). After stratification according to quintiles of the propensity score, which eliminated all significant differences for baseline characteristics, antipsychotic polypharmacy was not statistically associated with an increase of relapse: HR = 1.686 (0.812; 2.505).

CONCLUSION

After propensity score adjustment, antipsychotic polypharmacy is not statistically associated to an increase of relapse. Future randomised studies are needed to assess the impact of antipsychotic polypharmacy in schizophrenia.

Propensity scores: From naïve enthusiasm to intuitive understanding

Estimation of the effect of a binary exposure on an outcome in the presence of confounding is often carried out via outcome regression modelling. An alternative approach is to use propensity score methodology. The propensity score is the conditional probability of receiving the exposure given the observed covariates and can be used, under the assumption of no unmeasured confounders, to estimate the causal effect of the exposure. In this article, we provide a non-technical and intuitive discussion of propensity score methodology, motivating the use of the propensity score approach by analogy with randomised studies, and describe the four main ways in which this methodology can be implemented. We carefully describe the population parameters being estimated — an issue that is frequently overlooked in the medical literature. We illustrate these four methods using data from a study investigating the association between maternal choice to provide breast milk and the infant's subsequent neurodevelopment. We outline useful extensions of propensity score methodology and discuss directions for future research. Propensity score methods remain controversial and there is no consensus as to when, if ever, they should be used in place of traditional outcome regression models. We therefore end with a discussion of the relative advantages and disadvantages of each.

A Tutorial and Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In-Hospital Smoking Cessation Counseling on Mortality

Propensity score methods allow investigators to estimate causal treatment effects using observational or nonrandomized data. In this article we provide a practical illustration of the appropriate steps in conducting propensity score analyses. For illustrative purposes, we use a sample of current smokers who were discharged alive after being hospitalized with a diagnosis of acute myocardial infarction. The exposure of interest was receipt of smoking cessation counseling prior to hospital discharge and the outcome was mortality with 3 years of hospital discharge. We illustrate the following concepts: first, how to specify the propensity score model; second, how to match treated and untreated participants on the propensity score; third, how to compare the similarity of baseline characteristics between treated and untreated participants after stratifying on the propensity score, in a sample matched on the propensity score, or in a sample weighted by the inverse probability of treatment; fourth, how to estimate the effect of treatment on outcomes when using propensity score matching, stratification on the propensity score, inverse probability of treatment weighting using the propensity score, or covariate adjustment using the propensity score. Finally, we compare the results of the propensity score analyses with those obtained using conventional regression adjustment.

The performance of different propensity-score methods for estimating differences in proportions (risk differences or absolute risk reductions) in observational studies

Propensity score methods are increasingly being used to estimate the effects of treatments on health outcomes using observational data. There are four methods for using the propensity score to estimate treatment effects: covariate adjustment using the propensity score, stratification on the propensity score, propensity-score matching, and inverse probability of treatment weighting (IPTW) using the propensity score. When outcomes are binary, the effect of treatment on the outcome can be described using odds ratios, relative risks, risk differences, or the number needed to treat. Several clinical commentators suggested that risk differences and numbers needed to treat are more meaningful for clinical decision making than are odds ratios or relative risks. However, there is a paucity of information about the relative performance of the different propensity-score methods for estimating risk differences. We conducted a series of Monte Carlo simulations to examine this issue. We examined bias, variance estimation, coverage of confidence intervals, mean-squared error (MSE), and type I error rates. A doubly robust version of IPTW had superior performance compared with the other propensity-score methods. It resulted in unbiased estimation of risk differences, treatment effects with the lowest standard errors, confidence intervals with the correct coverage rates, and correct type I error rates. Stratification, matching on the propensity score, and covariate adjustment using the propensity score resulted in minor to modest bias in estimating risk differences. Estimators based on IPTW had lower MSE compared with other propensity-score methods. Differences between IPTW and propensity-score matching may reflect that these two methods estimate the average treatment effect and the average treatment effect for the treated, respectively.

Statistical criteria for selecting the optimal number of untreated subjects matched to each treated subject when using many-to-one matching on the propensity score

Propensity-score matching is increasingly being used to estimate the effects of treatments using observational data. In many-to-one (M:1) matching on the propensity score, M untreated subjects are matched to each treated subject using the propensity score. The authors used Monte Carlo simulations to examine the effect of the choice of M on the statistical performance of matched estimators. They considered matching 1-5 untreated subjects to each treated subject using both nearest-neighbor matching and caliper matching in 96 different scenarios. Increasing the number of untreated subjects matched to each treated subject tended to increase the bias in the estimated treatment effect; conversely, increasing the number of untreated subjects matched to each treated subject decreased the sampling variability of the estimated treatment effect. Using nearest-neighbor matching, the mean squared error of the estimated treatment effect was minimized in 67.7% of the scenarios when 1:1 matching was used. Using nearest-neighbor matching or caliper matching, the mean squared error was minimized in approximately 84% of the scenarios when, at most, 2 untreated subjects were matched to each treated subject. The authors recommend that, in most settings, researchers match either 1 or 2 untreated subjects to each treated subject when using propensity-score matching.

Applying propensity score methods in medical research: pitfalls and prospects

The authors review experimental and nonexperimental causal inference methods, focusing on assumptions for the validity of instrumental variables and propensity score (PS) methods. They provide guidance in four areas for the analysis and reporting of PS methods in medical research and selectively evaluate mainstream medical journal articles from 2000 to 2005 in the four areas, namely, examination of balance, overlapping support description, use of estimated PS for evaluation of treatment effect, and sensitivity analyses. In spite of the many pitfalls, when appropriately evaluated and applied, PS methods can be powerful tools in assessing average treatment effects in observational studies. Appropriate PS applications can create experimental conditions using observational data when randomized controlled trials are not feasible and, thus, lead researchers to an efficient estimator of the average treatment effect.

Propensity scores in intensive care and anaesthesiology literature: a systematic review

INTRODUCTION

Propensity score methods have been increasingly used in the last 10 years. However, the practical use of the propensity score (PS) has been reported as heterogeneous in several papers reviewing the use of propensity scores and giving some advice. No precedent work has focused on the specific application of PS in intensive care and anaesthesiology literature.

OBJECTIVES

After a brief development of the theory of propensity score, to assess the use and the quality of reporting of PS studies in intensive care and anaesthesiology, and to evaluate how past reviews have influenced the quality of the reporting.

STUDY DESIGN AND SETTING

Forty-seven articles published between 2006 and 2009 in the intensive care and anaesthesiology literature were evaluated. We extracted the characteristics of the report, the type of analysis, the details of matching procedures, the number of patients in treated and control groups, and the number of covariates included in the PS models.

RESULTS

Of the 47 articles reviewed, 26 used matching on PS, 12 used stratification on PS and 9 used adjustment on PS. The method used was reported in 81% of the articles, and the choice to conduct a paired analysis or not was reported in only 15%. The comparison with the previously published reviews showed little improvement in reporting in the last few years.

CONCLUSION

The quality of reporting propensity scores in intensive care and anaesthesiology literature should be improved. We provide some recommendations to the investigators in order to improve the reporting of PS analyses.

Surgeon volume metrics in laparoscopic cholecystectomy

AIM

Numerous reports in the 1990s pointed to a learning curve for laparoscopic cholecystectomy (LC), critical in achieving excellent outcomes. As LC is now standard therapy for acute cholecystitis (AC), we aimed to determine if surgeon volume is still vital to patient outcomes.

METHODS

The Nationwide Inpatient Sample was used to query 80,149 emergent/urgent cholecystectomies performed for AC from 1999 to 2005 in 12 states with available surgeon/hospital identifiers. Volume groups were determined based on thirds of number of cholecystectomies performed per year for AC; two groups were created [low volume (LV): <or=15/year; high volume (HV): >15/year]. Primary endpoints were the rate of open conversion, bile duct injury (BDI), in-hospital mortality, and prolonged length of stay (LOS). Propensity scores were used to create a matched cohort analysis. Logistic regression models were created to further assess the effect of surgeon volume on primary endpoints.

RESULTS

The number of cases performed by HV surgeons increased from 24% to 44% from 1999 to 2005. HV surgeons were more likely to perform LC, had fewer conversions, lower incidence of prolonged LOS, lower BDI, and lower in-hospital mortality. After matching the volume cohorts to create a case-controlled analysis, multivariate analysis confirmed that surgeon volume was an independent predictor of open conversion and prolonged LOS but not BDI and in-hospital mortality.

CONCLUSIONS

Increasing surgical volume remains associated with improved outcomes after surgery during emergent/urgent admission for AC with fewer open conversions and prolonged LOS. Our results suggest that referral to HV surgeons has improved outcomes after LC for AC.

Primer on statistical interpretation or methods report card on propensity-score matching in the cardiology literature from 2004 to 2006: a systematic review

BACKGROUND

Propensity-score matching is frequently used in the cardiology literature. Recent systematic reviews have found that this method is, in general, poorly implemented in the medical literature. The study objective was to examine the quality of the implementation of propensity-score matching in the general cardiology literature.

METHODS AND RESULTS

A total of 44 articles published in the American Heart Journal, the American Journal of Cardiology, Circulation, the European Heart Journal, Heart, the International Journal of Cardiology, and the Journal of the American College of Cardiology between January 1, 2004, and December 31, 2006, were examined. Twenty of the 44 studies did not provide adequate information on how the propensity-score-matched pairs were formed. Fourteen studies did not report whether matching on the propensity score balanced baseline characteristics between treated and untreated subjects in the matched sample. Only 4 studies explicitly used statistical methods appropriate for matched studies to compare baseline characteristics between treated and untreated subjects. Only 11 (25%) of the 44 studies explicitly used statistical methods appropriate for the analysis of matched data when estimating the effect of treatment on the outcomes. Only 2 studies described the matching method used, assessed balance in baseline covariates by appropriate methods, and used appropriate statistical methods to estimate the treatment effect and its significance.

CONCLUSIONS

Application of propensity-score matching was poor in the cardiology literature. Suggestions for improving the reporting and analysis of studies that use propensity-score matching are provided.

An application of propensity score methods to estimate the treatment effect of corticosteroids in patients with severe cutaneous adverse reactions

PURPOSE

To investigate whether propensity score (ps) methods could reasonably be applied to estimate the treatment effect on mortality, based on a comparatively small sample of patients with severe cutaneous adverse reactions (SCAR) and who come from different countries where physicians prefer different treatment schemes.

METHODS

Ps methods were applied to cope with confounding due to non-randomized treatment assignment for the analysis of the treatment data obtained in the case-control study EuroSCAR. For the study's purpose, the analysis focused on the comparison of the treatments: corticosteroids (STER) and supportive care only (SUPP).

RESULTS

206 French and German patients were treated either with SUPP or STER. Imbalances between treatment groups as well as between the countries were recognized. Concerning the balance between the treatment groups no ps model for the full cohort was satisfying. In addition, the inclusion of a variable for patient's country led to a separation of the patients by country. Thus, we developed ps models for each country separately and estimated the treatment effects (France: odds ratio (OR) 0.52, 95% confidence interval (CI) 0.09-3.10, Germany: OR 0.23, CI 0.06-0.92, Overall: OR 0.33 CI 0.11-1.04).

CONCLUSIONS

The application of the ps methods was successful and provided valuable information. We could confirm the findings of the original analysis which was based on standard logistic regression, especially concerning the necessity of a country-specific analysis. The observed country differences in the estimated treatment effects were less pronounced and thus seemed to be more reasonable than those of the past analysis.

Admission volume determines outcome for patients with acute pancreatitis

BACKGROUND & AIMS

There is controversy over the optimal management strategy for patients with acute pancreatitis (AP). Studies have shown a hospital volume benefit for in-hospital mortality after surgery, and we examined whether a similar mortality benefit exists for patients admitted with AP.

METHODS

Using the Nationwide Inpatient Sample, discharge records for all adult admissions with a primary diagnosis of AP (n = 416,489) from 1998 to 2006 were examined. Hospitals were categorized based on number of patients with AP; the highest third were defined as high volume (HV, >or=118 cases/year) and the lower two thirds as low volume (LV, <118 cases/year). A matched cohort based on propensity scores (n = 43,108 in each group) eliminated all demographic differences to create a case-controlled analysis. Adjusted mortality was the primary outcome measure.

RESULTS

In-hospital mortality for patients with AP was 1.6%. Hospital admissions for AP increased over the study period (P < .0001). HV hospitals tended to be large (82%), urban (99%), academic centers (59%) that cared for patients with greater comorbidities (P < .001). Adjusted length of stay was lower at HV compared with LV hospitals (odds ratio, 0.86; 95% confidence interval, 0.82-0.90). After adjusting for patient and hospital factors, the mortality rate was significantly lower for patients treated at HV hospitals (hazard ratio, 0.74; 95% confidence interval, 0.67-0.83).

CONCLUSIONS

The rates of admissions for AP in the United States are increasing. At hospitals that admit the most patients with AP, patients had a shorter length of stay, lower hospital charges, and lower mortality rates than controls in this matched analysis.

The impact of socioeconomic status on presentation and treatment of diverticular disease

INTRODUCTION

Diverticular disease is a common medical problem, but it is unknown if lower socioeconomic status (SES) affects patient outcomes in diverticular disease.

MATERIAL AND METHODS

The New York (NY) State Inpatient Database was used to query 8,117 cases of diverticular disease occurring in patients aged 65-85 in 2006. Race and SES were assessed by creating a composite score based on race, primary insurance payer, and median income bracket.

RESULTS

Primary outcomes were differences in disease presentation, use of elective surgery, complication rates when surgery was performed, and overall mortality and length of stay. Patients of lower SES were younger, more likely to be female, to have multiple co-morbid conditions, to present as emergent/urgent admissions, and to present with diverticulitis complicated by hemorrhage (p < 0.0001).

DISCUSSION

Overall, patients of low SES were less likely to receive surgical intervention, while rates of surgery were similar in elective cases. When surgery was performed, patients of lower SES had similar complication rates (25.4% vs. 20.2%, p = 0.06) and higher overall mortality (9.0% vs. 4.4%, p = 0.003).

CONCLUSION

Patients of low SES who are admitted with diverticular disease have an increased likelihood to present emergently, have worse disease on admission, and are less likely to receive surgery.

Is the growth in laparoscopic surgery reproducible with more complex procedures?

BACKGROUND

Laparoscopic (LAP) surgery has experienced significant growth since the early 1990s and is now considered the standard of care for many procedures like cholecystectomy. Increased expertise, training, and technological advancements have allowed the development of more complex LAP procedures including the removal of solid organs. Unlike LAP cholecystectomy, it is unclear whether complex LAP procedures are being performed with the same growth today.

METHODS

Using the Nationwide Inpatient Sample (NIS) from 1998 to 2006, patients who underwent elective LAP or open colectomy (n = 220,839), gastrectomy (n = 17,289), splenectomy (n = 9,174), nephrectomy (n = 64,171), or adrenalectomy (n = 5,556) were identified. The Elixhauser index was used to adjust for patient comorbidities. To account for patient selection and referral bias, a matched analysis was performed using propensity scores. The main endpoints were adjusted for in-hospital mortality and prolonged length of stay (LOS).

RESULTS

Complex LAP procedures account for a small percentage of total elective procedures (colectomy, 3.8%; splenectomy, 8.8%; gastrectomy, 2.4%; nephrectomy, 7.0%; and adrenalectomy, 14.2%). These procedures have been performed primarily at urban (94%) and teaching (64%) centers. Although all LAP procedures trended up, the growth was greatest in LAP colectomy and nephrectomy (P < .001). In a case-controlled analysis, there was a mortality benefit only for LAP colectomy (hazard ratio [HR] = 0.53; 95% confidence interval [CI] = 0.34-0.82) when compared with their respective open procedures. All LAP procedures except gastrectomy had a lower prolonged LOS compared with their open counterparts.

CONCLUSION

Despite the significant benefits of complex LAP procedures as measured by LOS and in-hospital mortality, the growth of these operations has been slow unlike the rapid acceptance of LAP cholecystectomy. Future studies to identify the possible causes of this slow growth should consider current training paradigms, technical capabilities, economic disincentive, and surgical specialization.

Hospital volume as a surrogate for laparoscopically assisted colectomy

BACKGROUND

Although laparoscopic colectomy is reported to have favorable outcomes compared with open colectomy, it has yet to gain widespread acceptance in the United States. This study sought to investigate whether hospital volume is a factor determining the use of laparoscopy for colectomy.

METHODS

Using the Nationwide Inpatient Sample (NIS, 1998-2006), patients undergoing elective colon resection with and without laparoscopy were identified. Unique hospital identifiers were used to divide hospital volume into equal thirds, with the highest third defined as high volume and the lower two-thirds defined as low volume. The primary end point was the use of laparoscopy after adjustment for patient and hospital covariates.

RESULTS

A total of 209,769 colon resections were performed in the study period. Overall, only 8,407 (4%) of these resections were performed with laparoscopy. High-volume centers, which tended to be large, urban teaching hospitals, treated more patients in the highest income bracket and patients with private insurance than low-volume hospitals (p < 0.0001). High-volume hospitals used laparoscopy more often than low-volume hospitals (5.2% vs. 3.4%). After adjustment for covariates using multivariate analysis and propensity scores, analysis showed that patients with private insurance and those in the highest income bracket were more likely to receive laparoscopy (p < 0.0009). High-volume hospitals were more likely to perform laparoscopically assisted colectomy than low-volume hospitals (odds ratio [OR], 1.42; 95% confidence interval [CI], 1.23-1.56).

CONCLUSIONS

Socioeconomic differences appear to exist between high- and low-volume hospitals in the use of laparoscopy. High hospital volume is associated with an increased likelihood that colectomy will be performed with laparoscopy.

The Relative Ability of Different Propensity Score Methods to Balance Measured Covariates Between Treated and Untreated Subjects in Observational Studies

The propensity score is a balancing score: conditional on the propensity score, treated and untreated subjects have the same distribution of observed baseline characteristics. Four methods of using the propensity score have been described in the literature: stratification on the propensity score, propensity score matching, inverse probability of treatment weighting using the propensity score, and covariate adjustment using the propensity score. However, the relative ability of these methods to reduce systematic differences between treated and untreated subjects has not been examined. The authors used an empirical case study and Monte Carlo simulations to examine the relative ability of the 4 methods to balance baseline covariates between treated and untreated subjects. They used standardized differences in the propensity score matched sample and in the weighted sample. For stratification on the propensity score, within-quintile standardized differences were computed comparing the distribution of baseline covariates between treated and untreated subjects within the same quintile of the propensity score. These quintile-specific standardized differences were then averaged across the quintiles. For covariate adjustment, the authors used the weighted conditional standardized absolute difference to compare balance between treated and untreated subjects. In both the empirical case study and in the Monte Carlo simulations, they found that matching on the propensity score and weighting using the inverse probability of treatment eliminated a greater degree of the systematic differences between treated and untreated subjects compared with the other 2 methods. In the Monte Carlo simulations, propensity score matching tended to have either comparable or marginally superior performance compared with propensity-score weighting.

Old wine in new skins: the sensitivity of established findings to new methods

Findings from an evaluation of a model system for delivering mental health services to youth were reassessed to determine the robustness of key findings to the use of methodologies unavailable to the original analysts. These analyses address a key concern about earlier findings-that the quasi-experimental design involved the comparison of two noncomparable groups. The authors employed propensity score methodology to reconsider between-group baseline differences in observed characteristics of participating families. The authors also considered the possible effect of unobserved between-group differences. The data support previous studies that show few differences in outcomes, but the findings are sensitive to unobserved heterogeneity.

The impact of unmeasured baseline effect modification on estimates from an inverse probability of treatment weighted logistic model

We present the results of a Monte Carlo simulation study in which we demonstrate how strong baseline interactions between a confounding variable and a treatment can create an important difference between the marginal effect of exposure on outcome (as estimated by an inverse probability of treatment weighted logistic model) and the conditional effect (as estimated by an adjusted logistic regression model). The scenarios that we explored included one with a rare outcome and a strong and prevalent effect measure modifier where, across 1,000 simulated data sets, the estimates from an adjusted logistic regression model (mean beta = 0.475) and an inverse probability of treatment weighted logistic model (mean beta = 2.144) do not coincide with the known true effect (beta = 0.68925) when the effect measure modifier is not accounted for. When the marginal and conditional estimates do not coincide despite a rare outcome this may suggest that there is heterogeneity in the effect of treatment between individuals. Failure to specify effect measure modification in the statistical model appears to results in systematic differences between the conditional and marginal estimates. When these differences in estimates are observed, testing for or including interactions or non-linear modeling terms may be advised.

Type I error rates, coverage of confidence intervals, and variance estimation in propensity-score matched analyses

Propensity-score matching is frequently used in the medical literature to reduce or eliminate the effect of treatment selection bias when estimating the effect of treatments or exposures on outcomes using observational data. In propensity-score matching, pairs of treated and untreated subjects with similar propensity scores are formed. Recent systematic reviews of the use of propensity-score matching found that the large majority of researchers ignore the matched nature of the propensity-score matched sample when estimating the statistical significance of the treatment effect. We conducted a series of Monte Carlo simulations to examine the impact of ignoring the matched nature of the propensity-score matched sample on Type I error rates, coverage of confidence intervals, and variance estimation of the treatment effect. We examined estimating differences in means, relative risks, odds ratios, rate ratios from Poisson models, and hazard ratios from Cox regression models. We demonstrated that accounting for the matched nature of the propensity-score matched sample tended to result in type I error rates that were closer to the advertised level compared to when matching was not incorporated into the analyses. Similarly, accounting for the matched nature of the sample tended to result in confidence intervals with coverage rates that were closer to the nominal level, compared to when matching was not taken into account. Finally, accounting for the matched nature of the sample resulted in estimates of standard error that more closely reflected the sampling variability of the treatment effect compared to when matching was not taken into account.

Goodness-of-fit diagnostics for the propensity score model when estimating treatment effects using covariate adjustment with the propensity score

The propensity score is defined to be a subject's probability of treatment selection, conditional on observed baseline covariates. Conditional on the propensity score, treated and untreated subjects have similar distributions of observed baseline covariates. In the medical literature, there are three commonly employed propensity-score methods: stratification (subclassification) on the propensity score, matching on the propensity score, and covariate adjustment using the propensity score. Methods have been developed to assess the adequacy of the propensity score model in the context of stratification on the propensity score and propensity-score matching. However, no comparable methods have been developed for covariate adjustment using the propensity score. Inferences about treatment effect made using propensity-score methods are only valid if, conditional on the propensity score, treated and untreated subjects have similar distributions of baseline covariates. We develop both quantitative and qualitative methods to assess the balance in baseline covariates between treated and untreated subjects. The quantitative method employs the weighted conditional standardized difference. This is the conditional difference in the mean of a covariate between treated and untreated subjects, in units of the pooled standard deviation, integrated over the distribution of the propensity score. The qualitative method employs quantile regression models to determine whether, conditional on the propensity score, treated and untreated subjects have similar distributions of continuous covariates. We illustrate our methods using a large dataset of patients discharged from hospital with a diagnosis of a heart attack (acute myocardial infarction). The exposure was receipt of a prescription for a beta-blocker at hospital discharge.

High mortality rate in rheumatoid arthritis with subluxation of the cervical spine: a cohort study of operated and nonoperated patients

STUDY DESIGN

In a prospective cohort study 532 patients with rheumatoid arthritis (RA) and subluxations of the cervical spine were consecutively collected during 1974-1999.

OBJECTIVE

The aims of the study were to assess important factors affecting the mortality rate and the timing of surgical intervention.

SUMMARY OF BACKGROUND DATA

The average follow-up time from the first visit to death or to the end of the study was 8.5 (SD, 5.7) years. Of the 217 operated patients 144 (66%) died, and of the 315 nonoperated patients 137 (43%) died.

METHODS

Patients were selected for operative intervention based on anterior, vertical and subaxial subluxations, pain, and/or cervical neurology. Survival analyses were used for comparisons between patients with RA and the normal population, and between the operated and those treated conservatively.

RESULTS

The survival rate for all RA patients was significantly reduced when compared with average survival in Norway (P < 0.001). The operated group had a significantly lower survival rate than the nonoperated group. In patients with severe instability of the cervical spine, the defined selection criteria for surgical intervention were specific. By comparison of calculated propensity scores, the operated and nonoperated groups were too different to be directly comparable. After surgery only 11 patients (5%) experienced residual pain in the neck or neurologic symptoms. None of these patients were alive at the end of the study, signifying that residual pain or neurologic symptoms are poor prognostic signs (P = 0.015). In the operated group, anterior subluxation and vertical settling greater than the lower indication limits did not have a significant influence on the survival rate, but there was a reduced survival for patients with subaxial subluxations. A clear association was found between increased vertical settling and sudden death.

CONCLUSION

RA with neck involvement is a progressive and serious condition with reduced lifetime expectancy. Hence, our interpretation is that operative intervention improves local symptoms and most likely changes the condition from worse to better by increasing lifetime expectancy in high risk patients. Since the per- and postoperative complications are few, a changed attitude toward more liberal indications for earlier surgery may reduce the symptoms and the mortality rate even more.