Optimal Multilevel Matching in Clustered Observational Studies: A Case Study of the Effectiveness of Private Schools Under a Large-Scale Voucher System

Approximate balancing weights for clustered observational study designs

In a clustered observational study, a treatment is assigned to groups and all units within the group are exposed to the treatment. We develop a new method for statistical adjustment in clustered observational studies using approximate balancing weights, a generalization of inverse propensity score weights that solve a convex optimization problem to find a set of weights that directly minimize a measure of covariate imbalance, subject to an additional penalty on the variance of the weights. We tailor the approximate balancing weights optimization problem to the clustered observational study setting by deriving an upper bound on the mean square error and finding weights that minimize this upper bound, linking the level of covariate balance to a bound on the bias. We implement the procedure by specializing the bound to a random cluster-level effects model, leading to a variance penalty that incorporates the signal-to-noise ratio and penalizes the weight on individuals and the total weight on groups differently according to the the intra-class correlation.

An overview of propensity score matching methods for clustered data

Propensity score matching is commonly used in observational studies to control for confounding and estimate the causal effects of a treatment or exposure. Frequently, in observational studies data are clustered, which adds to the complexity of using propensity score techniques. In this article, we give an overview of propensity score matching methods for clustered data, and highlight how propensity score matching can be used to account for not just measured confounders, but also unmeasured cluster level confounders. We also consider using machine learning methods such as generalized boosted models to estimate the propensity score and show that accounting for clustering when using these methods can greatly reduce the performance, particularly when there are a large number of clusters and a small number of subjects per cluster. In order to get around this we highlight scenarios where it may be possible to control for measured covariates using propensity score matching, while using fixed effects regression in the outcome model to control for cluster level covariates. Using simulation studies we compare the performance of different propensity score matching methods for clustered data across a number of different settings. Finally, as an illustrative example we apply propensity score matching methods for clustered data to study the causal effect of aspirin on hearing deterioration using data from the conservation of hearing study.

Propensity score methods for observational studies with clustered data: A review

Propensity score methods are a popular approach to mitigating confounding bias when estimating causal effects in observational studies. When study units are clustered (eg, patients nested within health systems), additional challenges arise such as accounting for unmeasured confounding at multiple levels and dependence between units within the same cluster. While clustered observational data are widely used to draw causal inferences in many fields, including medicine and healthcare, extensions of propensity score methods to clustered settings are still a relatively new area of research. This article presents a framework for estimating causal effects using propensity scores when study units are nested within clusters and are nonrandomly assigned to treatment conditions. We emphasize the need for investigators to examine the nature of the clustering, among other properties, of the observational data at hand in order to guide their choice of causal estimands and the corresponding propensity score approach.

Bridging the gap between multilevel modeling and economic methods

Many datasets used in the social sciences have a hierarchical structure, where lower units of aggregation are 'nested' in higher units. In many disciplines, such data are analyzed using multilevel modeling (MLM, also known as hierarchical linear modeling). However, MLM as a framework is relatively unknown in economics. Instead, economists use a range of separate econometric methods, including cluster-robust standard errors, fixed effects models, models with cross-level interactions, and estimated dependent variable models. Relying on an extensive literature review, this paper describes this methodological divide and provides a detailed comparison between MLM and 'economic methods' in their abilities to deal with three methodological challenges inherent in multilevel data ‒ clustering, omitted variables, and coefficients' heterogeneity across groups. We unfold the comparative advantages of these two methodological approaches and provide practical recommendations about which of them should be used, why, and in what settings.

Association Between Telepsychiatry Capability and Treatment of Patients With Mental Illness in the Emergency Department

OBJECTIVE

Because of limited access to psychiatrists, patients with acute mental illness in some emergency departments (EDs) may wait days for a consultation in the ED or as a medical-surgical admission. The study assessed whether telepsychiatry improves access to care and decreases ED wait times and hospital admissions.

METHODS

ED visits with a primary diagnosis of mental illness were identified from 2010-2018 Medicare claims. A total of 134 EDs across 22 states that implemented telepsychiatry between 2013 and 2016 were matched 1:1 with control EDs without telepsychiatry on several characteristics, including availability of in-person psychiatrist consultations. Outcomes included patients' likelihood of admission to a medical-surgical or psychiatric bed, mental illness spending, prolonged ED length of stay (LOS) (two or more midnights in the ED), 90-day mortality, and outpatient follow-up care. Using a difference-in-difference design, changes in outcomes between the 3 years before telepsychiatry adoption and the 2 years after were examined.

RESULTS

There were 172,708 ED mental illness visits across the 134 matched ED pairs in the study period. Telepsychiatry adoption was associated with increased admissions to a psychiatric bed (differential increase, 4.3 percentage points; p<0.001), decreased admissions to a medical-surgical bed (differential decrease, 2.0 percentage points; p<0.001), increased likelihood of a prolonged ED LOS (differential increase, 3.0 percentage points; p<0.001), and increased mental illness spending (differential increase, $292; p<0.01).

CONCLUSIONS

Telepsychiatry adoption was associated with a lower likelihood of admission to a medical-surgical bed but an increased likelihood of admission to a psychiatric bed and a prolonged ED LOS.

Community-Level Social Support Infrastructure and Adult Onset of Major Depressive Disorder in a South Asian Postconflict Setting

IMPORTANCE

Individual-level social support protects against major depressive disorder (MDD) among adults exposed to trauma. Little is known about the consequences of community-level interventions in the general population.

OBJECTIVE

To determine the potential consequences of neighborhood social infrastructure on incident MDD in a high-risk general population.

DESIGN, SETTING, AND PARTICIPANTS

This longitudinal, multilevel study estimated associations between a neighborhood-level program in a case-control design and subsequent individual outcomes across 10 years (2006-2015) in a cohort of young adults. Exogenously placed social programs simulate natural experiment conditions in a high-poverty population experiencing armed conflict (1998-2006). The western Chitwan valley in Nepal has a general population at high risk of MDD, with neighborhoods exposed to interventions to improve social support. From a random sample (response rate 93%) selected to represent the general population in 2016, participants aged 25 to 34 years in 2006 were studied. These individuals resided within 149 neighborhoods that varied in their availability of active social support programs. The analyses were conducted between October 2020 and November 2021.

EXPOSURES

The Small Farmers Development Program (SFDP) uses shared, joint liability financial credit among neighbors to build social capital and cohesion within neighborhoods.

MAIN OUTCOMES AND MEASURES

Onset of DSM-IV MDD after the conflict, assessed by the Nepal-specific, clinically validated World Mental Health Composite International Diagnostic Interview with a life history calendar. The hypothesis tested was that exposure to SFDP reduced adult onset of MDD.

RESULTS

Of the 1917 survey participants, 886 (46.2%) were women, and 856 (44.7%) were of Brahmin or Chhetri ethnicity. Of the 149 neighborhoods, 21 had an active SFDP group, and 156 of 1917 (8.1%) participants experienced MDD between 2006 and 2015. Discrete-time hazard models showed participants living in neighborhoods with an SFDP experienced incident MDD at nearly half the rate as others (odds ratio = 0.55; 95% CI, 0.30-1.02; P = .06). A multivariate, multilevel matching analysis showed the incidence of MDD among adults living in neighborhoods with an SFDP was 19 of 256 (7.4%), compared with 33 of 256 (12.9%) in the matched sample with no SFDP (z = 2.05; P = .04).

CONCLUSIONS AND RELEVANCE

Living in a neighborhood with community-level social support infrastructure was associated with reduced subsequent rates of adult-onset MDD, even in this high-risk population. Investments in such infrastructure may reduce population-level MDD, supporting clinical focus on potentially unpreventable cases.

Exposure to Armed Conflict in Childhood vs Older Ages and Subsequent Onset of Major Depressive Disorder

Importance

This study offers a rare opportunity to evaluate life-course differences in the likelihood of developing major depressive disorder (MDD) after exposure to georeferenced neighborhood-level violence during an armed conflict.

Objective

To examine age cohort (age <11 vs ≥11 years) differences in associations of neighborhood-level violence with subsequent depression onset, independently of individual exposure and other key characteristics.

Design, Setting, and Participants

The Chitwan Valley Family Study is a population-representative panel study (1995 to present) conducted in Western Chitwan in Nepal, a low-income country that experienced a medium-intensity armed conflict from 2000 to 2006. Data for violent events were collected during the armed conflict and were linked to lifetime histories of MDD (collected in 2016-2018). The present cohort study analyzes 10 623 participants within 151 neighborhoods, systematically selected and representative of Western Chitwan. All residents aged 15 to 59 years at MDD assessment were eligible (response rate, 93%). Data analysis was performed from May 2019 to July 2020.

Exposures

Georeferenced number of armed conflict-related physical beatings within 1 km of residential neighborhood.

Main Outcomes and Measures

The main outcome was onset of MDD, as defined by Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition), during or after the conflict, stratified by children (aged <11 years) and older individuals (aged ≥11 years), assessed by the Nepal-specific World Mental Health-Composite International Diagnostic Instrument 3.0 with a life history calendar.

Results

In total, 10 623 participants (5745 female [54.08%]; 4074 [38.35%] aged <11 years at the conflict start) contributed 171 899 person-years of exposure to the risk of MDD. Two or more beatings occurred within 1 km of 15 neighborhoods (9.9%). Discrete-time survival models showed that children (but not older individuals) living in neighborhoods with 2 or more beatings had a higher likelihood of developing MDD than those who lived in a community with no beatings (odds ratio, 1.82; 95% CI, 1.17-2.84; P = .008); there was also a significant interaction between age group and neighborhood beatings (odds ratio, 1.85; 95% CI, 1.27-2.70; P = .001). A confirmatory, multivariable, multilevel matching analysis showed a neighborhood association for children (z = -2.66; P = .008), but not older individuals (z = -0.454; P = .65). The mean (SE) incidence of MDD among children living in neighborhoods with 2 or more beatings nearby was 12.69% (2.37%) vs 5.08% (1.56%) in the matched unexposed sample.

Conclusions and Relevance

The youngest individuals may be the most at risk during times of violence, with mental health consequences lasting long after conflicts have subsided and should be a priority for population-level interventions. Future research should consider other disorders, other types of violence, and elderly individuals.

Robust estimation of the causal effect of time-varying neighborhood factors on health outcomes

The fundamental difficulty of establishing causal relationships between an exposure and an outcome in observational data involves disentangling causality from confounding factors. This problem underlies much of neighborhoods research, which abounds with studies that consider associations between neighborhood characteristics and health outcomes in longitudinal data. Such analyses are confounded by selection issues; individuals with above average health outcomes (or associated characteristics) may self-select into advantaged neighborhoods. Techniques commonly used to assess causal inferences in observational longitudinal data, such as inverse probability of treatment weighting (IPTW), may be inappropriate in neighborhoods data due to unique characteristics of such data. We advance the IPTW toolkit by introducing a procedure based on a multivariate kernel density function which is more appropriate for neighborhoods data. The proposed weighting method is applied in conjunction with a marginal structural model. Our empirical analyses use longitudinal data from the Health and Retirement Study; our exposure of interest is an index of neighborhood socioeconomic status (NSES), and we examine its influence on cognitive function. Our findings illustrate the importance of the choice of method for IPTW-the comparison weighting methods provide poor balance across the set of covariates (which is not the case for our preferred procedure) and yield misleading results when applied in the outcomes models. The utility of the multivariate kernel is also validated via simulation. In addition, our findings emphasize the importance of IPTW-controlling for covariates within a regression without IPTW indicates that NSES affects cognition, whereas IPTW-weighted models fail to show a statistically significant effect.

Robust policy evaluation from large-scale observational studies

Under the current policy decision making paradigm we make or evaluate a policy decision by intervening different socio-economic parameters and analyzing the impact of those interventions. This process involves identifying the causal relation between interventions and outcomes. Matching method is one of the popular techniques to identify such causal relations. However, in one-to-one matching, when a treatment or control unit has multiple pair assignment options with similar match quality, different matching algorithms often assign different pairs. Since all the matching algorithms assign pairs without considering the outcomes, it is possible that with the same data and same hypothesis, different experimenters can reach different conclusions creating an uncertainty in policy decision making. This problem becomes more prominent in the case of large-scale observational studies as there are more pair assignment options. Recently, a robust approach has been proposed to tackle the uncertainty that uses an integer programming model to explore all possible assignments. Though the proposed integer programming model is very efficient in making robust causal inference, it is not scalable to big data observational studies. With the current approach, an observational study with 50,000 samples will generate hundreds of thousands binary variables. Solving such integer programming problem is computationally expensive and becomes even worse with the increase of sample size. In this work, we consider causal inference testing with binary outcomes and propose computationally efficient algorithms that are adaptable for large-scale observational studies. By leveraging the structure of the optimization model, we propose a robustness condition that further reduces the computational burden. We validate the efficiency of the proposed algorithms by testing the causal relation between the Medicare Hospital Readmission Reduction Program (HRRP) and non-index readmissions (i.e., readmission to a hospital that is different from the hospital that discharged the patient) from the State of California Patient Discharge Database from 2010 to 2014. Our result shows that HRRP has a causal relation with the increase in non-index readmissions. The proposed algorithms proved to be highly scalable in testing causal relations from large-scale observational studies.

National Trends in Utilization and 1-Year Outcomes with Transplantation of HCV-Viremic Kidneys

BACKGROUND

Recent pilot trials have demonstrated the safety of transplanting HCV-viremic kidneys into HCV-seronegative recipients. However, it remains unclear if allograft function is impacted by donor HCV-viremia or recipient HCV-serostatus.

METHODS

We used national United States registry data to examine trends in HCV-viremic kidney use between 4/1/2015 and 3/31/2019. We applied advanced matching methods to compare eGFR for similar kidneys transplanted into highly similar recipients of kidney transplants.

RESULTS

Over time, HCV-seronegative recipients received a rising proportion of HCV-viremic kidneys. During the first quarter of 2019, 200 HCV-viremic kidneys were transplanted into HCV-seronegative recipients, versus 69 into HCV-seropositive recipients, while 105 HCV-viremic kidneys were discarded. The probability of HCV-viremic kidney discard has declined over time. Kidney transplant candidates willing to accept a HCV-seropositive kidney increased from 2936 to 16,809 from during this time period. When transplanted into HCV-seronegative recipients, HCV-viremic kidneys matched to HCV-non-viremic kidneys on predictors of organ quality, except HCV, had similar 1-year eGFR (66.3 versus 67.1 ml/min per 1.73 m², P=0.86). This was despite the much worse kidney donor profile index scores assigned to the HCV-viremic kidneys. Recipient HCV-serostatus was not associated with a clinically meaningful difference in 1-year eGFR (66.5 versus 71.1 ml/min per 1.73 m², P=0.056) after transplantation of HCV-viremic kidneys.

CONCLUSIONS

By 2019, HCV-seronegative patients received the majority of kidneys transplanted from HCV-viremic donors. Widely used organ quality scores underestimated the quality of HCV-viremic kidneys based on 1-year allograft function. Recipient HCV-serostatus was also not associated with worse short-term allograft function using HCV-viremic kidneys.

A review and empirical comparison of causal inference methods for clustered observational data with application to the evaluation of the effectiveness of medical devices

Observational studies are commonplace in medicine. A frequent concern is confounding bias due to differences in patient characteristics across treatment groups, but other important issues include dependency among observations nested within clusters (e.g. patients clustered within physicians or surgeons) and confounding due to cluster characteristics (e.g. physician or surgeon experience or training). Given the frequency with which these issues arise in medical research, as well as their relative complexity, methods for the analysis of clustered observational data are reviewed. We argue for estimating causal treatment effects using marginal models that either match or weight observations using a suitable distance metric (e.g. the propensity score). Simulation results demonstrated that methods incorporating clustering into calculation of the variance were generally more accurate than those that did not. Moreover, methods that account for cluster confounding when estimating the treatment effect were least biased and most accurate. Throughout the paper we illustrate the proposed methods in a medical device setting that compares the effectiveness of femoral heads used in total hip replacements. Whenever possible the clustered aspect of the data should be considered in the design of the study when constructing the distance measure or in the matching process, as well as in the analysis when estimating the variance of the treatment effect.