Expanding outcomes when considering the relative effectiveness of two evidence-based outpatient treatment programs for adolescents

A tutorial comparing different covariate balancing methods with an application evaluating the causal effects of substance use treatment programs for adolescents

Randomized controlled trials are the gold standard for measuring causal effects. However, they are often not always feasible, and causal treatment effects must be estimated from observational data. Observational studies do not allow robust conclusions about causal relationships unless statistical techniques account for the imbalance of pretreatment confounders across groups and key assumptions hold. Propensity score and balance weighting (PSBW) are useful techniques that aim to reduce the observed imbalances between treatment groups by weighting the groups to look alike on the observed confounders. Notably, there are many methods available to estimate PSBW. However, it is unclear a priori which will achieve the best trade-off between covariate balance and effective sample size for a given application. Moreover, it is critical to assess the validity of key assumptions required for robust estimation of the needed treatment effects, including the overlap and no unmeasured confounding assumptions. We present a step-by-step guide to the use of PSBW for estimation of causal treatment effects that includes steps on how to evaluate overlap before the analysis, obtain estimates of PSBW using multiple methods and select the optimal one, check for covariate balance on multiple metrics, and assess sensitivity of findings (both the estimated treatment effect and statistical significance) to unobserved confounding. We illustrate the key steps using a case study examining the relative effectiveness of substance use treatment programs and provide a user-friendly Shiny application that can implement the proposed steps for any application with binary treatments.

Volunteer Recovery Support for Adolescents: Using propensity score based methods to understand dosage effects within a randomized controlled trial

BACKGROUND

In a recently published randomized controlled trial (RCT) of Volunteer Recovery Support for Adolescents (VRSA), a secondary finding indicated that better adherence to planned VRSA telephone session frequency resulted in significantly higher remission rates relative to lower session adherence. However, interpretation of this dose-response relationship may have been confounded by participant characteristics such as baseline levels of substance use and mental health problems.

METHODS

The present study used statistical methods designed to approximate RCTs when comparing more than two nonequivalent groups that include an assessment of the potential impact of omitted variables. Classification and Regression Tree (CRT) analysis was used to establish the cut-point between high (H) and low (L) VRSA dosage groups. Because we were interested in generalizing to youth with poor attendance, the L-VRSA group served as the reference group. Balancing weights for H-VRSA and a services as usual (SAU) control group were calculated to ensure similarity of baseline pretreatment characteristics to the reference group, and sensitivity of findings to unobserved confounding variables was assessed.

RESULTS

Findings suggested that superior remission rates at the end of the intervention phase were the result of high adherence to planned VRSA session frequency. Recommendations to achieve high VRSA participation among a larger segment of youth and to test whether longer VRSA duration improves the stability of recovery outcomes are provided.

CONCLUSION

Few published dose-response studies have adequately controlled for selection confounds from both observed and unobserved confounding. As such, the present study aims to both assess the impact of different dosage levels of VRSA and provide a template for how to apply state-of-the-art statistical methods designed to approximate randomized controlled trials to such studies.

Best practices in statistical computing

The world is becoming increasingly complex, both in terms of the rich sources of data we have access to and the statistical and computational methods we can use on data. These factors create an ever-increasing risk for errors in code and the sensitivity of findings to data preparation and the execution of complex statistical and computing methods. The consequences of coding and data mistakes can be substantial. In this paper, we describe the key steps for implementing a code quality assurance (QA) process that researchers can follow to improve their coding practices throughout a project to assure the quality of the final data, code, analyses, and results. These steps include: (i) adherence to principles for code writing and style that follow best practices; (ii) clear written documentation that describes code, workflow, and key analytic decisions; (iii) careful version control; (iv) good data management; and (v) regular testing and review. Following these steps will greatly improve the ability of a study to assure results are accurate and reproducible. The responsibility for code QA falls not only on individual researchers but institutions, journals, and funding agencies as well.

Nonparametric Estimation of Population Average Dose-Response Curves using Entropy Balancing Weights for Continuous Exposures

Weighted estimators are commonly used for estimating exposure effects in observational settings to establish causal relations. These estimators have a long history of development when the exposure of interest is binary and where the weights are typically functions of an estimated propensity score. Recent developments in optimization-based estimators for constructing weights in binary exposure settings, such as those based on entropy balancing, have shown more promise in estimating treatment effects than those methods that focus on the direct estimation of the propensity score using likelihood-based methods. This paper explores recent developments of entropy balancing methods to continuous exposure settings and the estimation of population dose-response curves using nonparametric estimation combined with entropy balancing weights, focusing on factors that would be important to applied researchers in medical or health services research. The methods developed here are applied to data from a study assessing the effect of non-randomized components of an evidence-based substance use treatment program on emotional and substance use clinical outcomes.