A tutorial on individual participant data meta-analysis using Bayesian multilevel modeling to estimate alcohol intervention effects across heterogeneous studies

A simulation study of the performance of statistical models for count outcomes with excessive zeros

BACKGROUND

Outcome measures that are count variables with excessive zeros are common in health behaviors research. Examples include the number of standard drinks consumed or alcohol-related problems experienced over time. There is a lack of empirical data about the relative performance of prevailing statistical models for assessing the efficacy of interventions when outcomes are zero-inflated, particularly compared with recently developed marginalized count regression approaches for such data.

METHODS

The current simulation study examined five commonly used approaches for analyzing count outcomes, including two linear models (with outcomes on raw and log-transformed scales, respectively) and three prevailing count distribution-based models (ie, Poisson, negative binomial, and zero-inflated Poisson (ZIP) models). We also considered the marginalized zero-inflated Poisson (MZIP) model, a novel alternative that estimates the overall effects on the population mean while adjusting for zero-inflation. Motivated by alcohol misuse prevention trials, extensive simulations were conducted to evaluate and compare the statistical power and Type I error rate of the statistical models and approaches across data conditions that varied in sample size (N = 100 $$ N=100 $$ to 500), zero rate (0.2 to 0.8), and intervention effect sizes.

RESULTS

Under zero-inflation, the Poisson model failed to control the Type I error rate, resulting in higher than expected false positive results. When the intervention effects on the zero (vs. non-zero) and count parts were in the same direction, the MZIP model had the highest statistical power, followed by the linear model with outcomes on the raw scale, negative binomial model, and ZIP model. The performance of the linear model with a log-transformed outcome variable was unsatisfactory.

CONCLUSIONS

The MZIP model demonstrated better statistical properties in detecting true intervention effects and controlling false positive results for zero-inflated count outcomes. This MZIP model may serve as an appealing analytical approach to evaluating overall intervention effects in studies with count outcomes marked by excessive zeros.

The selection of statistical models for reporting count outcomes and intervention effects in brief alcohol intervention trials: A review and recommendations

Understanding the efficacy and relative effectiveness of a brief alcohol intervention (BAI) relies on obtaining a credible intervention effect estimate. Outcomes in BAI trials are often count variables, such as the number of drinks consumed, which may be overdispersed (i.e., greater variability than expected based on a given model) and zero-inflated (i.e., greater probability of zeros than expected based on a given model). Ignoring such distribution characteristics can lead to biased estimates and invalid statistical conclusions. In this critical review, we identified and reviewed 64 articles that reported count outcomes from a systematic review of BAI trials for adolescents and young adults from 2013 to 2018. Given many statistical models to choose from when analyzing count outcomes, we reviewed the models used and reporting practices in the BAI trial literature. A majority (61.3%) of analyses with count outcomes used linear models despite violations of normality assumptions; 75.6% of outcome variables demonstrated clear overdispersion. We provide an overview of available count models (Poisson, negative binomial, zero-inflated or hurdle, and marginalized zero-inflated Poisson regression) and formulate practical guidelines for reporting outcomes of BAIs. We provide a visual step-by-step decision guide for selecting appropriate statistical models and reporting results for count outcomes. We list accessible resources to help researchers select an appropriate model with which to analyze their data. Recent advances in count distribution-based models hold promise for evaluating count outcomes to gauge the efficacy and effectiveness of BAIs and identify critical covariates in alcohol epidemiologic research. We recommend that researchers report the distributional properties of count outcomes, such as the proportion of zero counts, and select an appropriate statistical analysis for count outcomes using the provided decision tree. By following these recommendations, future research may yield more accurate, transparent, and reproducible results.

Brief Alcohol Interventions are Effective through 6 Months: Findings from Marginalized Zero-inflated Poisson and Negative Binomial Models in a Two-step IPD Meta-analysis

To evaluate and optimize brief alcohol interventions (BAIs), it is critical to have a credible overall effect size estimate as a benchmark. Estimating such an effect size has been challenging because alcohol outcomes often represent responses from a mixture of individuals: those at high risk for alcohol misuse, occasional nondrinkers, and abstainers. Moreover, some BAIs exclusively focus on heavy drinkers, whereas others take a universal prevention approach. Depending on sample characteristics, the outcome distribution might have many zeros or very few zeros and overdispersion; consequently, the most appropriate statistical model may differ across studies. We synthesized individual participant data (IPD) from 19 studies in Project INTEGRATE (Mun et al., 2015b) that randomly allocated participants to intervention and control groups (N = 7,704 participants, 38.4% men, 74.7% White, 58.5% first-year students). We sequentially estimated marginalized zero-inflated Poisson (Long et al., 2014) or negative binomial regression models to obtain covariate-adjusted, study-specific intervention effect estimates in the first step, which were subsequently combined in a random-effects meta-analysis model in the second step. BAIs produced a statistically significant 8% advantage in the mean number of drinks at both 1-3 months (RR = 0.92, 95% CI = [0.85, 0.98]) and 6 months (RR = 0.92, 95% CI = [0.85, 0.99]) compared to controls. At 9-12 months, there was no statistically significant difference in the mean number of drinks between BAIs and controls. In conclusion, BAIs are effective at reducing the mean number of drinks through at least 6 months post intervention. IPD can play a critical role in deriving findings that could not be obtained in original individual studies or standard aggregate data meta-analyses.

Introduction to the Special Issue on Innovations and Applications of Integrative Data Analysis (IDA) and Related Data Harmonization Procedures in Prevention Science

This paper serves as an introduction to the special issue of Prevention Science entitled, "Innovations and Applications of Integrative Data Analysis (IDA) and Related Data Harmonization Procedures in Prevention Science." This special issue includes a collection of original papers from multiple disciplines that apply individual-level data synthesis methodologies, including IDA, individual participant meta-analysis, and other related methods to harmonize and integrate multiple datasets from intervention trials of the same or similar interventions. This work builds on a series of papers appearing in a prior Prevention Science special issue, entitled "Who Benefits from Programs to Prevent Adolescent Depression?" (Howe, Pantin, & Perrino, 2018). Since the publication of this prior work, the use of individual-level data synthesis has increased considerably in and outside of prevention. As such, there is a need for an update on current and future directions in IDA, with careful consideration of innovations and applications of these methods to fill important research gaps in prevention science. The papers in this issue are organized into two broad categories of (1) evidence synthesis papers that apply best practices in data harmonization and individual-level data synthesis and (2) new and emerging design, psychometric, and methodological issues and solutions. This collection of original papers is followed by two invited commentaries which provide insight and important reflections on the field and future directions for prevention science.

Which is Better for Individual Participant Data Meta-Analysis of Zero-Inflated Count Outcomes, One-Step or Two-Step Analysis? A Simulation Study

Meta-analysis using individual participant data (IPD) is an important methodology in intervention research because it (a) increases accuracy and precision of estimates, (b) allows researchers to investigate mediators and moderators of treatment effects, and (c) makes use of extant data. IPD meta-analysis can be conducted either via a one-step approach that uses data from all studies simultaneously, or a two-step approach, which aggregates data for each study and then combines them in a traditional meta-analysis model. Unfortunately, there are no evidence-based guidelines for how best to approach IPD meta-analysis for count outcomes with many zeroes, such as alcohol use. We used simulation to compare the performance of four hurdle models (3 one-step and 1 two-step models) for zero-inflated count IPD, under realistic data conditions. Overall, all models yielded adequate coverage and bias for the treatment effect in the count portion of the model, across all data conditions. However, in the zero portion, the treatment effect was underestimated in most models and data conditions, especially when there were fewer studies. The performance of both one- and two-step approaches depended on the formulation of the treatment effects, suggesting a need to carefully consider model assumptions and specifications when using IPD.

Do brief motivational interventions increase motivation for change in drinking among college students? A two-step meta-analysis of individual participant data

BACKGROUND

Brief motivational interventions (BMIs) are one of the most effective individually focused alcohol intervention strategies for college students. Despite the central theoretical role of motivation for change in BMIs, it is unclear whether BMIs increase motivation to change drinking behavior. We conducted a two-step meta-analysis of individual participant data (IPD) to examine whether BMIs increase motivation for change. N = 5903;59% women, 72% White) from Project INTEGRATE. The BMIs included individually delivered motivational interviewing with personalized feedback (MI + PF), stand-alone personalized feedback (PF), and group-based motivational interviewing (GMI).

METHODS

We included 15 trials of BMI (N = 5903;59% women, 72% White) from Project INTEGRATE. The BMIs included individually-delivered motivational interviewing with personalized feedback (MI + PF), stand-alone personalized feedback (PF), and group-based motivational interviewing (GMI). Different measures and responses used in the original trials were harmonized. Effect size estimates were derived from a model that adjusted for baseline motivation and demographic variables for each trial (step 1) and subsequently combined in a random-effects meta-analysis (step 2).

RESULTS

The overall intervention effect of BMIs on motivation for change was not statistically significant (standard mean difference [SMD]: 0.026, 95% CI: [-0.001, 0.053], p = 0.06, k = 19 comparisons). Of the three subtypes of BMIs, GMI, which tended to provide motivation-targeted content, had a statistically significant intervention effect on motivation, compared with controls (SMD: 0.055, 95% CI: [0.007, 0.103], p = 0.025, k = 5). By contrast, there was no evidence that MI + PF (SMD = 0.04, 95% CI: [-0.02, 0.10], k = 6, p = 0.20) nor PF increased motivation (SMD = 0.005, 95% CI: [-0.028, 0.039], k = 8, p = 0.75), compared with controls. Post hoc meta-regression analysis suggested that motivation sharply decreased each month within the first 3 months postintervention (b = -0.050, z = -2.80, p = 0.005 for k = 14).

CONCLUSIONS

Although BMIs provide motivational content and normative feedback and are assumed to motivate behavior change, the results do not wholly support the hypothesis that BMIs improve motivation for change. Changing motivation is difficult to assess during and following interventions, but it is still a theoretically important clinical endpoint. Further, the evidence cautiously suggests that changing motivation may be achievable, especially if motivation-targeted content components are provided.

Cross-Tailoring Integrative Alcohol and Risky Sexual Behavior Feedback for College Students: Protocol for a Hybrid Type 1 Effectiveness-Implementation Trial

BACKGROUND

Underage drinking and related risky sexual behavior (RSB) are major public health concerns on United States college campuses. Although technology-delivered personalized feedback interventions (PFIs) are considered a best practice for individual-level campus alcohol prevention, there is room for improving the effectiveness of this approach with regard to alcohol-related RSB.

OBJECTIVE

The aims of this study are to (1) evaluate the impact of a brief PFI that integrates content on alcohol use and RSB and is adapted to include a novel cross-tailored dynamic feedback (CDF) component for at-risk first-year college students and (2) identify implementation factors critical to the CDF's success to facilitate future scale-up in campus settings.

METHODS

This study uses a hybrid type 1 effectiveness-implementation design and will be conducted in 3 phases. Phase 1 is a stakeholder-engaged PFI+CDF adaptation guided by focus groups and usability testing. In phase 2, 600 first-year college students who drink and are sexually active will be recruited from 2 sites (n=300 per site) to participate in a 4-group randomized controlled trial to examine the effectiveness of PFI+CDF in reducing alcohol-related RSB. Eligible participants will complete a baseline survey during the first week of the semester and follow-up surveys at 1, 2, 3, 6, and 13 months post baseline. Phase 3 is a qualitative evaluation with stakeholders to better understand relevant implementation factors.

RESULTS

Recruitment and enrollment for phase 1 began in January 2022. Recruitment for phases 2 and 3 is planned for the summer of 2023 and 2024, respectively. Upon collection of data, the effectiveness of PFI+CDF will be examined, and factors critical to implementation will be evaluated.

CONCLUSIONS

This hybrid type 1 trial is designed to impact the field by testing an innovative adaptation that extends evidence-based alcohol programs to reduce alcohol-related RSB and provides insights related to implementation to bridge the gap between research and practice at the university level.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05011903; https://clinicaltrials.gov/ct2/show/NCT05011903.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/43986.

Optimizing cognitive and behavioral approaches for perinatal depression: A systematic review and meta-regression analysis

Cognitive behavioral therapies (CBT) have been demonstrated efficacious in treating perinatal depression (PND). This has been demonstrated in several meta-analyses of randomized controlled trials and quasi-experimental studies. However, there is a need for up-to-date meta-analytical evidence providing reliable estimates for CBT's effectiveness in treating and preventing PND. Furthermore, with the world moving toward precision medicine, approaches require a critical synthesis of psychotherapies, especially to unpack their mechanisms of action and to understand what approaches work best for whom. Therefore, the present systematic review and meta-regression analyses aim to answer these research questions. We searched six academic databases through February 2022 and identified 56 studies for an in-depth review. Using pretested data extraction sheets, we extracted patient-level and intervention-level characteristics and effect size data from each study. Random-effects meta-analyses and mixed-effect subgroup analyses were run to delineate the effectiveness and moderators of CBT interventions for PND, respectively. CBT-based interventions yielded a strong effect size (SMD = -0.74, 95% confidence interval [CI]: -0.91 to -0.56, n = 9,722) in alleviating depressive symptoms. These interventions were effective across different delivery formats (individual, group, and electronic) and could be delivered effectively by specialists and nonspecialists. Longer duration CBT interventions may not necessarily be more effective than shorter ones. Moreover, CBT-based interventions should consider including various behavioral ingredients to maximize intervention benefits.

A bias correction method in meta-analysis of randomized clinical trials with no adjustments for zero-inflated outcomes

Many clinical endpoint measures, such as the number of standard drinks consumed per week or the number of days that patients stayed in the hospital, are count data with excessive zeros. However, the zero‐inflated nature of such outcomes is sometimes ignored in analyses of clinical trials. This leads to biased estimates of study‐level intervention effect and, consequently, a biased estimate of the overall intervention effect in a meta‐analysis. The current study proposes a novel statistical approach, the Zero‐inflation Bias Correction (ZIBC) method, that can account for the bias introduced when using the Poisson regression model, despite a high rate of inflated zeros in the outcome distribution of a randomized clinical trial. This correction method only requires summary information from individual studies to correct intervention effect estimates as if they were appropriately estimated using the zero‐inflated Poisson regression model, thus it is attractive for meta‐analysis when individual participant‐level data are not available in some studies. Simulation studies and real data analyses showed that the ZIBC method performed well in correcting zero‐inflation bias in most situations.

A Structural Equation Modeling Approach to Meta-analytic Mediation Analysis Using Individual Participant Data: Testing Protective Behavioral Strategies as a Mediator of Brief Motivational Intervention Effects on Alcohol-Related Problems

This paper introduces a meta-analytic mediation analysis approach for individual participant data (IPD) from multiple studies. Mediation analysis evaluates whether the effectiveness of an intervention on health outcomes occurs because of change in a key behavior targeted by the intervention. However, individual trials are often statistically underpowered to test mediation hypotheses. Existing approaches for evaluating mediation in the meta-analytic context are limited by their reliance on aggregate data; thus, findings may be confounded with study-level differences unrelated to the pathway of interest. To overcome the limitations of existing meta-analytic mediation approaches, we used a one-stage estimation approach using structural equation modeling (SEM) to combine IPD from multiple studies for mediation analysis. This approach (1) accounts for the clustering of participants within studies, (2) accommodates missing data via multiple imputation, and (3) allows valid inferences about the indirect (i.e., mediated) effects via bootstrapped confidence intervals. We used data (N = 3691 from 10 studies) from Project INTEGRATE (Mun et al. Psychology of Addictive Behaviors, 29, 34–48, 2015) to illustrate the SEM approach to meta-analytic mediation analysis by testing whether improvements in the use of protective behavioral strategies mediate the effectiveness of brief motivational interventions for alcohol-related problems among college students. To facilitate the application of the methodology, we provide annotated computer code in R and data for replication. At a substantive level, stand-alone personalized feedback interventions reduced alcohol-related problems via greater use of protective behavioral strategies; however, the net-mediated effect across strategies was small in size, on average.

Enhanced Brief Motivational Intervention for College Student Drinkers With ADHD: Goal-Directed Activation as a Mechanism of Change

College students with attention-deficit/hyperactivity disorder (ADHD) are at risk for alcohol-related problems and disorders relative to their typically developing peers. Despite risk, the optimal therapeutic approach for reducing problem alcohol use in students with ADHD, and mechanisms of change underlying treatment effects in this population, are largely unknown. The current study evaluated putative mechanisms of change in a randomized controlled trial of two harm reduction interventions for college student drinkers with ADHD (N = 113; 49% male): brief motivational intervention plus supportive counseling (BMI + SC) versus brief motivational intervention plus behavioral activation (BMI + BA). Results showed that participants in the BMI + BA condition engaged in more goal-directed activation and less avoidant behavior over the course of treatment compared to those in the BMI + SC condition, in turn predicting reductions in alcohol-related negative consequences. Effects were more robust 1 month following intervention, and diminished by 3 months. Sensitivity analyses revealed a significant indirect effect of treatment condition on alcohol-related negative consequence via reductions in avoidance over treatment. Post hoc moderated mediations showed that BMI + BA engaged target mechanisms more robustly for students with more severe ADHD and depressive symptoms compared to BMI + SC. These findings support the application of BMI + BA intervention, particularly in targeting goal-directed activation and avoidance/rumination in at-risk student drinkers with ADHD.

Do Brief Alcohol Interventions Reduce Driving After Drinking Among College Students? A Two-step Meta-analysis of Individual Participant Data

Aims

College students who drink are at an increased risk of driving after drinking and alcohol-involved traffic accidents and deaths. Furthermore, the persistence of driving after drinking over time underscores a need for effective interventions to prevent future drunk driving in adulthood. The present study examined whether brief alcohol interventions (BAIs) for college students reduce driving after drinking.

Methods

A two-step meta-analysis of individual participant data (IPD) was conducted using a combined sample of 6801 college students from 15 randomized controlled trials (38% male, 72% White and 58% first-year students). BAIs included individually delivered Motivational Interviewing with Personalized Feedback (MI + PF), Group Motivational Interviewing (GMI), and stand-alone Personalized Feedback (PF) interventions. Two outcome variables, driving after two+/three+ drinks and driving after four+/five+ drinks, were checked, harmonized and analyzed separately for each study and then combined for meta-analysis and meta-regression analysis.

Results

BAIs lowered the risk of driving after four+/five+ drinks (19% difference in the odds of driving after drinking favoring BAIs vs. control), but not the risk of driving after two+/three+ drinks (9% difference). Subsequent subgroup analysis indicated that the MI + PF intervention was comparatively better than PF or GMI.

Conclusions

BAIs provide a harm reduction approach to college drinking. Hence, it is encouraging that BAIs reduce the risk of driving after heavy drinking among college students. However, there may be opportunities to enhance the intervention content and timing to be more relevant for driving after drinking and improve the outcome assessment and reporting to demonstrate its effect.

Relative efficiency of using summary versus individual data in random-effects meta-analysis

Meta-analysis is a statistical methodology for combining information from diverse sources so that a more reliable and efficient conclusion can be reached. It can be conducted by either synthesizing study-level summary statistics or drawing inference from an overarching model for individual participant data (IPD) if available. The latter is often viewed as the "gold standard." For random-effects models, however, it remains not fully understood whether the use of IPD indeed gains efficiency over summary statistics. In this paper, we examine the relative efficiency of the two methods under a general likelihood inference setting. We show theoretically and numerically that summary-statistics-based analysis is at most as efficient as IPD analysis, provided that the random effects follow the Gaussian distribution, and maximum likelihood estimation is used to obtain summary statistics. More specifically, (i) the two methods are equivalent in an asymptotic sense; and (ii) summary-statistics-based inference can incur an appreciable loss of efficiency if the sample sizes are not sufficiently large. Our results are established under the assumption that the between-study heterogeneity parameter remains constant regardless of the sample sizes, which is different from a previous study. Our findings are confirmed by the analyses of simulated data sets and a real-world study of alcohol interventions.

A CD-based mapping method for combining multiple related parameters from heterogeneous intervention trials

Effect size can differ as a function of the elapsed time since treatment or as a function of other key covariates, such as sex or age. In evidence synthesis, a better understanding of the precise conditions under which treatment does work or does not work well has been highly valued. With increasingly accessible individual patient or participant data (IPD), more precise and informative inference can be within our reach. However, simultaneously combining multiple related parameters across heterogeneous studies is challenging because each parameter from each study has a specific interpretation within the context of the study and other covariates in the model. This paper proposes a novel mapping method to combine study-specific estimates of multiple related parameters across heterogeneous studies, which ensures valid inference at all inference levels by combining sample-dependent functions known as Confidence Distributions (CD). We describe the "CD-based mapping method" and provide a data application example for a multivariate random-effects meta-analysis model. We estimated up to 13 study-specific regression parameters for each of 14 individual studies using IPD in the first step, and subsequently combined the study-specific vectors of parameters, yielding a full vector of hyperparameters in the second step of meta-analysis. Sensitivity analysis indicated that the CD-based mapping method is robust to model misspecification. This novel approach to multi-parameter synthesis provides a reasonable methodological solution when combining complex evidence using IPD.

A tutorial on individual participant data meta-analysis using Bayesian multilevel modeling to estimate alcohol intervention effects across heterogeneous studies

This paper provides a tutorial companion for the methodological approach implemented in Huh et al. (2015) that overcame two major challenges for individual participant data (IPD) meta-analysis. Specifically, we show how to validly combine data from heterogeneous studies with varying numbers of treatment arms, and how to analyze highly-skewed count outcomes with many zeroes (e.g., alcohol and substance use outcomes) to estimate overall effect sizes. These issues have important implications for the feasibility, applicability, and interpretation of IPD meta-analysis but have received little attention thus far in the applied research literature. We present a Bayesian multilevel modeling approach for combining multi-arm trials (i.e., those with two or more treatment groups) in a distribution-appropriate IPD analysis. Illustrative data come from Project INTEGRATE, an IPD meta-analysis study of brief motivational interventions to reduce excessive alcohol use and related harm among college students. Our approach preserves the original random allocation within studies, combines within-study estimates across all studies, overcomes between-study heterogeneity in trial design (i.e., number of treatment arms) and/or study-level missing data, and derives two related treatment outcomes in a multivariate IPD meta-analysis. This methodological approach is a favorable alternative to collapsing or excluding intervention groups within multi-arm trials, making it possible to directly compare multiple treatment arms in a one-step IPD meta-analysis. To facilitate application of the method, we provide annotated computer code in R along with the example data used in this tutorial.