Using Ecometric Data to Explore Sources of Cross-Site Impact Variance in Multi-Site Trials

In 2003, Bloom, Hill, and Riccio (BHR) published an influential paper introducing novel methods for explaining the variation in local impacts observed in multi-site randomized control trials of socio-economic interventions in terms of site-level mediators. This paper seeks to improve upon this previous work by using student-level data to measure site-level mediators and confounders. Development of asymptotic behavior backed up with simulations and an empirical example. Students and training providers. Two simulations and an empirical application to data from an evaluation of the Health Professions Opportunity Grants (HPOG) Program. This empirical analysis involved roughly 6600 participants across 37 local sites. We examine bias and mean square error of estimates of mediation coefficients as well as the true coverage of nominal 95-percent confidence intervals on the mediation coefficients. Simulations suggest that the new methods generally improve the quality of inferences even when there is no confounding. Applying this methodology to the HPOG study shows that program-average FTE months of study by month six was a significant mediator of both career progress and long-term degree/credential receipt. Evaluators can robustify their BHR-style analyses by the use of the methods proposed here.

Propensity Score Weighting with Missing Data on Covariates and Clustered Data Structure

Propensity score (PS) analyses are increasingly popular in behavioral sciences. Two issues often add complexities to PS analyses, including missing data in observed covariates and clustered data structure. In previous research, methods for conducting PS analyses with considering either issue alone were examined. In practice, the two issues often co-occur; but the performance of methods for PS analyses in the presence of both issues has not been evaluated previously. In this study, we consider PS weighting analysis when data are clustered and observed covariates have missing values. A simulation study is conducted to evaluate the performance of different missing data handling methods (complete-case, single-level imputation, or multilevel imputation) combined with different multilevel PS weighting methods (fixed- or random-effects PS models, inverse-propensity-weighting or the clustered weighting, weighted single-level or multilevel outcome models). The results suggest that the bias in average treatment effect estimation can be reduced, by better accounting for clustering in both the missing data handling stage (such as with the multilevel imputation) and the PS analysis stage (such as with the fixed-effects PS model, clustered weighting, and weighted multilevel outcome model). A real-data example is provided for illustration.

Designing Optimal, Data-Driven Policies from Multisite Randomized Trials

Optimal treatment regimes (OTRs) have been widely employed in computer science and personalized medicine to provide data-driven, optimal recommendations to individuals. However, previous research on OTRs has primarily focused on settings that are independent and identically distributed, with little attention given to the unique characteristics of educational settings, where students are nested within schools and there are hierarchical dependencies. The goal of this study is to propose a framework for designing OTRs from multisite randomized trials, a commonly used experimental design in education and psychology to evaluate educational programs. We investigate modifications to popular OTR methods, specifically Q-learning and weighting methods, in order to improve their performance in multisite randomized trials. A total of 12 modifications, 6 for Q-learning and 6 for weighting, are proposed by utilizing different multilevel models, moderators, and augmentations. Simulation studies reveal that all Q-learning modifications improve performance in multisite randomized trials and the modifications that incorporate random treatment effects show the most promise in handling cluster-level moderators. Among weighting methods, the modification that incorporates cluster dummies into moderator variables and augmentation terms performs best across simulation conditions. The proposed modifications are demonstrated through an application to estimate an OTR of conditional cash transfer programs using a multisite randomized trial in Colombia to maximize educational attainment.

Multiple imputation of missing data in multilevel ecological momentary assessments: an example using smoking cessation study data

Advances in digital technology have greatly increased the ease of collecting intensive longitudinal data (ILD) such as ecological momentary assessments (EMAs) in studies of behavior changes. Such data are typically multilevel (e.g., with repeated measures nested within individuals), and are inevitably characterized by some degrees of missingness. Previous studies have validated the utility of multiple imputation as a way to handle missing observations in ILD when the imputation model is properly specified to reflect time dependencies. In this study, we illustrate the importance of proper accommodation of multilevel ILD structures in performing multiple imputations, and compare the performance of a multilevel multiple imputation (multilevel MI) approach relative to other approaches that do not account for such structures in a Monte Carlo simulation study. Empirical EMA data from a tobacco cessation study are used to demonstrate the utility of the multilevel MI approach, and the implications of separating participant- and study-initiated EMAs in evaluating individuals' affective dynamics and urge.

Review of Semiconductor Flash Memory Devices for Material and Process Issues

Vertically integrated NAND (V-NAND) flash memory is the main data storage in modern handheld electronic devices, widening its share even in the data centers where installation and operation costs are critical. While the conventional scaling rule has been applied down to the design rule of ≈15 nm (year 2013), the current method of increasing device density is stacking up layers. Currently, 176-layer-stacked V-NAND flash memory is available on the market. Nonetheless, increasing the layers invokes several challenges, such as film stress management and deep contact hole etching. Also, there should be an upper bound for the attainable stacking layers (400-500) due to the total allowable chip thickness, which will be reached within 6-7 years. This review summarizes the current status and critical challenges of charge-trap-based flash memory devices, with a focus on the material (floating-gate vs charge-trap-layer), array-level circuit architecture (NOR vs NAND), physical integration structure (2D vs 3D), and cell-level programming technique (single vs multiple levels). Current efforts to improve fabrication processes and device performances using new materials are also introduced. The review suggests directions for future storage devices based on the ionic mechanism, which may overcome the inherent problems of flash memory devices.

Stan and BART for Causal Inference: Estimating Heterogeneous Treatment Effects Using the Power of Stan and the Flexibility of Machine Learning

A wide range of machine-learning-based approaches have been developed in the past decade, increasing our ability to accurately model nonlinear and nonadditive response surfaces. This has improved performance for inferential tasks such as estimating average treatment effects in situations where standard parametric models may not fit the data well. These methods have also shown promise for the related task of identifying heterogeneous treatment effects. However, the estimation of both overall and heterogeneous treatment effects can be hampered when data are structured within groups if we fail to correctly model the dependence between observations. Most machine learning methods do not readily accommodate such structure. This paper introduces a new algorithm, stan4bart, that combines the flexibility of Bayesian Additive Regression Trees (BART) for fitting nonlinear response surfaces with the computational and statistical efficiencies of using Stan for the parametric components of the model. We demonstrate how stan4bart can be used to estimate average, subgroup, and individual-level treatment effects with stronger performance than other flexible approaches that ignore the multilevel structure of the data as well as multilevel approaches that have strict parametric forms.

Causal Inference with Multilevel Data: A Comparison of Different Propensity Score Weighting Approaches

Propensity score methods are a widely recommended approach to adjust for confounding and to recover treatment effects with non-experimental, single-level data. This article reviews propensity score weighting estimators for multilevel data in which individuals (level 1) are nested in clusters (level 2) and nonrandomly assigned to either a treatment or control condition at level 1. We address the choice of a weighting strategy (inverse probability weights, trimming, overlap weights, calibration weights) and discuss key issues related to the specification of the propensity score model (fixed-effects model, multilevel random-effects model) in the context of multilevel data. In three simulation studies, we show that estimates based on calibration weights, which prioritize balancing the sample distribution of level-1 and (unmeasured) level-2 covariates, should be preferred under many scenarios (i.e., treatment effect heterogeneity, presence of strong level-2 confounding) and can accommodate covariate-by-cluster interactions. However, when level-1 covariate effects vary strongly across clusters (i.e., under random slopes), and this variation is present in both the treatment and outcome data-generating mechanisms, large cluster sizes are needed to obtain accurate estimates of the treatment effect. We also discuss the implementation of survey weights and present a real-data example that illustrates the different methods.

glca: An R Package for Multiple-Group Latent Class Analysis

Group similarities and differences may manifest themselves in a variety of ways in multiple-group latent class analysis (LCA). Sometimes, measurement models are identical across groups in LCA. In other situations, the measurement models may differ, suggesting that the latent structure itself is different between groups. Tests of measurement invariance shed light on this distinction. We created an R package glca that implements procedures for exploring differences in latent class structure between populations, taking multilevel data structure into account. The glca package deals with the fixed-effect LCA and the nonparametric random-effect LCA; the former can be applied in the situation where populations are segmented by the observed group variable itself, whereas the latter can be used when there are too many levels in the group variable to make a meaningful group comparisons by identifying a group-level latent variable. The glca package consists of functions for statistical test procedures for exploring group differences in various LCA models considering multilevel data structure.

The Impact of Ignoring a Crossed Factor in Cross-Classified Multilevel Modeling

The present study investigated estimate biases in cross-classified random effect modeling (CCREM) and hierarchical linear modeling (HLM) when ignoring a crossed factor in CCREM considering the impact of the feeder and the magnitude of coefficients. There were six simulation factors: the magnitude of coefficient, the correlation between the level 2 residuals, the number of groups, the average number of individuals sampled from each group, the intra-unit correlation coefficient, and the number of feeders. The targeted interests of the coefficients were four fixed effects and two random effects. The results showed that ignoring a crossed factor in cross-classified data causes a parameter bias for the random effects of level 2 predictors and a standard error bias for the fixed effects of intercepts, level 1 predictors, and level 2 predictors. Bayesian information criteria generally outperformed Akaike information criteria in detecting the correct model.

Modelling inter-individual differences in latent within-person variation: The confirmatory factor level variability model

Psychological theories often produce hypotheses that pertain to individual differences in within-person variability. To empirically test the predictions entailed by such hypotheses with longitudinal data, researchers often use multilevel approaches that allow them to model between-person differences in the mean level of a certain variable and the residual within-person variance. Currently, these approaches can be applied only when the data stem from a single variable. However, it is common practice in psychology to assess not just a single measure but rather several measures of a construct. In this paper we describe a model in which we combine the single-indicator model with confirmatory factor analysis. The new model allows individual differences in latent mean-level factors and latent within-person variability factors to be estimated. Furthermore, we show how the model's parameters can be estimated with a maximum likelihood estimator, and we illustrate the approach using an example that involves intensive longitudinal data.

An Innovative Concept for a Multivariate Plausibility Assessment of Simultaneously Recorded Data

The aim of this work was to develop an innovative multivariate plausibility assessment (MPA) algorithm in order to differentiate between 'physiologically normal', 'physiologically extreme' and 'implausible' observations in simultaneously recorded data. The underlying concept is based on the fact that different measurable parameters are often physiologically linked. If physiologically extreme observations occur due to disease, incident or hormonal cycles, usually more than one measurable trait is affected. In contrast, extreme values of a single trait are most likely implausible if all other traits show values in a normal range. For demonstration purposes, the MPA was applied on a time series data set which was collected on 100 cows in 10 commercial dairy farms. Continuous measurements comprised climate data, intra-reticular pH and temperature, jaw movement and locomotion behavior. Non-continuous measurements included milk yield, milk components, milk mid-infrared spectra and blood parameters. After the application of the MPA, in particular the pH data showed the most implausible observations with approximately 5% of the measured values. The other traits showed implausible values up to 2.5%. The MPA showed the ability to improve the data quality for downstream analyses by detecting implausible observations and to discover physiologically extreme conditions even within complex data structures. At this stage, the MPA is not a fully developed and validated management tool, but rather corresponds to a basic concept for future works, which can be extended and modified as required.

Generalized linear mixed-model (GLMM) trees: A flexible decision-tree method for multilevel and longitudinal data

Objective: Decision-tree methods are machine-learning methods which provide results that are relatively easy to interpret and apply by human decision makers. The resulting decision trees show how baseline patient characteristics can be combined to predict treatment outcomes for individual patients, for example. This paper introduces GLMM trees, a decision-tree method for multilevel and longitudinal data. Method: To illustrate, we apply GLMM trees to a dataset of 3,256 young people (mean age 11.33, 48% girls) receiving treatment at one of several mental-health service providers in the UK. Two treatment outcomes (mental-health difficulties scores corrected for baseline) were regressed on 18 demographic, case and severity characteristics at baseline. We compared the performance of GLMM trees with that of traditional GLMMs and random forests. Results: GLMM trees yielded modest predictive accuracy, with cross-validated multiple R values of .18 and .25. Predictive accuracy did not differ significantly from that of traditional GLMMs and random forests, while GLMM trees required evaluation of a lower number of variables. Conclusion: GLMM trees provide a useful data-analytic tool for clinical prediction problems. The supplemental material provides a tutorial for replicating the GLMM tree analyses in R.

Comparing a multivariate response Bayesian random effects logistic regression model with a latent variable item response theory model for provider profiling on multiple binary indicators simultaneously

Provider profiling entails comparing the performance of hospitals on indicators of quality of care. Many common indicators of healthcare quality are binary (eg, short‐term mortality, use of appropriate medications). Typically, provider profiling examines the variation in each indicator in isolation across hospitals. We developed Bayesian multivariate response random effects logistic regression models that allow one to simultaneously examine variation and covariation in multiple binary indicators across hospitals. Use of this model allows for (i) determining the probability that a hospital has poor performance on a single indicator; (ii) determining the probability that a hospital has poor performance on multiple indicators simultaneously; (iii) determining, by using the Mahalanobis distance, how far the performance of a given hospital is from that of an average hospital. We illustrate the utility of the method by applying it to 10 881 patients hospitalized with acute myocardial infarction at 102 hospitals. We considered six binary patient‐level indicators of quality of care: use of reperfusion, assessment of left ventricular ejection fraction, measurement of cardiac troponins, use of acetylsalicylic acid within 6 hours of hospital arrival, use of beta‐blockers within 12 hours of hospital arrival, and survival to 30 days after hospital admission. When considering the five measures evaluating processes of care, we found that there was a strong correlation between a hospital's performance on one indicator and its performance on a second indicator for five of the 10 possible comparisons. We compared inferences made using this approach with those obtained using a latent variable item response theory model.

The Impact of Test and Sample Characteristics on Model Selection and Classification Accuracy in the Multilevel Mixture IRT Model

The standard item response theory (IRT) model assumption of a single homogenous population may be violated in real data. Mixture extensions of IRT models have been proposed to account for latent heterogeneous populations, but these models are not designed to handle multilevel data structures. Ignoring the multilevel structure is problematic as it results in lower-level units aggregated with higher-level units and yields less accurate results, because of dependencies in the data. Multilevel data structures cause such dependencies between levels but can be modeled in a straightforward way in multilevel mixture IRT models. An important step in the use of multilevel mixture IRT models is the fit of the model to the data. This fit is often determined based on relative fit indices. Previous research on mixture IRT models has shown that performances of these indices and classification accuracy of these models can be affected by several factors including percentage of class-variant items, number of items, magnitude and size of clusters, and mixing proportions of latent classes. As yet, no studies appear to have been reported examining these issues for multilevel extensions of mixture IRT models. The current study aims to investigate the effects of several features of the data on the accuracy of model selection and parameter recovery. Results are reported on a simulation study designed to examine the following features of the data: percentages of class-variant items (30, 60, and 90%), numbers of latent classes in the data (with from 1 to 3 latent classes at level 1 and 1 and 2 latent classes at level 2), numbers of items (10, 30, and 50), numbers of clusters (50 and 100), cluster size (10 and 50), and mixing proportions [equal (0.5 and 0.5) vs. non-equal (0.25 and 0.75)]. Simulation results indicated that multilevel mixture IRT models resulted in less accurate estimates when the number of clusters and the cluster size were small. In addition, mean Root mean square error (RMSE) values increased as the percentage of class-variant items increased and parameters were recovered more accurately under the 30% class-variant item conditions. Mixing proportion type (i.e., equal vs. unequal latent class sizes) and numbers of items (10, 30, and 50), however, did not show any clear pattern. Sample size dependent fit indices BIC, CAIC, and SABIC performed poorly for the smaller level-1 sample size. For the remaining conditions, the SABIC index performed better than other fit indices.

Missing data strategies for time-varying confounders in comparative effectiveness studies of non-missing time-varying exposures and right-censored outcomes

The treatment of missing data in comparative effectiveness studies with right-censored outcomes and time-varying covariates is challenging because of the multilevel structure of the data. In particular, the performance of an accessible method like multiple imputation (MI) under an imputation model that ignores the multilevel structure is unknown and has not been compared to complete-case (CC) and single imputation methods that are most commonly applied in this context. Through an extensive simulation study, we compared statistical properties among CC analysis, last value carried forward, mean imputation, the use of missing indicators, and MI-based approaches with and without auxiliary variables under an extended Cox model when the interest lies in characterizing relationships between non-missing time-varying exposures and right-censored outcomes. MI demonstrated favorable properties under a moderate missing-at-random condition (absolute bias <0.1) and outperformed CC and single imputation methods, even when the MI method did not account for correlated observations in the imputation model. The performance of MI decreased with increasing complexity such as when the missing data mechanism involved the exposure of interest, but was still preferred over other methods considered and performed well in the presence of strong auxiliary variables. We recommend considering MI that ignores the multilevel structure in the imputation model when data are missing in a time-varying confounder, incorporating variables associated with missingness in the MI models as well as conducting sensitivity analyses across plausible assumptions.

The next-generation K-means algorithm

Typically, when referring to a model-based classification, the mixture distribution approach is understood. In contrast, we revive the hard-classification model-based approach developed by Banfield and Raftery (1993) for which K-means is equivalent to the maximum likelihood (ML) estimation. The next-generation K-means algorithm does not end after the classification is achieved, but moves forward to answer the following fundamental questions: Are there clusters, how many clusters are there, what are the statistical properties of the estimated means and index sets, what is the distribution of the coefficients in the clusterwise regression, and how to classify multilevel data? The statistical model-based approach for the K-means algorithm is the key, because it allows statistical simulations and studying the properties of classification following the track of the classical statistics. This paper illustrates the application of the ML classification to testing the no-clusters hypothesis, to studying various methods for selection of the number of clusters using simulations, robust clustering using Laplace distribution, studying properties of the coefficients in clusterwise regression, and finally to multilevel data by marrying the variance components model with K-means.

Cost-efficient designs for three-arm trials with treatment delivered by health professionals: Sample sizes for a combination of nested and crossed designs

Background This article studies the design of trials that compare three treatment conditions that are delivered by two types of health professionals. The one type of health professional delivers one treatment, and the other type delivers two treatments, hence, this design is a combination of a nested and crossed design. As each health professional treats multiple patients, the data have a nested structure. This nested structure has thus far been ignored in the design of such trials, which may result in an underestimate of the required sample size. In the design stage, the sample sizes should be determined such that a desired power is achieved for each of the three pairwise comparisons, while keeping costs or sample size at a minimum. Methods The statistical model that relates outcome to treatment condition and explicitly takes the nested data structure into account is presented. Mathematical expressions that relate sample size to power are derived for each of the three pairwise comparisons on the basis of this model. The cost-efficient design achieves sufficient power for each pairwise comparison at lowest costs. Alternatively, one may minimize the total number of patients. The sample sizes are found numerically and an Internet application is available for this purpose. The design is also compared to a nested design in which each health professional delivers just one treatment. Results Mathematical expressions show that this design is more efficient than the nested design. For each pairwise comparison, power increases with the number of health professionals and the number of patients per health professional. The methodology of finding a cost-efficient design is illustrated using a trial that compares treatments for social phobia. The optimal sample sizes reflect the costs for training and supervising psychologists and psychiatrists, and the patient-level costs in the three treatment conditions. Conclusion This article provides the methodology for designing trials that compare three treatment conditions while taking the nesting of patients within health professionals into account. As such, it helps to avoid underpowered trials. To use the methodology, a priori estimates of the total outcome variances and intraclass correlation coefficients must be obtained from experts' opinions or findings in the literature.

Ignoring a Multilevel Structure in Mixture Item Response Models: Impact on Parameter Recovery and Model Selection

The current study investigated the consequences of ignoring a multilevel structure for a mixture item response model to show when a multilevel mixture item response model is needed. Study 1 focused on examining the consequence of ignoring dependency for within-level latent classes. Simulation conditions that may affect model selection and parameter recovery in the context of a multilevel data structure were manipulated: class-specific ICC, cluster size, and number of clusters. The accuracy of model selection (based on information criteria) and quality of parameter recovery were used to evaluate the impact of ignoring a multilevel structure. Simulation results indicated that, for the range of class-specific ICCs examined here (.1 to .3), mixture item response models which ignored a higher level nesting structure resulted in less accurate estimates and standard errors (SEs) of item discrimination parameters when the number of clusters was larger than 24 and the cluster size was larger than six. Class-varying ICCs can have compensatory effects on bias. Also, the results suggested that a mixture item response model which ignored multilevel structure was not selected over the multilevel mixture item response model based on Bayesian information criterion (BIC) if the number of clusters and cluster size was at least 50, respectively. In Study 2, the consequences of unnecessarily fitting a multilevel mixture item response model to single-level data were examined. Reassuringly, in the context of single-level data, a multilevel mixture item response model was not selected by BIC, and its use would not distort the within-level item parameter estimates or SEs when the cluster size was at least 20. Based on these findings, it is concluded that, for class-specific ICC conditions examined here, a multilevel mixture item response model is recommended over a single-level item response model for a clustered dataset having cluster size >20 and the number of clusters >50 .

Microaggressions and Daily Experience

Psychologists use the term microaggressions to describe subtle forms of bias and discrimination experienced by members of marginalized groups. Lilienfeld (2017, this issue) makes an important contribution to the literature by presenting a critical review of the meaning and measurement of microaggression experiences. In this commentary, we argue that advancing the construct of microaggressions rests on research approaches that move beyond static representations of individuals to dynamic frameworks that observe people's lives as they unfold day to day. We discuss the conceptual potential of microaggressions as a bridging concept across multiple levels of analysis. We conclude that the intensive study of individuals over time can contribute to theory evaluation and offer new insights into the nature of unfolding processes that are theorized to be central to the manifestation of microaggressions in everyday life.

Green Space and Depression during Pregnancy: Results from the Growing Up in New Zealand Study

Background: Antenatal depression is an important contributor to poor maternal health experienced by some women. This study aimed to determine whether exposure to green space during pregnancy is associated with less depression, and whether this association is moderated by relevant factors, such as age, education, self-identified ethnicity, physical activity, residential rurality, and socioeconomic status. Methods: Health data were sourced from the cohort study “Growing Up in New Zealand” comprised of 6772 participants. Green space was estimated based on the proportion of green space within the Census Area Unit. Adjusted logistic mixed effect models were used to investigate the association between green space and antenatal depression after controlling for confounding variables. Results: Maternal exposure to green space were not associated with lower odds of antenatal depression. Indications of effect modifications due to relevant factors were not observed. Conclusions: This study did not determine an association between access to green space (measured based on the distance to the nearest green space) and antenatal depression. Therefore, a link between green space and antenatal depression was not established. For that reason, ensuring residential areas contain adequate green space may or may not be helpful in preventing antenatal depression and adverse health outcomes associated with this depression. More studies focusing on pregnant women in a range of social contexts, and considering both exposure and access to green space, are warranted to determine the relationships between green space and antenatal depression.

Propensity score weighting for a continuous exposure with multilevel data

Propensity score methods (e.g., matching, weighting, subclassification) provide a statistical approach for balancing dissimilar exposure groups on baseline covariates. These methods were developed in the context of data with no hierarchical structure or clustering. Yet in many applications the data have a clustered structure that is of substantive importance, such as when individuals are nested within healthcare providers or within schools. Recent work has extended propensity score methods to a multilevel setting, primarily focusing on binary exposures. In this paper, we focus on propensity score weighting for a continuous, rather than binary, exposure in a multilevel setting. Using simulations, we compare several specifications of the propensity score: a random effects model, a fixed effects model, and a single-level model. Additionally, our simulations compare the performance of marginal versus cluster-mean stabilized propensity score weights. In our results, regression specifications that accounted for the multilevel structure reduced bias, particularly when cluster-level confounders were omitted. Furthermore, cluster mean weights outperformed marginal weights.

Modelling partially cross-classified multilevel data

This article proposes an approach to modelling partially cross-classified multilevel data where some of the level-1 observations are nested in one random factor and some are cross-classified by two random factors. Comparisons between a proposed approach to two other commonly used approaches which treat the partially cross-classified data as either fully nested or fully cross-classified are completed with a simulation study. Results show that the proposed approach demonstrates desirable performance in terms of parameter estimates and statistical inferences. Both the fully nested model and the fully cross-classified model suffer from biased estimates of some variance components and statistical inferences of some fixed effects. Results also indicate that the proposed model is robust against cluster size imbalance.