Deriving optimal data-analytic regimes from benchmarking studies

Estimating optimal decision trees for treatment assignment: The case of K > 2 treatment alternatives

For many problems in clinical practice, multiple treatment alternatives are available. Given data from a randomized controlled trial or an observational study, an important challenge is to estimate an optimal decision rule that specifies for each client the most effective treatment alternative, given his or her pattern of pretreatment characteristics. In the present paper we will look for such a rule within the insightful family of classification trees. Unfortunately, however, there is dearth of readily accessible software tools for optimal decision tree estimation in the case of more than two treatment alternatives. Moreover, this primary tree estimation problem is also cursed with two secondary problems: a structural missingness in typical studies on treatment evaluation (because every individual is assigned to a single treatment alternative only), and a major issue of replicability. In this paper we propose solutions for both the primary and the secondary problems at stake. We evaluate the proposed solution in a simulation study, and illustrate with an application on the search for an optimal tree-based treatment regime in a randomized controlled trial on K = 3 different types of aftercare for younger women with early-stage breast cancer. We conclude by arguing that the proposed solutions may have relevance for several other classification problems inside and outside the domain of optimal treatment assignment.

Psychopathological networks: Theory, methods and practice

In recent years, network approaches to psychopathology have sparked much debate and have had a significant impact on how mental disorders are perceived in the field of clinical psychology. However, there are many important challenges in moving from theory to empirical research and clinical practice and vice versa. Therefore, in this article, we bring together different points of view on psychological networks by methodologists and clinicians to give a critical overview on these challenges, and to present an agenda for addressing these challenges. In contrast to previous reviews, we especially focus on methodological issues related to temporal networks. This includes topics such as selecting and assessing the quality of the nodes in the network, distinguishing between- and within-person effects in networks, relating items that are measured at different time scales, and dealing with changes in network structures. These issues are not only important for researchers using network models on empirical data, but also for clinicians, who are increasingly likely to encounter (person-specific) networks in the consulting room.

Accounting for auto-dependency in binary dyadic time series data: A comparison of model- and permutation-based approaches for testing pairwise associations

Many theories have been put forward on how people become synchronized or co-regulate each other in daily interactions. These theories are often tested by observing a dyad and coding the presence of multiple target behaviours in small time intervals. The sequencing and co-occurrence of the partners' behaviours across time are then quantified by means of association measures (e.g., kappa coefficient, Jaccard similarity index, proportion of agreement). We demonstrate that the association values obtained are not easy to interpret, because they depend on the marginal frequencies and the amount of auto-dependency in the data. Moreover, often no inferential framework is available to test the significance of the association. Even if a significance test exists (e.g., kappa coefficient) auto-dependencies are not taken into account, which, as we will show, can seriously inflate the Type I error rate. We compare the effectiveness of a model- and a permutation-based framework for significance testing. Results of two simulation studies show that within both frameworks test variants exist that successfully account for auto-dependency, as the Type I error rate is under control, while power is good.

Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology

OBJECTIVE

One of the promises of the experience sampling methodology (ESM) is that a statistical analysis of an individual's emotions, cognitions and behaviors in everyday-life could be used to identify relevant treatment targets. A requisite for clinical implementation is that outcomes of such person-specific time-series analyses are not wholly contingent on the researcher performing them.

METHODS

To evaluate this, we crowdsourced the analysis of one individual patient's ESM data to 12 prominent research teams, asking them what symptom(s) they would advise the treating clinician to target in subsequent treatment.

RESULTS

Variation was evident at different stages of the analysis, from preprocessing steps (e.g., variable selection, clustering, handling of missing data) to the type of statistics and rationale for selecting targets. Most teams did include a type of vector autoregressive model, examining relations between symptoms over time. Although most teams were confident their selected targets would provide useful information to the clinician, not one recommendation was similar: both the number (0-16) and nature of selected targets varied widely.

CONCLUSION

This study makes transparent that the selection of treatment targets based on personalized models using ESM data is currently highly conditional on subjective analytical choices and highlights key conceptual and methodological issues that need to be addressed in moving towards clinical implementation.

Retrieving relevant factors with exploratory SEM and principal-covariate regression: A comparison

Behavioral researchers often linearly regress a criterion on multiple predictors, aiming to gain insight into the relations between the criterion and predictors. Obtaining this insight from the ordinary least squares (OLS) regression solution may be troublesome, because OLS regression weights show only the effect of a predictor on top of the effects of other predictors. Moreover, when the number of predictors grows larger, it becomes likely that the predictors will be highly collinear, which makes the regression weights' estimates unstable (i.e., the "bouncing beta" problem). Among other procedures, dimension-reduction-based methods have been proposed for dealing with these problems. These methods yield insight into the data by reducing the predictors to a smaller number of summarizing variables and regressing the criterion on these summarizing variables. Two promising methods are principal-covariate regression (PCovR) and exploratory structural equation modeling (ESEM). Both simultaneously optimize reduction and prediction, but they are based on different frameworks. The resulting solutions have not yet been compared; it is thus unclear what the strengths and weaknesses are of both methods. In this article, we focus on the extents to which PCovR and ESEM are able to extract the factors that truly underlie the predictor scores and can predict a single criterion. The results of two simulation studies showed that for a typical behavioral dataset, ESEM (using the BIC for model selection) in this regard is successful more often than PCovR. Yet, in 93% of the datasets PCovR performed equally well, and in the case of 48 predictors, 100 observations, and large differences in the strengths of the factors, PCovR even outperformed ESEM.