Measurement in Intensive Longitudinal Data

Assessing and accounting for measurement in intensive longitudinal studies: current practices, considerations, and avenues for improvement

PURPOSE

Intensive longitudinal studies, in which participants complete questionnaires multiple times a day over an extended period, are increasingly popular in the social sciences in general and quality-of-life research in particular. The intensive longitudinal methods allow for studying the dynamics of constructs (e.g., how much patient-reported outcomes vary across time). These methods promise higher ecological validity and lower recall bias than traditional methods that question participants only once, since the high frequency means that participants complete questionnaires in their everyday lives and do not have to retrospectively report about a large time interval. However, to ensure the validity of the results obtained from analyzing the intensive longitudinal data (ILD), greater awareness and understanding of appropriate measurement practices are needed.

METHOD

We surveyed 42 researchers experienced with ILD regarding their measurement practices and reasons for suboptimal practices.

RESULTS

Results showed that researchers typically do not use measures validated specifically for ILD. Participants assessing the psychometric properties and invariance of measures in their current studies was even less common, as was accounting for these properties when analyzing dynamics. This was mainly because participants did not have the necessary knowledge to conduct these assessments or were unaware of their importance for drawing valid inferences. Open science practices, in contrast, appear reasonably well ingrained in ILD studies.

CONCLUSION

Measurement practices in ILD still need improvement in some key areas; we provide recommendations in order to create a solid foundation for measuring and analyzing psychological constructs.

Assessing Between- and Within-Person Reliabilities of Items and Scale for Daily Procrastination: A Multilevel and Dynamic Approach

Intensive longitudinal data (ILD) has been collected to capture the dynamic fluctuations of procrastination; however, researchers have typically measured daily procrastination by modifying trait measures (e.g., adding a time reference "today") without adequately testing their reliabilities. The main purpose of this study was to use an advanced approach, dynamic structural equation modeling, to assess the between- and within-person reliabilities of a widely used six-item measure of daily procrastination. A total of 252 participants completed retrospective measures of various types of trait procrastination and daily measures of procrastination over 34 consecutive days. The results showed that the entire scale for daily procrastination and five of its six items had high between- and within-person reliabilities, but one item had much lower reliabilities, suggesting that this item may be inappropriate in everyday contexts. Furthermore, we found moderate to strong associations between the latent trait factor of procrastination and trait measures of procrastination. In addition, we identified substantial between-person variation in person-specific reliabilities and explored its relevant factors. Overall, this study assessed the reliabilities of a daily measure of procrastination, which facilitated future studies to obtain more reliable and consistent results and to better estimate the reliability of ILD.

Dynamic structural equation models with binary and ordinal outcomes in Mplus

Intensive longitudinal designs are increasingly popular, as are dynamic structural equation models (DSEM) to accommodate unique features of these designs. Many helpful resources on DSEM exist, though they focus on continuous outcomes while categorical outcomes are omitted, briefly mentioned, or considered as a straightforward extension. This viewpoint regarding categorical outcomes is not unwarranted for technical audiences, but there are non-trivial nuances in model building and interpretation with categorical outcomes that are not necessarily straightforward for empirical researchers. Furthermore, categorical outcomes are common given that binary behavioral indicators or Likert responses are frequently solicited as low-burden variables to discourage participant non-response. This tutorial paper is therefore dedicated to providing an accessible treatment of DSEM in Mplus exclusively for categorical outcomes. We cover the general probit model whereby the raw categorical responses are assumed to come from an underlying normal process. We cover probit DSEM and expound why existing treatments have considered categorical outcomes as a straightforward extension of the continuous case. Data from a motivating ecological momentary assessment study with a binary outcome are used to demonstrate an unconditional model, a model with disaggregated covariates, and a model for data with a time trend. We provide annotated Mplus code for these models and discuss interpretation of the results. We then discuss model specification and interpretation in the case of an ordinal outcome and provide an example to highlight differences between ordinal and binary outcomes. We conclude with a discussion of caveats and extensions.

Dynamic Structural Equation Models with Missing Data: Data Requirements on N and T

Dynamic structural equation modeling (DSEM) is a useful technique for analyzing intensive longitudinal data. A challenge of applying DSEM is the missing data problem. The impact of missing data on DSEM, especially on widely applied DSEM such as the two-level vector autoregressive (VAR) cross-lagged models, however, is understudied. To fill the research gap, we evaluated how well the fixed effects and variance parameters in two-level bivariate VAR models are recovered under different missingness percentages, sample sizes, the number of time points, and heterogeneity in missingness distributions through two simulation studies. To facilitate the use of DSEM under customized data and model scenarios (different from those in our simulations), we provided illustrative examples of how to conduct Monte Carlo simulations in Mplus to determine whether a data configuration is sufficient to obtain accurate and precise results from a specific DSEM.

The impact of different data handling strategies in exploratory and confirmatory factor analysis of diary measures: an evaluation using simulated and real-world asthma nighttime symptoms diary data

BACKGROUND

Daily diaries are an important modality for patient-reported outcome assessment. They typically comprise multiple questions, so understanding their underlying structure is key to appropriate analysis and interpretation. Structural evaluation of such measures poses challenges due to the high volume of repeated measurements. Potential strategies include selecting a single day, averaging item-level observations over time, or using all data while accounting for its multilevel structure.

METHOD

The above strategies were evaluated in a simulated dataset via exploratory and confirmatory factor modelling by comparing their impact on various estimates (i.e., inter-item correlations, factor loadings, model fit). Each strategy was additionally explored using real-world data from an observational study (the Asthma Nighttime Symptoms Diary).

RESULTS

Both single day and item average strategies resulted in biased factor loadings. The former displayed lower overall bias (single day: 0.064; item average: 0.121) and mean square error (single day: 0.007; item average: 0.016) but greater frequency of incorrect factor number identification compared with the latter (single day: 46.4%; item average: 0%). Increased estimated inter-item correlations were apparent in the item-average method. Non-trivial between- and within-person variance highlighted the utility of a multilevel approach. However, convergence issues and Heywood cases were more common under the multilevel approach (90.2% and 100.0%, respectively).

CONCLUSIONS

Our findings suggest that a multilevel approach can enhance our insight when evaluating the structural properties of daily diary data; however, implementation challenges still remain. Our work offers guidance on the impact of data handling decisions in diary assessment.

On some challenges of psychological research in late adulthood and aging

The World population is aging and, consequently, understanding late adulthood and aging processes is a major scientific priority. Research in this segment of the life is particularly challenging: the lifespan approach, necessary for thorough aging investigations, implies theoretical multidisciplinary and methodological multivariate perspectives; variability during aging is particularly high compared to previous life phases; study designs must account for specific predicaments (such as confound between age, cohort, and time of measurement; selective study participation and attrition; assumed age-invariance of measurements; retest effects). Furthermore, promising methods for data analysis develop on a daily basis, requiring continuous technical training. I will discuss some of these challenges, summarize extant potential solutions, and touch on some ethical and broader social responsibility issues.

Avoidance and fear day by day in social anxiety disorder

OBJECTIVE

Theories assert that avoidance maintains maladaptive anxiety over time, yet a clear prospective test of this effect in the day-by-day lives of people with social anxiety disorder (SAD) is lacking.

METHOD

We used intensive longitudinal data to test prospective relationships between social fear and social avoidance in 32 participants with SAD who reported on a total of 4256 time points.

RESULTS

Results suggested that avoidance strongly predicted future anxiety, but only in a minority of people with SAD. Relationships between anxiety and avoidance varied considerably across individuals. Pre-registered tests found that the strength of autocorrelation for social fear is a good target for future testing of prediction of exposure response. Participants with lower autocorrelations were less likely to show between-session habituation.

CONCLUSIONS

Overall, results suggest avoidance maintains fear in SAD for at least some individuals, but also indicates considerable variability. Further intensive longitudinal data is needed to examine individuals with SAD across varying time courses.

100 days of Adolescence: Elucidating Externalizing Behaviors Through the Daily Assessment of Inhibitory Control

Inhibitory control is a transdiagnostic risk factor for externalizing behaviors, particularly during adolescence. Despite advances in understanding links between inhibitory control and externalizing behaviors across youth on average, significant questions remain about how these links play out in the day-to-day lives of individual adolescents. The goals of the current study were to: (1) validate a novel 100-occasion measure of inhibitory control; (2) assess links between day-to-day fluctuations in inhibitory control and individual differences in externalizing behaviors; and (3) illustrate the potential of intensive longitudinal studies for person-specific analyses of adolescent externalizing behaviors. Participants were 106 youth (57.5% female, Mage = 13.34 years; SDage = 1.92) who completed a virtual baseline session followed by 100 daily surveys, including an adapted Stroop Color Word task designed to assess inhibitory control. Results suggested that the novel task was generally reliable and valid, and that inhibitory control fluctuated across days in ways that were meaningfully associated with individual differences in baseline impulsive behaviors. Results of illustrative personalized analyses suggested that inhibitory control had more influence in the daily networks of adolescents who used substances during the 100 days than in a matched set of adolescents who did not. This work marks a path forward in intensive longitudinal research by validating a novel inhibitory control measure, revealing that daily fluctuations in inhibitory control may be a unique construct broadly relevant to adolescent externalizing problems, and at the same time, highlighting that links between daily inhibitory control and impulsive behaviors are adolescent-specific.

Workers' emotional exhaustion and mental well-being over the COVID-19 pandemic: a Dynamic Structural Equation Modeling (DSEM) approach

The COVID-19 pandemic has presented significant challenges to the workforce, particularly concerning emotional and mental well-being. Given the prolonged periods of work-related stress, unexpected organizational changes, and uncertainties about work faced during the pandemic, it becomes imperative to study occupational health constructs under a dynamic methodological perspective, to understand their stable and unstable characteristics better. In this study, drawing on the Dynamic Structural Equation Modeling (DSEM) framework, we used a combination of multilevel AR(1) models, Residual-DSEM (RDSEM), multilevel bivariate VAR(1) models, and multilevel location-scale models to investigate the autoregression, trend, and (residual) cross-lagged relationships between emotional exhaustion (EmEx) and mental well-being (MWB) over the COVID-19 pandemic. Data were collected weekly on 533 workers from Germany (91.18%) and Italy (8.82%) who completed a self-reported battery (total number of observations = 3,946). Consistent with our hypotheses, results were as follows: (a) regarding autoregression, the autoregressive component for both EmEx and MWB was positive and significant, as well as it was their associated between-level variability; (b) regarding trend, over time EmEx significantly increased, while MWB significantly declined, furthermore both changes had a significant between-level variability; (c) regarding the longitudinal bivariate (cross-lagged) relationships, EmEx and MWB negatively and significantly affected each other from week to week, furthermore both cross-lagged relationships showed to have significant between-level variance. Overall, our study pointed attention to the vicious cycle between EmEx and MWB, even after controlling for their autoregressive component and trend, and supported the utility of DSEM in occupational health psychology studies.

Using Dynamic Structural Equation Modeling to Examine Between- and Within-Persons Factor Structure of the DASS-21

The recent integration of traditional time series analysis and confirmatory factor analysis techniques allows researchers to evaluate the psychometric properties of measurement instruments at between- and within-persons levels while accounting for autoregressive dependencies. The current study applies a dynamic structural equation modeling (SEM) latent factor analysis (i.e., DSEM-CFA) to a sample of 333 individuals who completed the DASS-21 at their regular therapy sessions. The results of the DSEM-CFA illuminate the reliability, invariance, and structural features of each DASS-21 subscale both between and within persons. The results suggest that the DASS-21 reliably measures depression, anxiety, and stress symptoms when evaluating differences between persons, but does not reliably assess within-persons fluctuations in symptoms over time. The results also suggest that currently accepted methods of modeling sensitivity to change within an instrument are likely lacking and the DSEM-CFA provides insight into reliability within and between persons, which is extremely important for instruments used across time.

How to explore within-person and between-person measurement model differences in intensive longitudinal data with the R package lmfa

Intensive longitudinal data (ILD) have become popular for studying within-person dynamics in psychological constructs (or between-person differences therein). Before investigating the dynamics, it is crucial to examine whether the measurement model (MM) is the same across subjects and time and, thus, whether the measured constructs have the same meaning. If the MM differs (e.g., because of changes in item interpretation or response styles), observations cannot be validly compared. Exploring differences in the MM for ILD can be done with latent Markov factor analysis (LMFA), which classifies observations based on the underlying MM (for many subjects and time points simultaneously) and thus shows which observations are comparable. However, the complexity of the method or the fact that no open-source software for LMFA existed until now may have hindered researchers from applying the method in practice. In this article, we provide a step-by-step tutorial for the new user-friendly software package lmfa, which allows researchers to easily perform the analysis LMFA in the freely available software R to investigate MM differences in their own ILD.

Challenges and Solutions to the Measurement of Neurocognitive Mechanisms in Developmental Settings

Identifying early neurocognitive mechanisms that confer risk for mental health problems is one important avenue as we seek to develop successful early interventions. Currently, however, we have limited understanding of the neurocognitive mechanisms involved in shaping mental health trajectories from childhood through young adulthood, and this constrains our ability to develop effective clinical interventions. In particular, there is an urgent need to develop more sensitive, reliable, and scalable measures of individual differences for use in developmental settings. In this review, we outline methodological shortcomings that explain why widely used task-based measures of neurocognition currently tell us little about mental health risk. We discuss specific challenges that arise when studying neurocognitive mechanisms in developmental settings, and we share suggestions for overcoming them. We also propose a novel experimental approach-which we refer to as "cognitive microscopy"-that involves adaptive design optimization, temporally sensitive task administration, and multilevel modeling. This approach addresses some of the methodological shortcomings outlined above and provides measures of stability, variability, and developmental change in neurocognitive mechanisms within a multivariate framework.

Establishing the Reliability, Validity, and Prognostic Utility of the Momentary Pain Catastrophizing Scale for Use in Ecological Momentary Assessment Research

Despite the marked increase in ecological momentary assessment research, few reliable and valid measures of momentary experiences have been established. The goal of this preregistered study was to establish the reliability, validity, and prognostic utility of the momentary Pain Catastrophizing Scale (mPCS), a 3-item measure developed to assess situational pain catastrophizing. Participants in 2 studies of postsurgical pain outcomes completed the mPCS 3 to 5 times per day prior to surgery (N = 494, T = 20,271 total assessments). The mPCS showed good psychometric properties, including multilevel reliability and factor invariance across time. Participant-level average mPCS was strongly positively correlated with dispositional pain catastrophizing as assessed by the Pain Catastrophizing Scale (r = .55 and .69 in study 1 and study 2, respectively). To establish prognostic utility, we then examined whether the mPCS improved prediction of postsurgical pain outcomes above and beyond one-time assessment of dispositional pain catastrophizing. Indeed, greater variability in momentary pain catastrophizing prior to surgery was uniquely associated with increased pain immediately after surgery (b = .58, P = .005), after controlling for preoperative pain levels and dispositional pain catastrophizing. Greater average mPCS score prior to surgery was also uniquely associated with lesser day-to-day improvement in postsurgical pain (b = .01, P = .003), whereas dispositional pain catastrophizing was not (b = -.007, P = .099). These results show that the mPCS is a reliable and valid tool for ecological momentary assessment research and highlight its potential utility over and above retrospective measures of pain catastrophizing. PERSPECTIVE: This article presents the psychometric properties and prognostic utility of a new measure to assess momentary pain catastrophizing. This brief, 3-item measure will allow researchers and clinicians to assess fluctuations in pain catastrophizing during individuals' daily lives, as well as dynamic relationships between catastrophizing, pain, and related factors.

Investigating Individual Variation Using Dynamic Structural Equation Modeling: A Tutorial with Tinnitus

A growing body of research suggests that standard group-based models might provide little insight regarding individuals. In the current study, we sought to compare group-based and individual predictors of bothersome tinnitus, illustrating how researchers can use dynamic structural equation modeling (DSEM) for intensive longitudinal data to examine whether findings from analyses of the group apply to individuals. A total of 43 subjects with bothersome tinnitus responded to up to 200 surveys each. In multi-level DSEM models, survey items loaded on three factors (tinnitus bother, cognitive symptoms, and anxiety) and results indicated a reciprocal relationship between tinnitus bother and anxiety. In fully idiographic models, the three-factor model fit poorly for two individuals, and the multilevel model did not generalize to most individuals, possibly due to limited power. Research examining heterogeneous conditions such as tinnitus bother may benefit from methods such as DSEM that allow researchers to model dynamic relationships.

Analysis of dynamic psychological processes to understand and promote physical activity behaviour using intensive longitudinal methods: a primer

Physical activity behaviour displays temporal variability, and is influenced by a range of dynamic psychological processes (e.g., affect) and shaped by various co-occurring events (e.g., social/environmental factors, interpersonal dynamics). Yet, most physical activity research tends not to examine the dynamic psychological processes implicated in adopting and maintaining physical activity. Intensive longitudinal methods (ILM) represent one particularly salient means of studying the complex psychological dynamics that underlie and result from physical activity behaviour. With the increased recent interest in using intensive longitudinal data to understand specific dynamic psychological processes, the field of exercise and health psychology is well-positioned to draw from state-of-the-art measurement and statistical approaches that have been developed and operationalised in other fields of enquiry. The purpose of this review is to provide an overview of some of the fundamental dynamic measurement and modelling approaches applicable to the study of physical activity behaviour change, as well as the dynamic psychological processes that contribute to such change.

Variability in intensively assessed mood: Systematic sources and factor structure in outpatients with opioid use disorder

In intensive longitudinal studies using ecological momentary assessment, mood is typically assessed by repeatedly obtaining ratings for a large set of adjectives. Summarizing and analyzing these mood data can be problematic because the reliability and factor structure of such measures have rarely been evaluated in this context, which-unlike cross-sectional studies-captures between- and within-person processes. Our study examined how mood ratings (obtained thrice daily for 8 weeks; n = 306, person moments = 39,321) systematically vary and covary in outpatients receiving medication for opioid use disorder (MOUD). We used generalizability theory to quantify several aspects of reliability, and multilevel confirmatory factor analysis (MCFA) to detect factor structures within and across people. Generalizability analyses showed that the largest proportion of systematic variance across mood items was at the person level, followed by the person-by-day interaction and the (comparatively small) person-by-moment interaction for items reflecting low arousal. The best-fitting MCFA model had a three-factor structure both at the between- and within-person levels: positive mood, negative mood, and low-arousal states (with low arousal considered as either a separate factor or a subfactor of negative mood). We conclude that (a) mood varied more between days than between moments and (b) low arousal may be worth scoring and reporting separately from positive and negative mood states, at least in a MOUD population. Our three-factor structure differs from prior analyses of mood; more work is needed to understand the extent to which it generalizes to other populations. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Estimating and Testing Causal Mediation Effects in Single-Case Experimental Designs Using State-Space Modeling

In this article, we present single-case causal mediation analysis as the application of causal mediation analysis to data collected within a single-case experiment. This method combines the focus on the individual with the focus on mechanisms of change, rendering it a promising approach for both mediation and single-case researchers. For this purpose, we propose a new method based on time-discrete state-space modeling to estimate the direct and indirect treatment effects. We demonstrate how to estimate the model for a single-case experiment on stress and craving in a routine alcohol consumer before and after an imposed period of abstinence. Furthermore, we present a simulation study that examines the estimation and testing of the standardized indirect effect. All parameters used to generate the data were recovered with acceptable precision. We use maximum likelihood and permutation procedures to calculate p-values and standard errors of the parameters estimates. The new method is promising for testing mediated effects in single-case experimental designs. We further discuss limitations of the new method with respect to causal inference, as well as more technical concerns, such as the choice of the time lags between the measurements.

Just-in-Time Adaptive Interventions for Suicide Prevention: Promise, Challenges, and Future Directions

The suicide rate (currently 14 per 100,000) has barely changed in the United States over the past 100 years. There is a need for new ways of preventing suicide. Further, research has revealed that suicidal thoughts and behaviors and the factors that drive them are dynamic, heterogeneous, and interactive. Most existing interventions for suicidal thoughts and behaviors are infrequent, not accessible when most needed, and not systematically tailored to the person using their own data (e.g., from their own smartphone). Advances in technology offer an opportunity to develop new interventions that may better match the dynamic, heterogeneous, and interactive nature of suicidal thoughts and behaviors. Just-In-Time Adaptive Interventions (JITAIs), which use smartphones and wearables, are designed to provide the right type of support at the right time by adapting to changes in internal states and external contexts, offering a promising pathway toward more effective suicide prevention. In this review, we highlight the potential of JITAIs for suicide prevention, challenges ahead (e.g., measurement, ethics), and possible solutions to these challenges.