Control Engineering Methods for the Design of Robust Behavioral Treatments

Adaptive treatment and robust control

A control theory perspective on determination of optimal dynamic treatment regimes is considered. The aim is to adapt statistical methodology that has been developed for medical or other biostatistical applications to incorporate powerful control techniques that have been designed for engineering or other technological problems. Data tend to be sparse and noisy in the biostatistical area and interest has tended to be in statistical inference for treatment effects. In engineering fields, experimental data can be more easily obtained and reproduced and interest is more often in performance and stability of proposed controllers rather than modeling and inference per se. We propose that modeling and estimation should be based on standard statistical techniques but subsequent treatment policy should be obtained from robust control. To bring focus, we concentrate on A-learning methodology as developed in the biostatistical literature and H ∞ -synthesis from control theory. Simulations and two applications demonstrate robustness of the H ∞ strategy compared to standard A-learning in the presence of model misspecification or measurement error.

Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity

With the recent proliferation of mobile health technologies, health scientists are increasingly interested in developing just-in-time adaptive interventions (JITAIs), typically delivered via notifications on mobile devices and designed to help users prevent negative health outcomes and to promote the adoption and maintenance of healthy behaviors. A JITAI involves a sequence of decision rules (i.e., treatment policies) that take the user's current context as input and specify whether and what type of intervention should be provided at the moment. In this work, we describe a reinforcement learning (RL) algorithm that continuously learns and improves the treatment policy embedded in the JITAI as data is being collected from the user. This work is motivated by our collaboration on designing an RL algorithm for HeartSteps V2 based on data collected HeartSteps V1. HeartSteps is a physical activity mobile health application. The RL algorithm developed in this work is being used in HeartSteps V2 to decide, five times per day, whether to deliver a context-tailored activity suggestion.

Development of a Control-Oriented Model of Social Cognitive Theory for Optimized mHealth Behavioral Interventions

Mobile health (mHealth) technologies are contributing to the increasing relevance of control engineering principles in understanding and improving health behaviors, such as physical activity. Social Cognitive Theory (SCT), one of the most influential theories of health behavior, has been used as the conceptual basis for behavioral interventions for smoking cessation, weight management, and other health-related outcomes. This paper presents a control-oriented dynamical systems model of SCT based on fluid analogies that can be used in system identification and control design problems relevant to the design and analysis of intensively adaptive interventions. Following model development, a series of simulation scenarios illustrating the basic workings of the model are presented. The model's usefulness is demonstrated in the solution of two important practical problems: 1) semiphysical model estimation from data gathered in a physical activity intervention (the MILES study) and 2) as a means for discerning the range of "ambitious but doable" daily step goals in a closed-loop behavioral intervention aimed at sedentary adults. The model is the basis for ongoing experimental validation efforts, and should encourage additional research in applying control engineering technologies to the social and behavioral sciences.

Precision Medicine

Precision medicine seeks to maximize the quality of healthcare by individualizing the healthcare process to the uniquely evolving health status of each patient. This endeavor spans a broad range of scientific areas including drug discovery, genetics/genomics, health communication, and causal inference all in support of evidence-based, i.e., data-driven, decision making. Precision medicine is formalized as a treatment regime which comprises a sequence of decision rules, one per decision point, which map up-to-date patient information to a recommended action. The potential actions could be the selection of which drug to use, the selection of dose, timing of administration, specific diet or exercise recommendation, or other aspects of treatment or care. Statistics research in precision medicine is broadly focused on methodological development for estimation of and inference for treatment regimes which maximize some cumulative clinical outcome. In this review, we provide an overview of this vibrant area of research and present important and emerging challenges.

Personalized models of physical activity responses to text message micro-interventions: A proof-of-concept application of control systems engineering methods

OBJECTIVES

The conceptual models underlying physical activity interventions have been based largely on differences between more and less active people. Yet physical activity is a dynamic behavior, and such models are not sensitive to factors that regulate behavior at a momentary level or how people respond to individual attempts at intervening. We demonstrate how a control systems engineering approach can be applied to develop personalized models of behavioral responses to an intensive text message-based intervention.

DESIGN & METHOD

To establish proof-of-concept for this approach, 10 adults wore activity monitors for 16 weeks and received five text messages daily at random times. Message content was randomly selected from three types of messages designed to target (1) social-cognitive processes associated with increasing physical activity, (2) social-cognitive processes associated with reducing sedentary behavior, or (3) general facts unrelated to either physical activity or sedentary behavior. A dynamical systems model was estimated for each participant to examine the magnitude and timing of responses to each type of text message.

RESULTS

Models revealed heterogeneous responses to different message types that varied between people and between weekdays and weekends.

CONCLUSIONS

This proof-of-concept demonstration suggests that parameters from this model can be used to develop personalized algorithms for intervention delivery. More generally, these results demonstrate the potential utility of control systems engineering models for optimizing physical activity interventions.

Evaluating the effect of smoking cessation treatment on a complex dynamical system

OBJECTIVE

To understand the dynamic relations among tobacco withdrawal symptoms to inform the development of effective smoking cessation treatments. Dynamical system models from control engineering are introduced and utilized to evaluate complex treatment effects. We demonstrate how dynamical models can be used to examine how distinct withdrawal-related processes are related over time and how treatment influences these relations.

METHOD

Intensive longitudinal data from a randomized placebo-controlled smoking cessation trial (N=1504) are used to estimate a dynamical model of withdrawal-related processes including momentary craving, negative affect, quitting self-efficacy, and cessation fatigue for each of six treatment conditions (nicotine patch, nicotine lozenge, bupropion, patch + lozenge, bupropion + lozenge, and placebo).

RESULTS

Estimation and simulation results show that (1) withdrawal measurements are interrelated over time, (2) nicotine patch + nicotine lozenge showed reduced cessation fatigue and enhanced self-efficacy in the long-term while bupropion + nicotine lozenge was more effective at reducing negative affect and craving, and (3) although nicotine patch + nicotine lozenge had a better initial effect on cessation fatigue and self-efficacy, nicotine lozenge had a stronger effect on negative affect and nicotine patch had a stronger impact on craving.

CONCLUSIONS

This approach can be used to provide new evidence illustrating (a) the total impact of treatment conditions (via steady state values) and (b) the total initial impact (via rate of initial change values) on smoking-related outcomes for separate treatment conditions, noting that the conditions that produce the largest change may be different than the conditions that produce the fastest change.

System Identification Algorithm for Non-Uniformly Sampled Data

Considerable effort has been devoted to the development of algorithms for identification of parsimonious discrete time models from noisy input/output data sets since this facilitates controller design. Several methods, such as nuclear norm minimization, have been used to provide approximate solutions to this non-convex problem. However, even though the field of continuous time system identification is now mature, results on parsimonious model identification of continuous time systems are still very limited. In this paper, an atomic norm minimization method is proposed for this purpose that can handle non-uniformly sampled data without preprocessing. The proposed approach provides an efficient way to use noisy, non-uniformly sampled data to determine a reliable, low-order continuous time model. Numerical performance is illustrated using academic examples and simulated behavioral data from a smoking cessation study.