Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines

Contextualising flexible nonlinear periodization as a person-adaptive behavioral model for exercise maintenance

There is a growing focus on developing person-adaptive strategies to support sustained exercise behaviour, necessitating conceptual models to guide future research and applications. This paper introduces Flexible nonlinear periodisation (FNLP) - a proposed, but underdeveloped person-adaptive model originating in sport-specific conditioning - that, pending empirical refinement and evaluation, may be applied in health promotion and disease prevention settings. To initiate such efforts, the procedures of FNLP (i.e., acutely and dynamically matching exercise demand to individual assessments of mental and physical readiness) are integrated with contemporary health behaviour evidence and theory to propose a modified FNLP model and to show hypothesised pathways by which FNLP may support exercise adherence (e.g., flexible goal setting, management of affective responses, and provision of autonomy/variety-support). Considerations for future research are also provided to guide iterative, evidence-based efforts for further development, acceptability, implementation, and evaluation.

Optimizing an adaptive digital oral health intervention for promoting oral self-care behaviors: Micro-randomized trial protocol

Dental disease continues to be one of the most prevalent chronic diseases in the United States. Although oral self-care behaviors (OSCB), involving systematic twice-a-day tooth brushing, can prevent dental disease, this basic behavior is not sufficiently practiced. Recent advances in digital technology offer tremendous potential for promoting OSCB by delivering Just-In-Time Adaptive Interventions (JITAIs)- interventions that leverage dynamic information about the person's state and context to effectively prompt them to engage in a desired behavior in real-time, real-world settings. However, limited research attention has been given to systematically investigating how to best prompt individuals to engage in OSCB in daily life, and under what conditions prompting would be most beneficial. This paper describes the protocol for a Micro-Randomized Trial (MRT) to inform the development of a JITAI for promoting ideal OSCB, namely, brushing twice daily, for two minutes each time, in all four dental quadrants (i.e., 2x2x4). Sensors within an electric toothbrush (eBrush) will be used to track OSCB and a matching mobile app (Oralytics) will deliver on-demand feedback and educational information. The MRT will micro-randomize participants twice daily (morning and evening) to either (a) a prompt (push notification) containing one of several theoretically grounded engagement strategies or (b) no prompt. The goal is to investigate whether, what type of, and under what conditions prompting increases engagement in ideal OSCB. The results will build the empirical foundation necessary to develop an optimized JITAI that will be evaluated relative to a suitable control in a future randomized controlled trial.

The impact of using reinforcement learning to personalize communication on medication adherence: findings from the REINFORCE trial

Text messaging can promote healthy behaviors, like adherence to medication, yet its effectiveness remains modest, in part because message content is rarely personalized. Reinforcement learning has been used in consumer technology to personalize content but with limited application in healthcare. We tested a reinforcement learning program that identifies individual responsiveness ("adherence") to text message content and personalizes messaging accordingly. We randomized 60 individuals with diabetes and glycated hemoglobin A1c [HbA1c] ≥ 7.5% to reinforcement learning intervention or control (no messages). Both arms received electronic pill bottles to measure adherence. The intervention improved absolute adjusted adherence by 13.6% (95%CI: 1.7%-27.1%) versus control and was more effective in patients with HbA1c 7.5- < 9.0% (36.6%, 95%CI: 25.1%-48.2%, interaction p < 0.001). We also explored whether individual patient characteristics were associated with differential response to tested behavioral factors and unique clusters of responsiveness. Reinforcement learning may be a promising approach to improve adherence and personalize communication at scale.

Automated Messaging Delivered Alongside Behavioral Treatment for Weight Loss: Qualitative Study

BACKGROUND

Mobile health interventions for weight loss frequently use automated messaging. However, this intervention modality appears to have limited weight loss efficacy. Furthermore, data on users' subjective experiences while receiving automated messaging-based interventions for weight loss are scarce, especially for more advanced messaging systems providing users with individually tailored, data-informed feedback.

OBJECTIVE

The purpose of this study was to characterize the experiences of individuals with overweight or obesity who received automated messages for 6-12 months as part of a behavioral weight loss trial.

METHODS

Participants (n=40) provided Likert-scale ratings of messaging acceptability and completed a structured qualitative interview (n=39) focused on their experiences with the messaging system and generating suggestions for improvement. Interview data were analyzed using thematic analysis.

RESULTS

Participants found the messages most useful for summarizing goal progress and least useful for suggesting new behavioral strategies. Overall message acceptability was moderate (2.67 out of 5). From the interviews, 2 meta-themes emerged. Participants indicated that although the messages provided useful reminders of intervention goals and skills, they did not adequately capture their lived experiences while losing weight.

CONCLUSIONS

Many participants found the automated messages insufficiently tailored to their personal weight loss experiences. Future studies should explore alternative methods for message tailoring (eg, allowing for a higher degree of participant input and interactivity) that may boost treatment engagement and efficacy.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05231824; https://clinicaltrials.gov/study/NCT05231824.

Online model selection by learning how compositional kernels evolve

Motivated by the need for efficient, personalized learning in mobile health, we investigate the problem of online compositional kernel selection for multi-task Gaussian Process regression. Existing composition selection methods do not satisfy our strict criteria in health; selection must occur quickly, and the selected kernels must maintain the appropriate level of complexity, sparsity, and stability as data arrives online. We introduce the Kernel Evolution Model (KEM), a generative process on how to evolve kernel compositions in a way that manages the bias-variance trade-off as we observe more data about a user. Using pilot data, we learn a set of kernel evolutions that can be used to quickly select kernels for new test users. KEM reliably selects high-performing kernels for a range of synthetic and real data sets, including two health data sets.

A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement

Virtual Mental Health Assistants (VMHAs) continuously evolve to support the overloaded global healthcare system, which receives approximately 60 million primary care visits and 6 million emergency room visits annually. These systems, developed by clinical psychologists, psychiatrists, and AI researchers, are designed to aid in Cognitive Behavioral Therapy (CBT). The main focus of VMHAs is to provide relevant information to mental health professionals (MHPs) and engage in meaningful conversations to support individuals with mental health conditions. However, certain gaps prevent VMHAs from fully delivering on their promise during active communications. One of the gaps is their inability to explain their decisions to patients and MHPs, making conversations less trustworthy. Additionally, VMHAs can be vulnerable in providing unsafe responses to patient queries, further undermining their reliability. In this review, we assess the current state of VMHAs on the grounds of user-level explainability and safety, a set of desired properties for the broader adoption of VMHAs. This includes the examination of ChatGPT, a conversation agent developed on AI-driven models: GPT3.5 and GPT-4, that has been proposed for use in providing mental health services. By harnessing the collaborative and impactful contributions of AI, natural language processing, and the mental health professionals (MHPs) community, the review identifies opportunities for technological progress in VMHAs to ensure their capabilities include explainable and safe behaviors. It also emphasizes the importance of measures to guarantee that these advancements align with the promise of fostering trustworthy conversations.

Effective Behavior Change Techniques in Digital Health Interventions for the Prevention or Management of Noncommunicable Diseases: An Umbrella Review

BACKGROUND

Despite an abundance of digital health interventions (DHIs) targeting the prevention and management of noncommunicable diseases (NCDs), it is unclear what specific components make a DHI effective.

PURPOSE

This narrative umbrella review aimed to identify the most effective behavior change techniques (BCTs) in DHIs that address the prevention or management of NCDs.

METHODS

Five electronic databases were searched for articles published in English between January 2007 and December 2022. Studies were included if they were systematic reviews or meta-analyses of DHIs targeting the modification of one or more NCD-related risk factors in adults. BCTs were coded using the Behavior Change Technique Taxonomy v1. Study quality was assessed using AMSTAR 2.

RESULTS

Eighty-five articles, spanning 12 health domains and comprising over 865,000 individual participants, were included in the review. We found evidence that DHIs are effective in improving health outcomes for patients with cardiovascular disease, cancer, type 2 diabetes, and asthma, and health-related behaviors including physical activity, sedentary behavior, diet, weight management, medication adherence, and abstinence from substance use. There was strong evidence to suggest that credible source, social support, prompts and cues, graded tasks, goals and planning, feedback and monitoring, human coaching and personalization components increase the effectiveness of DHIs targeting the prevention and management of NCDs.

CONCLUSIONS

This review identifies the most common and effective BCTs used in DHIs, which warrant prioritization for integration into future interventions. These findings are critical for the future development and upscaling of DHIs and should inform best practice guidelines.

The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning

Discount regularization, using a shorter planning horizon when calculating the optimal policy, is a popular choice to restrict planning to a less complex set of policies when estimating an MDP from sparse or noisy data (Jiang et al., 2015). It is commonly understood that discount regularization functions by de-emphasizing or ignoring delayed effects. In this paper, we reveal an alternate view of discount regularization that exposes unintended consequences. We demonstrate that planning under a lower discount factor produces an identical optimal policy to planning using any prior on the transition matrix that has the same distribution for all states and actions. In fact, it functions like a prior with stronger regularization on state-action pairs with more transition data. This leads to poor performance when the transition matrix is estimated from data sets with uneven amounts of data across state-action pairs. Our equivalence theorem leads to an explicit formula to set regularization parameters locally for individual state-action pairs rather than globally. We demonstrate the failures of discount regularization and how we remedy them using our state-action-specific method across simple empirical examples as well as a medical cancer simulator.

Reward Design For An Online Reinforcement Learning Algorithm Supporting Oral Self-Care

While dental disease is largely preventable, professional advice on optimal oral hygiene practices is often forgotten or abandoned by patients. Therefore patients may benefit from timely and personalized encouragement to engage in oral self-care behaviors. In this paper, we develop an online reinforcement learning (RL) algorithm for use in optimizing the delivery of mobile-based prompts to encourage oral hygiene behaviors. One of the main challenges in developing such an algorithm is ensuring that the algorithm considers the impact of current actions on the effectiveness of future actions (i.e., delayed effects), especially when the algorithm has been designed to run stably and autonomously in a constrained, real-world setting characterized by highly noisy, sparse data. We address this challenge by designing a quality reward that maximizes the desired health outcome (i.e., high-quality brushing) while minimizing user burden. We also highlight a procedure for optimizing the hyperparameters of the reward by building a simulation environment test bed and evaluating candidates using the test bed. The RL algorithm discussed in this paper will be deployed in Oralytics. To the best of our knowledge, Oralytics is the first mobile health study utilizing an RL algorithm designed to prevent dental disease by optimizing the delivery of motivational messages supporting oral self-care behaviors.

Microrandomized Trials: Developing Just-in-Time Adaptive Interventions for Better Public Health

Just-in-time adaptive interventions (JITAIs) represent an intervention design that adapts the provision and type of support over time to an individual's changing status and contexts, intending to deliver the right support on the right occasion. As a novel strategy for delivering mobile health interventions, JITAIs have the potential to improve access to quality care in underserved communities and, thus, alleviate health disparities, a significant public health concern. Valid experimental designs and analysis methods are required to inform the development of JITAIs. Here, we briefly review the cutting-edge design of microrandomized trials (MRTs), covering both the classical MRT design and its outcome-adaptive counterpart. Associated statistical challenges related to the design and analysis of MRTs are also discussed. Two case studies are provided to illustrate the aforementioned concepts and designs throughout the article. We hope our work leads to better design and application of JITAIs, advancing public health research and practice. (Am J Public Health. 2023;113(1):60-69. https://doi.org/10.2105/AJPH.2022.307150).