Predicting Attrition in a Text-Based Nutrition Education Program: Survival Analysis of Text2BHealthy

Can a Single Variable Predict Early Dropout From Digital Health Interventions? Comparison of Predictive Models From Two Large Randomized Trials

BACKGROUND

A single generalizable metric that accurately predicts early dropout from digital health interventions has the potential to readily inform intervention targets and treatment augmentations that could boost retention and intervention outcomes. We recently identified a type of early dropout from digital health interventions for smoking cessation, specifically, users who logged in during the first week of the intervention and had little to no activity thereafter. These users also had a substantially lower smoking cessation rate with our iCanQuit smoking cessation app compared with users who used the app for longer periods.

OBJECTIVE

This study aimed to explore whether log-in count data, using standard statistical methods, can precisely predict whether an individual will become an iCanQuit early dropout while validating the approach using other statistical methods and randomized trial data from 3 other digital interventions for smoking cessation (combined randomized N=4529).

METHODS

Standard logistic regression models were used to predict early dropouts for individuals receiving the iCanQuit smoking cessation intervention app, the National Cancer Institute QuitGuide smoking cessation intervention app, the WebQuit.org smoking cessation intervention website, and the Smokefree.gov smoking cessation intervention website. The main predictors were the number of times a participant logged in per day during the first 7 days following randomization. The area under the curve (AUC) assessed the performance of the logistic regression models, which were compared with decision trees, support vector machine, and neural network models. We also examined whether 13 baseline variables that included a variety of demographics (eg, race and ethnicity, gender, and age) and smoking characteristics (eg, use of e-cigarettes and confidence in being smoke free) might improve this prediction.

RESULTS

The AUC for each logistic regression model using only the first 7 days of log-in count variables was 0.94 (95% CI 0.90-0.97) for iCanQuit, 0.88 (95% CI 0.83-0.93) for QuitGuide, 0.85 (95% CI 0.80-0.88) for WebQuit.org, and 0.60 (95% CI 0.54-0.66) for Smokefree.gov. Replacing logistic regression models with more complex decision trees, support vector machines, or neural network models did not significantly increase the AUC, nor did including additional baseline variables as predictors. The sensitivity and specificity were generally good, and they were excellent for iCanQuit (ie, 0.91 and 0.85, respectively, at the 0.5 classification threshold).

CONCLUSIONS

Logistic regression models using only the first 7 days of log-in count data were generally good at predicting early dropouts. These models performed well when using simple, automated, and readily available log-in count data, whereas including self-reported baseline variables did not improve the prediction. The results will inform the early identification of people at risk of early dropout from digital health interventions with the goal of intervening further by providing them with augmented treatments to increase their retention and, ultimately, their intervention outcomes.

Hospital homebound students and K-12 online schooling

The flexibility afforded by online education may provide opportunities for learners with disability who require absence from traditional learning environments. This study sought to describe how a subset of learners with disability, those with hospital-homebound designation, perform in K-12 online classes, particularly as compared to non-hospital homebound counterparts. A cross-sectional analysis was performed of all Florida Virtual School course enrollments from August 1, 2012 to July 31, 2018. Researchers analyzed 2,534 course enrollments associated with K-12 students who, at the time of their course enrollment, had hospital-homebound designation, and a comparison group of 5,470,591 enrollments from K-12 students without hospital-homebound status. Data analysis showed three important outcomes. First, hospital-homebound designated student academic performance was equivalent to their non-hospital homebound counterparts. Second, however, hospital-homebound course enrollments were 26% more likely to result in a withdrawal prior to grade generation. Third, these withdrawals were potentially mitigated when H/H designated students were enrolled in five or more classes or in classes with five or more students. The results of this study provided evidence that when they can remain enrolled, hospital-homebound learners experience equivalent academic outcomes in online learning environments. These findings suggest that healthcare professionals should be made aware of the potentially equivalent outcomes for their patients. Moreover, virtual schools should seek to identify and create supports for these students.

Evaluating students' engagement with an online learning environment during and after COVID-19 related school closures: A survival analysis approach

Background

Due to the COVID-19 pandemic schools all over the world were closed and thereby students had to be instructed from distance. Consequently, the use of online learning environments for online distance learning increased massively. However, the perseverance of using online learning environments during and after school closures remains to be investigated.

Method

We examined German students’ (n ≈ 300,000 students; ≈ 18 million computed problem sets) engagement in an online learning environment for mathematics by means of survival analysis.

Results

We observed that the total number of students who registered increased considerably during and after school closures compared to the previous three years. Importantly, however, the proportion of students engaged also decreased more rapidly over time.

Conclusion

The application of survival analysis provided valuable insights into students’ engagement in online learning - or conversely students’ increased dropout rates - over time. Its application to educational settings allows to address a broader range of questions on students’ engagement in online learning environments in the future.

Turn on, Tune in, and Drop out: Predictors of Attrition in a Prospective Observational Cohort Study on Psychedelic Use

Background

The resurgence of research and public interest in the positive psychological effects of psychedelics, together with advancements in digital data collection techniques, have brought forth a new type of research design, which involves prospectively gathering large-scale naturalistic data from psychedelic users; that is, before and after the use of a psychedelic compound. A methodological limitation of such studies is their high attrition rate, particularly owing to participants who stop responding after initial study enrollment. Importantly, study dropout can introduce systematic biases that may affect the interpretability of results.

Objective

Based on a previously collected sample (baseline n=654), here we investigated potential determinants of study attrition in web-based prospective studies on psychedelic use.

Methods

Logistic regression models were used to examine demographic, psychological trait and state, and psychedelic-specific predictors of dropout. Predictors were assessed 1 week before, 1 day after, and 2 weeks after psychedelic use, with attrition being defined as noncompletion of the key endpoint 4 weeks post experience.

Results

Predictors of attrition were found among demographic variables including age (β=0.024; P=.007) and educational levels, as well as personality traits, specifically conscientiousness (β=–0.079; P=.02) and extraversion (β=0.082; P=.01). Contrary to prior hypotheses, neither baseline attitudes toward psychedelics nor the intensity of acute challenging experiences were predictive of dropout.

Conclusions

The baseline predictors of attrition identified here are consistent with those reported in longitudinal studies in other scientific disciplines, suggesting their transdisciplinary relevance. Moreover, the lack of an association between attrition and psychedelic advocacy or negative drug experiences in our sample contextualizes concerns about problematic biases in these and related data.

Implementation: The Next Giant Hurdle to Clinical Transformation With Digital Health

Clinical implementation of digital health is a major hurdle to overcome in the coming years. Considering the role of the Journal of Medical Internet Research in the past 20 years and looking toward the journal’s future, this viewpoint acknowledges the vision of medicine and the role that digital health plays in that vision. It also highlights barriers to implementation of digital health as an obstacle to achieving that vision. In particular, this paper focuses on how digital health research must start looking toward implementation as an area of inquiry and the role that the Journal of Medical Internet Research and its' sister journals from JMIR Publications can play in this process.

Predicting Dropouts From an Electronic Health Platform for Lifestyle Interventions: Analysis of Methods and Predictors

BACKGROUND

The increasing prevalence and economic impact of chronic diseases challenge health care systems globally. Digital solutions can potentially improve efficiency and quality of care, but these initiatives struggle with nonusage attrition. Machine learning methods have been proven to predict dropouts in other settings but lack implementation in health care.

OBJECTIVE

This study aimed to gain insight into the causes of attrition for patients in an electronic health (eHealth) intervention for chronic lifestyle diseases and evaluate if attrition can be predicted and consequently prevented. We aimed to build predictive models that can identify patients in a digital lifestyle intervention at high risk of dropout by analyzing several predictor variables applied in different models and to further assess the possibilities and impact of implementing such models into an eHealth platform.

METHODS

Data from 2684 patients using an eHealth platform were iteratively analyzed using logistic regression, decision trees, and random forest models. The dataset was split into a 79.99% (2147/2684) training and cross-validation set and a 20.0% (537/2684) holdout test set. Trends in activity patterns were analyzed to assess engagement over time. Development and implementation were performed iteratively with health coaches.

RESULTS

Patients in the test dataset were classified as dropouts with an 89% precision using a random forest model and 11 predictor variables. The most significant predictors were the provider of the intervention, 2 weeks inactivity, and the number of advices received from the health coach. Engagement in the platform dropped significantly leading up to the time of dropout.

CONCLUSIONS

Dropouts from eHealth lifestyle interventions can be predicted using various data mining methods. This can support health coaches in preventing attrition by receiving proactive warnings. The best performing predictive model was found to be the random forest.

A randomized controlled trial for overweight and obesity in preschoolers: the More and Less Europe study - an intervention within the STOP project

Background

Childhood overweight and obesity is a serious public health issue with an increase being observed in preschool-aged children. Treating childhood obesity is difficult and few countries use standardized treatments. Therefore, there is a need to find effective approaches that are feasible for both health care providers and families. Thus, the overall aim of this study is to assess the acceptance and effectiveness of a parent support program (the More and Less, ML) for the management of overweight and obesity followed by a mobile health (mHealth) program (the MINISTOP application) in a socially diverse population of families.

Methods/design

A two-arm, parallel design randomized controlled trial in 300 2-to 6-year-old children with overweight and obesity from Romania, Spain and Sweden (n = 100 from each). Following baseline assessments children are randomized into the intervention or control group in a 1:1 ratio. The intervention, the ML program, consists of 10-weekly group sessions which focus on evidence-based parenting practices, followed by the previously validated MINISTOP application for 6-months to support healthy eating and physical activity behaviors. The primary outcome is change in body mass index (BMI) z-score after 9-months and secondary outcomes include: waist circumference, eating behavior (Child Eating Behavior Questionnaire), parenting behavior (Comprehensive Feeding Practices Questionnaire), physical activity (ActiGraph wGT3x-BT), dietary patterns (based on metabolic markers from urine and 24 h dietary recalls), epigenetic and gut hormones (fasting blood samples), and the overall acceptance of the overweight and obesity management in young children (semi-structured interviews). Outcomes are measured at baseline and after: 10-weeks (only BMI z-score, waist circumference), 9-months (all outcomes), 15- and 21-months (all outcomes except physical activity, dietary patterns, epigenetics and gut hormones) post-baseline.

Discussion

This study will evaluate a parent support program for weight management in young children in three European countries. To boost the effect of the ML program the families will be supported by an app for 6-months. If the program is found to be effective, it has the potential to be implemented into routine care to reduce overweight and obesity in young children and the app could prove to be a viable option for sustained effects of the care provided.

Trial registration

ClinicalTrials.gov NCT03800823; 11 Jan 2019.