A Framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) in Digital Interventions: Viewpoint

A web-based physical activity intervention targeting affect regulation: a randomized feasibility trial

Early career professionals (ECPs) are a critical target for physical activity (PA) promotion. Affect contributes to an established PA intention-behaviour gap and is pertinent among ECPs.

OBJECTIVE

The purpose of this study was to examine the feasibility and acceptability of a web-based intervention and explore the effects on secondary outcomes (moderate-to-vigorous PA (MVPA), emotion regulation, multi-process action control constructs).

METHODS

Adults aged 25-44 who were employed at least part-time in a desk-based job and not meeting PA guidelines (<150 min MVPA) were recruited and randomized into a 6-week online intervention integrating acceptance and commitment principles and affect regulation strategies, or a control group.

RESULTS

Forty adults were recruited and randomized to the web-based intervention (n = 21) and waitlist control (n = 19). The recruitment rate was 29%, retention was 75%, engagement was 68%, and satisfaction was high in both quantitative and qualitative assessment. Participants allocated to the intervention improved MVPA (ηp2=0.30), emotion regulation (ηp2 =0.49), behavioural regulation (ηp2=0.53), affective attitude (ηp2=0.23), identity (ηp2=0.24), and constructs of mindfulness (ηp2=0.44), and valued living (ηp2=0.20).

CONCLUSIONS

Primary outcomes concerning feasibility were adequate and secondary outcomes improved, suggesting a full-scale randomized controlled trial is feasible with minor modifications. A large-scale study is warranted to establish intervention effectiveness.

Online behavioural interventions for children and young people with eczema: a quantitative evaluation

BACKGROUND

Two online behavioural interventions (one website for parents/carers of children with eczema; and one for young people with eczema) have been shown in randomised controlled trials to facilitate a sustained improvement in eczema severity.

AIM

To describe intervention use and examine potential mediators of intervention outcomes and contextual factors that may influence intervention delivery and outcomes.

DESIGN AND SETTING

Quantitative process evaluation in UK primary care.

METHOD

Parents/carers and young people were recruited through primary care. Intervention use was recorded and summarised descriptively. Logistic regression explored sociodemographic and other factors associated with intervention engagement. Mediation analysis investigated whether patient enablement (ability to understand and cope with health issues), treatment use, and barriers to adherence were mediators of intervention effect. Subgroup analysis compared intervention effects among pre-specified participant subsets.

RESULTS

A total of 340 parents/carers and 337 young people were recruited. Most parents/carers (87%, n = 148/171) and young people (91%, n = 153/168) in the intervention group viewed the core introduction by 24 weeks. At 24 weeks, users had spent approximately 20 minutes on average on the interventions. Among parents/carers, greater intervention engagement was associated with higher education levels, uncertainty about carrying out treatments, and doubts about treatment efficacy at baseline. Among young people, higher intervention use was associated with higher baseline eczema severity. Patient enablement (the ability to understand and cope with health issues) accounted for approximately 30% of the intervention effect among parents/carers and 50% among young people.

CONCLUSION

This study demonstrated that positive intervention outcomes depended on a modest time commitment from users. This provides further support that the wider implementation of Eczema Care Online is justified.

Talking in primary care (TIP): protocol for a cluster-randomised controlled trial in UK primary care to assess clinical and cost-effectiveness of communication skills e-learning for practitioners on patients' musculoskeletal pain and enablement

INTRODUCTION

Effective communication can help optimise healthcare interactions and patient outcomes. However, few interventions have been tested clinically, subjected to cost-effectiveness analysis or are sufficiently brief and well-described for implementation in primary care. This paper presents the protocol for determining the effectiveness and cost-effectiveness of a rigorously developed brief eLearning tool, EMPathicO, among patients with and without musculoskeletal pain.

METHODS AND ANALYSIS

A cluster randomised controlled trial in general practitioner (GP) surgeries in England and Wales serving patients from diverse geographic, socioeconomic and ethnic backgrounds. GP surgeries are randomised (1:1) to receive EMPathicO e-learning immediately, or at trial end. Eligible practitioners (eg, GPs, physiotherapists and nurse practitioners) are involved in managing primary care patients with musculoskeletal pain. Patient recruitment is managed by practice staff and researchers. Target recruitment is 840 adults with and 840 without musculoskeletal pain consulting face-to-face, by telephone or video. Patients complete web-based questionnaires at preconsultation baseline, 1 week and 1, 3 and 6 months later. There are two patient-reported primary outcomes: pain intensity and patient enablement. Cost-effectiveness is considered from the National Health Service and societal perspectives. Secondary and process measures include practitioner patterns of use of EMPathicO, practitioner-reported self-efficacy and intentions, patient-reported symptom severity, quality of life, satisfaction, perceptions of practitioner empathy and optimism, treatment expectancies, anxiety, depression and continuity of care. Purposive subsamples of patients, practitioners and practice staff take part in up to two qualitative, semistructured interviews.

ETHICS APPROVAL AND DISSEMINATION

Approved by the South Central Hampshire B Research Ethics Committee on 1 July 2022 and the Health Research Authority and Health and Care Research Wales on 6 July 2022 (REC reference 22/SC/0145; IRAS project ID 312208). Results will be disseminated via peer-reviewed academic publications, conference presentations and patient and practitioner outlets. If successful, EMPathicO could quickly be made available at a low cost to primary care practices across the country.

TRIAL REGISTRATION NUMBER

ISRCTN18010240.

Characterising user engagement with mHealth for chronic disease self-management and impact on machine learning performance

Mobile Health (mHealth) has the potential to be transformative in the management of chronic conditions. Machine learning can leverage self-reported data collected with apps to predict periods of increased health risk, alert users, and signpost interventions. Despite this, mHealth must balance the treatment burden of frequent self-reporting and predictive performance and safety. Here we report how user engagement with a widely used and clinically validated mHealth app, myCOPD (designed for the self-management of Chronic Obstructive Pulmonary Disease), directly impacts the performance of a machine learning model predicting an acute worsening of condition (i.e., exacerbations). We classify how users typically engage with myCOPD, finding that 60.3% of users engage frequently, however, less frequent users can show transitional engagement (18.4%), becoming more engaged immediately ( < 21 days) before exacerbating. Machine learning performed better for users who engaged the most, however, this performance decrease can be mostly offset for less frequent users who engage more near exacerbation. We conduct interviews and focus groups with myCOPD users, highlighting digital diaries and disease acuity as key factors for engagement. Users of mHealth can feel overburdened when self-reporting data necessary for predictive modelling and confidence of recognising exacerbations is a significant barrier to accurate self-reported data. We demonstrate that users of mHealth should be encouraged to engage when they notice changes to their condition (rather than clinically defined symptoms) to achieve data that is still predictive for machine learning, while reducing the likelihood of disengagement through desensitisation.

Shifting Mindsets: The Impact of a Patient Portal on Functioning and Recovery in a Mental Health Setting

OBJECTIVE

This study aims to understand whether higher use of a patient portal can have an impact on mental health functioning and recovery.

METHOD

A mixed methods approach was used for this study. In 2019-2021, patients with mental health diagnoses at outpatient clinics in an academic centre were invited to complete World Health Organization Disability Assessment Scale 12 (WHODAS-12) and Mental Health Recovery Measure surveys at baseline, 3 months, and 6 months after signing up for the portal. At the 3-month time point, patients were invited to a semistructured interview with a member of the team to contextualize the findings obtained from the surveys. Analytics data was also collected from the platform to understand usage patterns on the portal.

RESULTS

Overall, 113 participants were included in the analysis. There was no significant change in mental health functioning and recovery scores over the 6-month period. However, suboptimal usage was observed as 46% of participants did not complete any tasks within the portal. Thirty-five participants had low use of the portal (1-9 interactions) and 18 participants had high usage (10+ interactions). There were also no differences in mental health functioning and recovery scores between low and high users of the portal. Qualitative interviews highlighted many opportunities where the portal can support overall functioning and mental health recovery.

CONCLUSIONS

Collectively, this study suggests that higher use of a portal had no impact, either positive or negative, on mental health outcomes. While it may offer convenience and improved patient satisfaction, adequate support is needed to fully enable these opportunities for patient care. As the type of interaction with the portal was not specifically addressed, future work should focus on looking at ways to support patient engagement and portal usage throughout their care journey.

Evaluating a smartphone-based symptom self-monitoring app for psychosis in China (YouXin): A non-randomised validity and feasibility study with a mixed-methods design

Background

Psychosis causes a significant burden globally, including in China, where limited mental health resources hinder access to care. Smartphone-based remote monitoring offers a promising solution. This study aimed to assess the validity, feasibility, acceptability, and safety of a symptom self-monitoring smartphone app, YouXin, for people with psychosis in China.

Methods

A pre-registered non-randomised validity and feasibility study with a mixed-methods design. Participants with psychosis were recruited from a major tertiary psychiatric hospital in Beijing, China. Participants utilised the YouXin app to self-monitor psychosis and mood symptoms for four weeks. Feasibility outcomes were recruitment, retention and outcome measures completeness. Active symptom monitoring (ASM) validity was tested against corresponding clinical assessments (PANSS and CDS) using Spearman correlation. Ten participants completed qualitative interviews at study end to explore acceptability of the app and trial procedures.

Results

Feasibility parameters were met. The target recruitment sample of 40 participants was met, with 82.5% completing outcome measures, 60% achieving acceptable ASM engagement (completing >33% of all prompts), and 33% recording sufficient passive monitoring data to extract mobility indicators. Five ASM domains (hallucinations, suspiciousness, guilt feelings, delusions, grandiosity) achieved moderate correlation with clinical assessment. Both quantitative and qualitative evaluation showed high acceptability of YouXin. Clinical measurements indicated no symptom and functional deterioration. No adverse events were reported, suggesting YouXin is safe to use in this clinical population.

Conclusions

The trial feasibility, acceptability and safety parameters were met and a powered efficacy study is indicated. However, refinements are needed to improve ASM validity and increase passive monitoring data completeness.

User engagement in a randomised controlled trial for a digital health intervention for early psychosis (Actissist 2.0 trial)

Digital Health Interventions (DHIs) can help support people with mental health problems. Achieving satisfactory levels of patient engagement is a crucial, yet often underexplored, pre-requisite for health improvement. Actissist is a co-produced DHI delivered via a smartphone app for people with early psychosis, based on Cognitive Behaviour Therapy principles. This study describes and compares engagement patterns among participants in the two arms of the Actissist 2.0 randomised controlled trial. Engagement frequency and duration were measured among participants using the Actissist app in the intervention arm (n = 87) and the ClinTouch symptom monitoring only app used as the control condition (n = 81). Overall, 47.1 % of Actissist and 45.7 % of ClinTouch users completed at least a third of scheduled alerts while active in the study. The mean frequency (77.1 versus 60.2 total responses) and the median duration (80 versus 75 days until last response) of engagement were not significantly higher among Actissist users compared to ClinTouch users. Older age, White ethnicity, using their own smartphone device and, among Actissist users, an increased sense of therapeutic alliance were significantly associated with increased engagement. Through exploiting detailed usage data, this study identifies possible participant-level and DHI-level predictors of engagement to inform the practical implementation of future DHIs.

A Digital System (YouXin) to Facilitate Self-Management by People With Psychosis in China: Protocol for a Nonrandomized Validity and Feasibility Study With a Mixed Methods Design

BACKGROUND

Psychosis is one of the most disabling mental health conditions and causes significant personal, social, and economic burden. Accurate and timely symptom monitoring is critical to offering prompt and time-sensitive clinical services. Digital health is a promising solution for the barriers encountered by conventional symptom monitoring approaches, including accessibility, the ecological validity of assessments, and recall bias. However, to date, there has been no digital health technology developed to support self-management for people with psychosis in China.

OBJECTIVE

We report the study protocol to evaluate the validity, feasibility, acceptability, usability, and safety of a symptom self-monitoring smartphone app (YouXin; Chinese name ) for people with psychosis in China.

METHODS

This is a nonrandomized validity and feasibility study with a mixed methods design. The study was approved by the University of Manchester and Beijing Anding Hospital Research Ethics Committee. YouXin is a smartphone app designed to facilitate symptom self-monitoring for people with psychosis. YouXin has 2 core functions: active monitoring of symptoms (ie, smartphone survey) and passive monitoring of behavioral activity (ie, passive data collection via embedded smartphone sensors). The development process of YouXin utilized a systematic coproduction approach. A series of coproduction consultation meetings was conducted by the principal researcher with service users and clinicians to maximize the usability and acceptability of the app for end users. Participants with psychosis aged 16 years to 65 years were recruited from Beijing Anding Hospital, Beijing, China. All participants were invited to use the YouXin app to self-monitor symptoms for 4 weeks. At the end of the 4-week follow-up, we invited participants to take part in a qualitative interview to explore the acceptability of the app and trial procedures postintervention.

RESULTS

Recruitment to the study was initiated in August 2022. Of the 47 participants who were approached for the study from August 2022 to October 2022, 41 participants agreed to take part in the study. We excluded 1 of the 41 participants for not meeting the inclusion criteria, leaving a total of 40 participants who began the study. As of December 2022, 40 participants had completed the study, and the recruitment was complete.

CONCLUSIONS

This study is the first to develop and test a symptom self-monitoring app specifically designed for people with psychosis in China. If the study shows the feasibility of YouXin, a potential future direction is to integrate the app into clinical workflows to facilitate digital mental health care for people with psychosis in China. This study will inform improvements to the app, trial procedures, and implementation strategies with this population. Moreover, the findings of this trial could lead to optimization of digital health technologies designed for people with psychosis in China.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/45170.

Engagement with a nationally-implemented digital behaviour change intervention: Usage patterns over the 9-month duration of the National Health Service Digital Diabetes Prevention Programme

Background

Digital behaviour change interventions may offer a scalable way to promote weight loss by increasing physical activity and improving diet. However, user engagement is necessary for such benefits to be achieved. There is a dearth of research that assesses engagement with nationally implemented digital programmes offered in routine practice. The National Health Service Digital Diabetes Prevention Programme (NHS-DDPP) is a nine-month digital behaviour change intervention delivered by independent providers for adults in England who are at high risk of developing type 2 diabetes. This study reports engagement with the NHS-DDPP for users enrolled onto the programme over the nine-month duration.

Methods

Anonymous usage data was obtained for a cohort of service users (n = 1826) enrolled on the NHS-DDPP with three independent providers, between December 2020 and June 2021. Usage data were obtained for time spent in app, and frequency of use of NHS-DDPP intervention features in the apps including self-monitoring, goal setting, receiving educational content (via articles) and social support (via health coaches and group forums), to allow patterns of usage of these key features to be quantified across the nine-month intervention. Median usage was calculated within nine 30-day engagement periods to allow a longitudinal analysis of the dose of usage for each feature.

Results

App usage declined from a median of 32 min (IQR 191) in month one to 0 min (IQR 14) in month nine. Users self-monitored their behaviours (e.g., physical activity and diet) a median of 117 times (IQR 451) in the apps over the nine-month programme. The open group discussion forums were utilised less regularly (accessed a median of 0 times at all time-points). There was higher engagement with some intervention features (e.g., goal setting) when support from a health coach was linked to those features.

Conclusions

App usage decreased over the nine-month programme, although the rate at which the decrease occurred varied substantially between individuals and providers. Health coach support may promote engagement with specific intervention features. Future research should assess whether engagement with particular features of digital diabetes prevention programmes is associated with outcomes such as reduced bodyweight and HbA1c levels.

Exploring engagement with the CBT-informed Actissist smartphone application for early psychosis

BACKGROUND

Individuals with psychosis report favourable attitudes towards psychological interventions delivered via smartphone apps. Evidence for acceptability, safety, feasibility and efficacy is promising but in-depth reporting of app engagement in trials is sparse.

AIMS

To examine how people with psychosis engaged with the cognitive behaviour therapy (CBT)-informed Actissist app over a 12-week intervention period, and to examine factors associated with app engagement.

METHODS

Secondary data from participants in the intervention arm (n = 24) of a proof-of-concept randomised controlled trial of the Actissist app were analysed. The app prompted participants to engage with app-based CBT-informed material in five domains (voices, socialization, cannabis use, paranoia, perceived criticism) at pseudo-random intervals (three notifications per day, six days per week). Participants could self-initiate use any time. App use was financially incentivised.

RESULTS

Participants responded to 47% of app notifications. Most app engagements (87%) were app-initiated rather than self-initiated. Participants engaged most with the voices domain, then paranoia. Age and employment status were significantly associated with overall app engagement.

CONCLUSION

Individuals with psychosis engaged well with Actissist, particularly with areas focussing on voice-hearing and paranoia. App-generated reminders successfully prompted app engagement. As financial incentives may have increased app engagement, future studies of non-incentivized engagement in larger samples are needed.

The Role of Aesthetics in Intentions to Use Digital Health Interventions

Digital interventions are increasingly recognised as cost-effective treatment solutions for a number of health concerns, but adoption and use of these interventions can be low, affecting outcomes. This research sought to identify how individual aesthetic facets and perceived trust may influence perceptions toward and intentions to use an online health intervention by building on the Technology Acceptance Model, where perceived attractiveness, perceived usefulness, perceived ease of use and perceived enjoyment are thought to predict behavioural intentions towards a website. An online questionnaire study assessed perceptions of nine stimuli varying in four aesthetic facets (simplicity, diversity, colour & craftsmanship), utilising a quasi-experimental within-subjects design with a repetition among three different groups: individuals from the general population who were shown stimuli referring to general health (GP-H) (N = 257); individuals experiencing an eating disorder and shown stimuli referring to eating disorders (ED-ED) (N = 109); and individuals from the general population who were shown stimuli referring to eating disorders (GP-ED) (N = 235). Linear mixed models demonstrated that perceptions of simplicity and craftsmanship significantly influenced perceptions of usefulness, ease of use, enjoyment and trust, which in turn influenced behavioural intentions. This study demonstrates that developing the TAM model to add a further construct of perceived trust could be beneficial for digital health intervention developers. In this study, simplicity and craftsmanship were identified as the aesthetic facets with the greatest impact on user perceptions of digital health interventions.

A Digital Intervention to Improve Mental Health and Interpersonal Resilience in Young People Who Have Experienced Technology-Assisted Sexual Abuse: Protocol for a Nonrandomized Feasibility Clinical Trial and Nested Qualitative Study

BACKGROUND

No evidence-based support has been offered to young people (YP) who have experienced technology-assisted sexual abuse (TASA). Interventions aimed at improving mentalization (the ability to understand the mental states of oneself and others) are increasingly being applied to treat YP with various clinical issues. Digital technology use among YP is now common. A digital intervention aimed at improving mentalization in YP who have experienced TASA may reduce the risk of revictimization and future harm and make YP more resilient and able to manage distress that might result from TASA experiences.

OBJECTIVE

In this paper, we describe a protocol for determining the feasibility of the i-Minds trial and the acceptability, safety, and usability of the digital intervention (the i-Minds app) and explore how to best integrate i-Minds into existing routine care pathways.

METHODS

This is a mixed methods nonrandomized study aimed to determine the feasibility, acceptability, safety, and usability of the intervention. Participants aged between 12 and 18 years who report distress associated with TASA exposure will be recruited from the United Kingdom from the National Health Service (NHS) Trust Child and Adolescent Mental Health Services, sexual assault referral centers, and a web-based e-therapy provider. All participants will receive the i-Minds app for 6 weeks. Coproduced with YP and a range of stakeholders, the i-Minds app focuses on 4 main topics: mentalization, TASA and its impact, emotional and mental health, and trauma. A daily prompt will encourage YP to use the app, which is designed to be used in a stand-alone manner alongside routine care. We will follow participants up after the intervention and conduct interviews with stakeholders to explore the acceptability of the app and trial procedures and identify areas for improvement. Informed by the normalization process theory, we will examine barriers and enablers relevant to the future integration of the intervention into existing care pathways, including traditional clinic-based NHS and NHS e-therapy providers.

RESULTS

This study was approved by the Research Ethics Board of Scotland. We expect data to be collected from up to 60 YP. We expect to conduct approximately 20 qualitative interviews with participants and 20 health care professionals who referred YP to the study. The results of this study have been submitted for publication.

CONCLUSIONS

This study will provide preliminary evidence on the feasibility of recruiting YP to a trial of this nature and on the acceptability, safety, and usability of the i-Minds app, including how to best integrate it into existing routine care. The findings will inform the decision to proceed with a powered efficacy trial.

TRIAL REGISTRATION

International Standard Randomised Controlled Trial Number Registry (ISRCTN) ISRCTN43130832; https://www.isrctn.com/ISRCTN43130832.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/40539.

SATO (IDEAS expAnded wiTh BCIO): Workflow for designers of patient-centered mobile health behaviour change intervention applications

Designing effective theory-driven digital behaviour change interventions (DBCI) is a challenging task. To ease the design process, and assist with knowledge sharing and evaluation of the DBCI, we propose the SATO (IDEAS expAnded wiTh BCIO) design workflow based on the IDEAS (Integrate, Design, Assess, and Share) framework and aligned with the Behaviour Change Intervention Ontology (BCIO). BCIO is a structural representation of the knowledge in behaviour change domain supporting evaluation of behaviour change interventions (BCIs) but it is not straightforward to utilise it during DBCI design. IDEAS (Integrate, Design, Assess, and Share) framework guides multi-disciplinary teams through the mobile health (mHealth) application development life-cycle but it is not aligned with BCIO entities. SATO couples BCIO entities with workflow steps and extends IDEAS Integrate stage with consideration of customisation and personalisation. We provide a checklist of the activities that should be performed during intervention planning with concrete examples and a tutorial accompanied with case studies from the Cancer Better Life Experience (CAPABLE) European project. In the process of creating this workflow, we found the necessity to extend the BCIO to support the scenarios of multiple clinical goals in the same application. To ensure the SATO steps are easy to follow for the incomers to the field, we performed a preliminary evaluation of the workflow with two knowledge engineers, working on novel mHealth app design tasks.

Eczema Care Online behavioural interventions to support self-care for children and young people: two independent, pragmatic, randomised controlled trials

OBJECTIVE

To determine the effectiveness of two online behavioural interventions, one for parents and carers and one for young people, to support eczema self-management.

DESIGN

Two independent, pragmatic, parallel group, unmasked, randomised controlled trials.

SETTING

98 general practices in England.

PARTICIPANTS

Parents and carers of children (0-12 years) with eczema (trial 1) and young people (13-25 years) with eczema (trial 2), excluding people with inactive or very mild eczema (≤5 on POEM, the Patient-Oriented Eczema Measure).

INTERVENTIONS

Participants were randomised (1:1) using online software to receive usual eczema care or an online (www.EczemaCareOnline.org.uk) behavioural intervention for eczema plus usual care.

MAIN OUTCOME MEASURES

Primary outcome was eczema symptoms rated using POEM (range 0-28, with 28 being very severe) every four weeks over 24 weeks. Outcomes were reported by parents or carers for children and by self-report for young people. Secondary outcomes included POEM score every four weeks over 52 weeks, quality of life, eczema control, itch intensity (young people only), patient enablement, treatment use, perceived barriers to treatment use, and intervention use. Analyses were carried out separately for the two trials and according to intention-to-treat principles.

RESULTS

340 parents or carers of children (169 usual care; 171 intervention) and 337 young people (169 usual care; 168 intervention) were randomised. The mean baseline POEM score was 12.8 (standard deviation 5.3) for parents and carers and 15.2 (5.4) for young people. Three young people withdrew from follow-up but did not withdraw their data. All randomised participants were included in the analyses. At 24 weeks, follow-up rates were 91.5% (311/340) for parents or carers and 90.2% (304/337) for young people. After controlling for baseline eczema severity and confounders, compared with usual care groups over 24 weeks, eczema severity improved in the intervention groups: mean difference in POEM score -1.5 (95% confidence interval -2.5 to -0.6; P=0.002) for parents or carers and -1.9 (-3.0 to -0.8; P<0.001) for young people. The number needed to treat to achieve a 2.5 difference in POEM score at 24 weeks was 6 in both trials. Improvements were sustained to 52 weeks in both trials. Enablement showed a statistically significant difference favouring the intervention group in both trials: adjusted mean difference at 24 weeks -0.7 (95% confidence interval -1.0 to -0.4) for parents or carers and -0.9 (-1.3 to -0.6) for young people. No harms were identified in either group.

CONCLUSIONS

Two online interventions for self-management of eczema aimed at parents or carers of children with eczema and at young people with eczema provide a useful, sustained benefit in managing eczema severity in children and young people when offered in addition to usual eczema care.

TRIAL REGISTRATION

ISRCTN registry ISRCTN79282252.

The Roles of Adherence and Usage Activity in Adolescents' Intervention Gains During Brief Guided Online Acceptance and Commitment Therapy

OBJECTIVE

This study investigated the roles of adherence and usage activity in adolescents' (n = 161) gains during a 5-week web intervention program based on acceptance and commitment therapy (ACT).

METHOD

Program adherence was calculated as adherence percentage in relation to intended usage, whereas completion percentage, usage time, and usage weeks were used as indicators for usage activity. Subjective well-being was measured by self-reported life satisfaction and stress before and after the intervention.

RESULTS

First, regression analysis results showed that higher adherence predicted an increase in life satisfaction during intervention. Second, three subgroups of adolescents were identified using K-means cluster analysis in regard to adherence, usage activity and intervention gains: (1) "Adhered, committed users with relatively large intervention gains" (35%), (2) "Less committed users with no intervention gains" (42%), and (3) "Non-committed users with no intervention gains" (23%). The results showed that the highest gains from the Youth Compass intervention program are most likely obtained when the program is used as intended in its design. In addition, time investment and engagement in doing exercises seem as important as filling the minimum adherence criterion.

CONCLUSIONS

The results support the feasibility of ACT-based web intervention programs in promoting adolescent well-being, although more attention should be paid to motivating adolescents to commit to them and invest enough time in them.

A theory-based video intervention to enhance communication and engagement in online health communities: two experiments

Background

Online communities and social networking sites have great potential for supporting health behavior change. However, interventions vary greatly in participants’ engagement rates and, consequently, their effectiveness. Theory-based interventions in real-world contexts are needed to further increase engagement and effectiveness.

Methods

We experimentally tested whether a video intervention teaching Self-Determination-Theory-based communication strategies increases need-supportive communication strategy use over one week (Study 1, N = 76) and perceived need support, engagement, and goal attainment in a behavior change intervention supported by a forum-based online community (Study 2, N = 537). In Study 2, participants chose a goal (increasing either fruit or vegetable consumption or increasing moderate or vigorous physical activity) and joined an online community for 2 weeks. Data from both experiments were analyzed with mixed models and follow-up tests.

Results

In Study 1, participants in the intervention but not in the control group showed an increase in the number of need-supportive communication strategies used both immediately and one week after the intervention (condition × time interaction, partial η² = 0.31). In Study 2, participants who watched the intervention video had a higher number of postings and reported a higher subjective forum use frequency (but not a higher number of logins) compared to participants who watched the control video. However, the effect on the subjective forum visit frequency was not robust. There were no intervention effects on perceived need support, goal attainment, or secondary outcomes. The results might be explained by low application of need-supportive communication strategies.

Conclusion

A brief video intervention may be a suitable, low-cost intervention to promote need-supportive communication strategy use, benefitting both engagement and behavior change. Future studies should incorporate additional means to further improve communication strategy uptake and engagement in online communities.

A Digital Intervention for Respiratory Tract Infections (Internet Dr): Process Evaluation to Understand How to Support Self-care for Minor Ailments

Background

Approximately 57 million physician appointments annually in the United Kingdom are for minor ailments. These illnesses could be self-cared for, which would potentially lower patients’ anxiety, increase their confidence, and be more convenient. In a randomized controlled trial of the Internet Dr digital intervention, patients with access to the intervention had fewer consultations for respiratory tract infections (RTIs). Having established intervention efficacy, further examination of trial data is required to understand how the intervention works.

Objective

This paper reports a process evaluation of Internet Dr usage by the intervention group. The evaluation aims to demonstrate how meaningful usage metrics (ie, interactions that are specific and relevant to the intervention) can be derived from the theoretical principles underlying the intervention, then applied to examine whether these interactions are effective in supporting self-care for RTIs, for whom, and at what time.

Methods

The Internet Dr trial recorded patients’ characteristics and usage data over 24 weeks. At follow-up, users reported whether their levels of enablement to cope with their illness changed over the trial period. The Medical Research Council process evaluation guidance and checklists from the framework for Analyzing and Measuring Usage and Engagement Data were applied to structure research questions examining associations between usage and enablement.

Results

Viewing pages containing advice on caring for RTIs were identified as a meaningful metric for measuring intervention usage. Almost half of the users (616/1491, 42.31%) viewed at least one advice page, with most people (478/616, 77.6%) accessing them when they initially enrolled in the study. Users who viewed an advice page reported increased enablement to cope with their illness as a result of having participated in the study compared with users who did not (mean 2.12, SD 2.92 vs mean 1.65, SD 3.10; mean difference 0.469, 95% CI 0.082-0.856). The target population was users who had visited their general practitioners for an RTI in the year before the trial, and analyses revealed that this group was more likely to access advice pages (odds ratio 1.35, 95% CI 1.159-1.571; P<.001).

Conclusions

The process evaluation identifies viewing advice pages as associated with increased enablement to self-care, even when accessed in the absence of a RTI, meaning that dissemination activities need not be restricted to targeting users who are ill. The intervention was effective at reaching the target population of users who had previously consulted their general practitioners. However, attrition before reaching advice pages was high, highlighting the necessity of prioritizing access during the design phase. These findings provide guidance on how the intervention may be improved and disseminated and have wider implications for minor ailment interventions.

Estimating the impact of engagement with digital health interventions on patient outcomes in randomized trials

OBJECTIVE

Guidance is needed on studying engagement and treatment effects in digital health interventions, including levels required for benefit. We evaluated multiple analytic approaches for understanding the association between engagement and clinical outcomes.

MATERIALS AND METHODS

We defined engagement as intervention participants' response rate to interactive text messages, and considered moderation, standard regression, mediation, and a modified instrumental variable (IV) analysis to investigate the relationship between engagement and clinical outcomes. We applied each approach to two randomized controlled trials featuring text message content in the intervention: REACH (Rapid Encouragement/Education and Communications for Health), which targeted diabetes, and VERB (Vanderbilt Emergency Room Bundle), which targeted hypertension.

RESULTS

In REACH, the treatment effect on hemoglobin A1c was estimated to be -0.73% (95% CI: [-1.29, -0.21]; P = 0.008), and in VERB, the treatment effect on systolic blood pressure was estimated to be -10.1 mmHg (95% CI: [-17.7, -2.8]; P = 0.007). Only the IV analyses suggested an effect of engagement on outcomes; the difference in treatment effects between engagers and non-engagers was -0.29% to -0.51% in the REACH study and -1.08 to -3.25 mmHg in the VERB study.

DISCUSSION

Standard regression and mediation have less power than a modified IV analysis, but the IV approach requires specification of assumptions. This is the first review of the strengths and limitations of various approaches to evaluating the impact of engagement on outcomes.

CONCLUSIONS

Understanding the role of engagement in digital health interventions can help reveal when and how these interventions achieve desired outcomes.

The effectiveness of digital interventions for increasing physical activity in individuals of low socioeconomic status: a systematic review and meta-analysis

BACKGROUND

Digital technologies such as wearables, websites and mobile applications are increasingly used in interventions targeting physical activity (PA). Increasing access to such technologies makes an attractive prospect for helping individuals of low socioeconomic status (SES) in becoming more active and healthier. However, little is known about their effectiveness in such populations. The aim of this systematic review was to explore whether digital interventions were effective in promoting PA in low SES populations, whether interventions are of equal benefit to higher SES individuals and whether the number or type of behaviour change techniques (BCTs) used in digital PA interventions was associated with intervention effects.

METHODS

A systematic search strategy was used to identify eligible studies from MEDLINE, Embase, PsycINFO, Web of Science, Scopus and The Cochrane Library, published between January 1990 and March 2020. Randomised controlled trials, using digital technology as the primary intervention tool, and a control group that did not receive any digital technology-based intervention were included, provided they had a measure of PA as an outcome. Lastly, studies that did not have any measure of SES were excluded from the review. Risk of Bias was assessed using the Cochrane Risk of Bias tool version 2.

RESULTS

Of the 14,589 records initially identified, 19 studies were included in the final meta-analysis. Using random-effects models, in low SES there was a standardised mean difference (SMD (95%CI)) in PA between intervention and control groups of 0.06 (- 0.08,0.20). In high SES the SMD was 0.34 (0.22,0.45). Heterogeneity was modest in both low (I2 = 0.18) and high (I2 = 0) SES groups. The studies used a range of digital technologies and BCTs in their interventions, but the main findings were consistent across all of the sub-group analyses (digital interventions with a PA only focus, country, chronic disease, and duration of intervention) and there was no association with the number or type of BCTs.

DISCUSSION

Digital interventions targeting PA do not show equivalent efficacy for people of low and high SES. For people of low SES, there is no evidence that digital PA interventions are effective, irrespective of the behaviour change techniques used. In contrast, the same interventions in high SES participants do indicate effectiveness. To reduce inequalities and improve effectiveness, future development of digital interventions aimed at improving PA must make more effort to meet the needs of low SES people within the target population.

Feasibility of a Mobile Health App for Routine Outcome Monitoring and Feedback in SMART Recovery Mutual Support Groups: Stage 1 Mixed Methods Pilot Study

Background

Mutual support groups are an important source of long-term help for people impacted by addictive behaviors. Routine outcome monitoring (ROM) and feedback are yet to be implemented in these settings. SMART Recovery mutual support groups focus on self-empowerment and use evidence-based techniques (eg, motivational and behavioral strategies). Trained facilitators lead all SMART Recovery groups, providing an opportunity to implement ROM.

Objective

The aim of this stage 1 pilot study is to explore the feasibility, acceptability, and preliminary outcomes of a novel, purpose-built mobile health ROM and feedback app (SMART Track) in mutual support groups coordinated by SMART Recovery Australia (SRAU) over 8 weeks.

Methods

SMART Track was developed during phase 1 of this study using participatory design methods and an iterative development process. During phase 2, 72 SRAU group participants were recruited to a nonrandomized, prospective, single-arm trial of the SMART Track app. Four modes of data collection were used: ROM data directly entered by participants into the app; app data analytics captured by Amplitude Analytics (number of visits, number of unique users, visit duration, time of visit, and user retention); baseline, 2-, and 8-week follow-up assessments conducted through telephone; and qualitative telephone interviews with a convenience sample of study participants (20/72, 28%) and facilitators (n=8).

Results

Of the 72 study participants, 68 (94%) created a SMART Track account, 64 (88%) used SMART Track at least once, and 42 (58%) used the app for more than 5 weeks. During week 1, 83% (60/72) of participants entered ROM data for one or more outcomes, decreasing to 31% (22/72) by the end of 8 weeks. The two main screens designed to provide personal feedback data (Urges screen and Overall Progress screen) were the most frequently visited sections of the app. Qualitative feedback from participants and facilitators supported the acceptability of SMART Track and the need for improved integration into the SRAU groups. Participants reported significant reductions between the baseline and 8- week scores on the Severity of Dependence Scale (mean difference 1.93, SD 3.02; 95% CI 1.12-2.73) and the Kessler Psychological Distress Scale-10 (mean difference 3.96, SD 8.31; 95% CI 1.75-6.17), but no change on the Substance Use Recovery Evaluator (mean difference 0.11, SD 7.97; 95% CI –2.02 to 2.24) was reported.

Conclusions

Findings support the feasibility, acceptability, and utility of SMART Track. Given that sustained engagement with mobile health apps is notoriously difficult to achieve, our findings are promising. SMART Track offers a potential solution for ROM and personal feedback, particularly for people with substance use disorders who attend mutual support groups.

Trial Registration

Australian New Zealand Clinical Trials Registry ACTRN12619000686101; https://anzctr.org.au/Trial/Registration/TrialReview.aspx?id=377336

International Registered Report Identifier (IRRID)

RR2-10.2196/15113

A Web-Based Intervention (Germ Defence) to Increase Handwashing During a Pandemic: Process Evaluations of a Randomized Controlled Trial and Public Dissemination

Background

Washing hands helps prevent transmission of seasonal and pandemic respiratory viruses. In a randomized controlled trial (RCT) during the swine flu outbreak, participants with access to a fully automated, digital intervention promoting handwashing reported washing their hands more often and experienced fewer respiratory tract infections than those without access to the intervention. Based on these findings, the intervention was adapted, renamed as “Germ Defence,” and a study was designed to assess the preliminary dissemination of the intervention to the general public to help prevent the spread of seasonal colds and flu.

Objective

This study compares the process evaluations of the RCT and Germ Defence dissemination to examine (1) how web-based research enrollment procedures affected those who used the intervention, (2) intervention usage in the 2 contexts, and (3) whether increased intentions to wash hands are replicated once disseminated.

Methods

The RCT ran between 2010 and 2012 recruiting participants offline from general practices, with restricted access to the intervention (N=9155). Germ Defence was disseminated as an open access website for use by the general public from 2016 to 2019 (N=624). The process evaluation plan was developed using Medical Research Council guidance and the framework for Analyzing and Measuring Usage and Engagement Data. Both interventions contained a goal-setting section where users self-reported current and intended handwashing behavior across 7 situations.

Results

During web-based enrolment, 54.3% (17,511/32,250) of the RCT participants dropped out of the study compared to 36.5% (358/982) of Germ Defence users. Having reached the start of the intervention, 93.8% (8586/9155) of RCT users completed the core section, whereas 65.1% (406/624) of Germ Defence users reached the same point. Users across both studies selected to increase their handwashing in 5 out of 7 situations, including before eating snacks (RCT mean difference 1.040, 95% CI 1.016-1.063; Germ Defence mean difference 0.949, 95% CI 0.766-1.132) and after blowing their nose, sneezing, or coughing (RCT mean difference 0.995, 95% CI 0.972-1.019; Germ Defence mean difference 0.842, 95% CI 0.675-1.008).

Conclusions

By comparing the preliminary dissemination of Germ Defence to the RCT, we were able to examine the potential effects of the research procedures on uptake and attrition such as the sizeable dropout during the RCT enrolment procedure that may have led to a more motivated sample. The Germ Defence study highlighted the points of attrition within the intervention. Despite sample bias in the trial context, the intervention replicated increases in intentions to handwash when used “in the wild.” This preliminary dissemination study informed the adaptation of the intervention for the COVID-19 health emergency, and it has now been disseminated globally.

Trial Registration

ISRCTN Registry ISRCTN75058295; https://www.isrctn.com/ISRCTN75058295

[Adherence to digital health interventions: definitions, methods, and open questions]

Many digital interventions rely on the participation of their users to have a positive impact. In various areas it can be observed that the use of digital interventions is often reduced or fully discontinued by the users after a short period of time. This is seen as one of the main factors that can limit the effectiveness of digital interventions. In this context, the concept of adherence to digital interventions is becoming increasingly important. Adherence to digital interventions is roughly defined as "the degree to which the user followed the program as it was designed," which can also be paraphrased as "intended use" or "use as it is designed." However, both the theoretical-conceptual and practical discussions regarding adherence to digital interventions still receive too little attention.The aim of this narrative review article is to shed more light on the concept of adherence to digital interventions and to distinguish it from related concepts. It also discusses the methods and metrics that can be used to operationalize adherence and the predictors that positively influence adherence. Finally, needs for action to better address adherence are considered critically.

Evaluating Digital Program Support for the Physical Activity 4 Everyone (PA4E1) School Program: Mixed Methods Study

BACKGROUND

Effectively scaled-up physical activity interventions are urgently needed to address the high prevalence of physical inactivity. To facilitate scale-up of an efficacious school-based physical activity program (Physical Activity 4 Everyone [PA4E1]), provision of implementation support to physical education (PE) teachers was adapted from face-to-face and paper-based delivery modes to partial delivery via a website. A lack of engagement (usage and subjective experience) with digital delivery modes, including websites, may in part explain the typical reduction in effectiveness of scaled-up interventions that use digital delivery modes. A process evaluation focused on the PA4E1 website was undertaken.

OBJECTIVE

The 2 objectives were to (1) describe the usage of the PA4E1 program website by in-school champions (PE teachers leading the program within their schools) and PE teachers using quantitative methods; (2) examine the usage, subjective experience, and usability of the PA4E1 program website from the perspective of in-school champions using mixed methods.

METHODS

The first objective used website usage data collected across all users (n=273) throughout the 9 school terms of the PA4E1 implementation support. The 4 usage measures were sessions, page views, average session duration, and downloads. Descriptive statistics were calculated and explored across the duration of the 26-month program. The second objective used mixed methods, triangulating data from the first objective with data from a think-aloud survey and usability test completed by in-school champions (n=13) at 12 months. Qualitative data were analyzed thematically alongside descriptive statistics from the quantitative data in a triangulation matrix, generating cross-cutting themes using the "following a thread" approach.

RESULTS

For the first objective, in-school champions averaged 48.0 sessions per user, PE teachers 5.8 sessions. PE teacher sessions were of longer duration (10.5 vs 7.6 minutes) and included more page views (5.4 vs 3.4). The results from the mixed methods analysis for the second objective found 9 themes and 2 meta-themes. The first meta-theme indicated that the website was an acceptable and appropriate delivery mode, and usability of the website was high. The second meta-theme found that the website content was acceptable and appropriate, and identified specific suggestions for improvement.

CONCLUSIONS

Digital health interventions targeting physical activity often experience issues of lack of user engagement. By contrast, the findings from both the quantitative and mixed methods analyses indicate high usage and overall acceptability and appropriateness of the PA4E1 website to school teachers. The findings support the value of the website within a multidelivery mode implementation intervention to support schools to implement physical activity promoting practices. The analysis identified suggested intervention refinements, which may be adopted for future iterations and further scale-up of the PA4E1 program.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry ACTRN12617000681358; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=372870.

Identifying Content-Based Engagement Patterns in a Smoking Cessation Website and Associations With User Characteristics and Cessation Outcomes: A Sequence and Cluster Analysis

INTRODUCTION

Using WebQuit as a case study, a smoking cessation website grounded in Acceptance and Commitment Therapy, we aimed to identify sequence clusters of content usage and examine their associations with baseline characteristics, change to a key mechanism of action, and smoking cessation.

METHODS

Participants were adult smokers allocated to the WebQuit arm in a randomized controlled trial (n = 1,313). WebQuit contains theory-informed content including goal setting, self-monitoring and feedback, and values- and acceptance-based exercises. Sequence analysis was used to temporally order 30-s website usage segments for each participant. Similarities between sequences were assessed with the optimal matching distance algorithm and used as input in an agglomerative hierarchical clustering analysis. Associations between sequence clusters and baseline characteristics, acceptance of cravings at 3 months and self-reported 30-day point prevalence abstinence at 12 months were examined with linear and logistic regression.

RESULTS

Three qualitatively different sequence clusters were identified. "Disengagers" (576/1,313) almost exclusively used the goal-setting feature. "Tryers" (375/1,313) used goal setting and two of the values- and acceptance-based components ("Be Aware," "Be Willing"). "Committers" (362/1,313) primarily used two of the values- and acceptance-based components ("Be Willing," "Be Inspired"), goal setting, and self-monitoring and feedback. Compared with Disengagers, Committers demonstrated greater increases in acceptance of cravings (p = .01) and 64% greater odds of quit success (ORadj = 1.64, 95% CI = 1.18, 2.29, p = .003).

DISCUSSION

WebQuit users were categorized into Disengagers, Tryers, and Committers based on their qualitatively different content usage patterns. Committers saw increases in a key mechanism of action and greater odds of quit success.

IMPLICATIONS

This case study demonstrates how employing sequence and cluster analysis of usage data can help researchers and practitioners gain a better understanding of how users engage with a given eHealth intervention over time and use findings to test theory and/or to improve future iterations to the intervention. Future WebQuit users may benefit from being directed to the values- and acceptance-based and the self-monitoring and feedback components via reminders over the course of the program.

Feasibility randomised controlled trial comparing TRAK-ACL digital rehabilitation intervention plus treatment as usual versus treatment as usual for patients following anterior cruciate ligament reconstruction

Objectives

To evaluate the feasibility of trialling taxonomy for the rehabilitation of knee conditions—ACL (TRAK-ACL), a digital health intervention that provides health information, personalised exercise plans and remote clinical support combined with treatment as usual (TAU), for people following ACL reconstruction.

Methods

The study design was a two-arm parallel randomised controlled trial (RCT). Eligible participants were English-speaking adults who had undergone ACL reconstruction within the last 12 weeks, had access to the internet and could provide informed consent. Recruitment took place at three sites in the UK. TRAK-ACL intervention was an interactive website informed by behaviour change technique combined with TAU. The comparator was TAU. Outcomes were: recruitment and retention; completeness of outcome measures at follow-up; fidelity of intervention delivery and engagement with the intervention. Individuals were randomised using a computer-generated random number sequence. Blinded assessors allocated groups and collected outcome measures.

Results

Fifty-nine people were assessed for eligibility at two of the participating sites, and 51 were randomised; 26 were allocated to TRAK-ACL and 25 to TAU. Follow-up data were collected on 44 and 40 participants at 3 and 6 months, respectively. All outcome measures were completed fully at 6 months except the Client Service Receipt Inventory. Two patients in each arm did not receive the treatment they were randomised to. Engagement with TRAK-ACL intervention was a median of 5 logins (IQR 3–13 logins), over 18 weeks (SD 12.2 weeks).

Conclusion

TRAK-ACL would be suitable for evaluation of effectiveness in a fully powered RCT.

Adapting Behavioral Interventions for a Changing Public Health Context: A Worked Example of Implementing a Digital Intervention During a Global Pandemic Using Rapid Optimisation Methods

Background: A rigorous approach is needed to inform rapid adaptation and optimisation of behavioral interventions in evolving public health contexts, such as the Covid-19 pandemic. This helps ensure that interventions are relevant, persuasive, and feasible while remaining evidence-based. This paper provides a set of iterative methods to rapidly adapt and optimize an intervention during implementation. These methods are demonstrated through the example of optimizing an effective online handwashing intervention called Germ Defense. Methods: Three revised versions of the intervention were rapidly optimized and launched within short timeframes of 1-2 months. Optimisations were informed by: regular stakeholder engagement; emerging scientific evidence, and changing government guidance; rapid qualitative research (telephone think-aloud interviews and open-text surveys), and analyses of usage data. All feedback was rapidly collated, using the Table of Changes method from the Person-Based Approach to prioritize potential optimisations in terms of their likely impact on behavior change. Written feedback from stakeholders on each new iteration of the intervention also informed specific optimisations of the content. Results: Working closely with clinical stakeholders ensured that the intervention was clinically accurate, for example, confirming that information about transmission and exposure was consistent with evidence. Patient and Public Involvement (PPI) contributors identified important clarifications to intervention content, such as whether Covid-19 can be transmitted via air as well as surfaces, and ensured that information about difficult behaviors (such as self-isolation) was supportive and feasible. Iterative updates were made in line with emerging evidence, including changes to the information about face-coverings and opening windows. Qualitative research provided insights into barriers to engaging with the intervention and target behaviors, with open-text surveys providing a useful supplement to detailed think-aloud interviews. Usage data helped identify common points of disengagement, which guided decisions about optimisations. The Table of Changes was modified to facilitate rapid collation and prioritization of multiple sources of feedback to inform optimisations. Engagement with PPI informed the optimisation process. Conclusions: Rapid optimisation methods of this kind may in future be used to help improve the speed and efficiency of adaptation, optimization, and implementation of interventions, in line with calls for more rapid, pragmatic health research methods.

Assessing the usability and user engagement of Thought Spot - A digital mental health help-seeking solution for transition-aged youth

OBJECTIVE

To evaluate the perceived usability of and user engagement with a digital platform (Thought Spot) designed to enhance mental health and wellness help-seeking among transition-aged youth (TAY; 17-29-years old).

MATERIALS AND METHODS

Survey responses and usage patterns were collected as part of a randomized controlled trial evaluating the efficacy of Thought Spot. Participants given Thought Spot completed an adapted Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire to measure perceived usability of the platform. User engagement patterns on Thought Spot were examined using analytics data collected throughout the study (March 2018-June 2019).

RESULTS

A total of 131 transition-aged participants completed the USE questionnaire and logged on to Thought Spot at least once. Ease of learning scored higher than ease of use, usefulness and satisfaction. Participants identified numerous strengths and challenges related to usability, visual appeal, functionality and usefulness of the content. In terms of user engagement, most participants stopped using the platform after 3 weeks. Participants searched and were interested in a variety of resources, including mental health, counselling and social services.

DISCUSSION

Participants reported mixed experiences while using Thought Spot and exhibited low levels of long-term user engagement. User satisfaction, the willingness to recommend Thought Spot to others, and the willingness for future use appeared to be influenced by content relevance, ease of learning, available features, and other contextual factors. Analysis of the types of resources viewed and searches conducted by TAY end-users provided insight into their behaviour and needs.

CONCLUSION

Users had mixed perceptions about the usability of Thought Spot, which may have contributed to the high attrition rate. User satisfaction and engagement appears to be influenced by content relevance, ease of learning, and the types of features available. Further investigation to understand the contextual factors that affect TAYs' adoption and engagement with digital mental health tools is required.

Engagement With a Web-Based Intervention to Reduce Harmful Drinking: Secondary Analysis of a Randomized Controlled Trial

Background

Engagement with digital behavior change interventions (DBCIs) is considered a prerequisite for intervention efficacy. However, in many trials on DBCIs, participants use the intervention either only little or not at all.

Objective

To analyze engagement with a web-based intervention to reduce harmful drinking, we explored (1) whether engagement with a web-based alcohol intervention is related to drinking outcomes, (2) which user characteristics are associated with measures of engagement, and (3) whether reported outcomes are associated with data captured by voluntary intervention questionnaires.

Methods

We analyzed data of the intervention arm of a randomized controlled trial on a DBCI to reduce risky alcohol consumption. Data were collected at baseline (T0), after 90 days (T1), and at the end of the 180-day usage period (T2). Engagement with the intervention was measured via system usage data as well as self-reported usage. Drinking behavior was measured as average daily alcohol consumption as well as the number of binge drinking days. User characteristics included demographics, baseline drinking behavior, readiness to change, alcohol-related outcome expectancies, and alcohol abstinence self-efficacy. Following a bivariate approach, we performed two-tailed Welch’s t tests and Wilcoxon signed rank/Mann-Whitney U tests or calculated correlation coefficients.

Results

The data of 306 users were analyzed. Time spent engaging with the intervention as measured by system usage did not match self-reported usage. Higher self-reported usage was associated with higher reductions in average daily alcohol consumption (T1: ρ=0.39, P<.001; T2: ρ=0.29, P=.015) and in binge drinking days (T1: ρ=0.62, P<.001; T2: ρ=0.3, P=.006). Higher usage was reported from users who were single (T1: P<.001; T2: P<.001), users without children (T1: P<.001; T2: P<.001), users who did not start or finish secondary education (T1: P<.001; T2: P<.001), users without academic education (T1: P<.001; T2: P<.001), and those who worked (T1: P=.001; T2: P=.004). Relationships between self-reported usage and clinical or psychological baseline characteristics were complex. For system usage, the findings were mixed. Reductions in drinking captured by intervention questionnaires were associated with reported outcomes.

Conclusions

Though self-reported usage could be consistently linked to better outcomes and multiple user characteristics, our findings add to the overall inconclusive evidence that can be found throughout the literature. Our findings indicate potential benefits of self-reports as measures of engagement and intervention questionnaires as a basis for tailoring of intervention content. Future studies should adopt a theory-driven approach to engagement research utilizing psychometrically sound self-report questionnaires and include short ecological momentary assessments within the DBCIs.

Trial Registration

German Clinical Trials Register DRKS00006104; https://tinyurl.com/y22oc5jo

Understanding Self-Guided Web-Based Educational Interventions for Patients With Chronic Health Conditions: Systematic Review of Intervention Features and Adherence

Background

Chronic diseases contribute to 71% of deaths worldwide every year, and an estimated 15 million people between the ages of 30 and 69 years die mainly because of cardiovascular disease, cancer, chronic respiratory diseases, or diabetes. Web-based educational interventions may facilitate disease management. These are also considered to be a flexible and low-cost method to deliver tailored information to patients. Previous studies concluded that the implementation of different features and the degree of adherence to the intervention are key factors in determining the success of the intervention. However, limited research has been conducted to understand the acceptability of specific features and user adherence to self-guided web interventions.

Objective

This systematic review aims to understand how web-based intervention features are evaluated, to investigate their acceptability, and to describe how adherence to web-based self-guided interventions is defined and measured.

Methods

Studies published on self-guided web-based educational interventions for people (≥14 years old) with chronic health conditions published between January 2005 and June 2020 were reviewed following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement protocol. The search was performed using the PubMed, Cochrane Library, and EMBASE (Excerpta Medica dataBASE) databases; the reference lists of the selected articles were also reviewed. The comparison of the interventions and analysis of the features were based on the published content from the selected articles.

Results

A total of 20 studies were included. Seven principal features were identified, with goal setting, self-monitoring, and feedback being the most frequently used. The acceptability of the features was measured based on the comments collected from users, their association with clinical outcomes, or device adherence. The use of quizzes was positively reported by participants. Self-monitoring, goal setting, feedback, and discussion forums yielded mixed results. The negative acceptability was related to the choice of the discussion topic, lack of face-to-face contact, and technical issues. This review shows that the evaluation of adherence to educational interventions was inconsistent among the studies, limiting comparisons. A clear definition of adherence to an intervention is lacking.

Conclusions

Although limited information was available, it appears that features related to interaction and personalization are important for improving clinical outcomes and users’ experience. When designing web-based interventions, the selection of features should be based on the targeted population’s needs, the balance between positive and negative impacts of having human involvement in the intervention, and the reduction of technical barriers. There is a lack of consensus on the method of evaluating adherence to an intervention. Both investigations of the acceptability features and adherence should be considered when designing and evaluating web-based interventions. A proof-of-concept or pilot study would be useful for establishing the required level of engagement needed to define adherence.

Internet-Based Lifestyle Intervention to Prevent Type 2 Diabetes Through Healthy Habits: Design and 6-Month Usage Results of Randomized Controlled Trial

Background

Type 2 diabetes can be prevented through lifestyle changes, but sustainable and scalable lifestyle interventions are still lacking. Habit-based approaches offer an opportunity to induce long-term behavior changes.

Objective

The purposes of this study were to describe an internet-based lifestyle intervention for people at risk for type 2 diabetes targeted to support formation of healthy habits and explore its user engagement during the first 6 months of a randomized controlled trial (RCT).

Methods

The app provides an online store that offers more than 400 simple and contextualized habit-forming behavioral suggestions triggered by daily life activities. Users can browse, inspect, and select them; report their performances; and reflect on their own activities. Users can also get reminders, information on other users’ activities, and information on the prevention of type 2 diabetes. An unblended parallel RCT was carried out to evaluate the effectiveness of the app in comparison with routine care. User engagement is reported for the first 6 months of the trial based on the use log data of the participants, who were 18- to 70-year-old community-dwelling adults at an increased risk of type 2 diabetes.

Results

Of 3271 participants recruited online, 2909 were eligible to participate in the RCT. Participants were randomized using a computerized randomization system to the control group (n=971), internet-based intervention (digital, n=967), and internet-based intervention with face-to-face group coaching (F2F+digital, n=971). Mean age of control group participants was 55.0 years, digital group 55.2 years, and F2F+digital 55.2 years. The majority of participants were female, 81.1% (787/971) in the control group, 78.3% (757/967) in the digital group, and 80.7% (784/971) in the F2F+digital group. Of the participants allocated to the digital and F2F+digital groups, 99.53% (1929/1938) logged in to the app at least once, 98.55% (1901/1938) selected at least one habit, and 95.13% (1835/1938) reported at least one habit performance. The app was mostly used on a weekly basis. During the first 6 months, the number of active users on a weekly level varied from 93.05% (1795/1929) on week 1 to 51.79% (999/1929) on week 26. The daily use activity was not as high. The digital and F2F+digital groups used the app on a median of 23.0 and 24.5 days and for 79.4 and 85.1 minutes total duration, respectively. A total of 1,089,555 habit performances were reported during the first 6 months. There were no significant differences in the use metrics between the groups with regard to cumulative use metrics.

Conclusions

Results demonstrate that internet-based lifestyle interventions can be delivered to large groups including community-dwelling middle-aged and older adults, many with limited experience in digital app use, without additional user training. This intermediate analysis of use behavior showed relatively good engagement, with the percentage of active weekly users remaining over 50% at 6 months. However, we do not yet know if the weekly engagement was enough to change the lifestyles of the participants.

Trial Registration

ClinicalTrials.gov NCT03156478; https://clinicaltrials.gov/ct2/show/NCT03156478

User Engagement Among Diverse Adults in a 12-Month Text Message-Delivered Diabetes Support Intervention: Results from a Randomized Controlled Trial

Background

Text message–delivered interventions are a feasible and scalable approach for improving chronic disease self-care and reducing health disparities; however, information on long-term user engagement with these interventions is limited.

Objective

The aim of this study is to examine user engagement in a 12-month text message–delivered intervention supporting diabetes self-care, called REACH (Rapid Education/Encouragement And Communications for Health), among racially and socioeconomically diverse patients with type 2 diabetes (T2D). We explored time trends in engagement, associations between patient characteristics and engagement, and whether the addition of a human component or allowing patients to change their text frequency affected engagement. Qualitative data informed patients’ subjective experience of their engagement.

Methods

We recruited patients with T2D for a randomized trial evaluating mobile phone support relative to enhanced treatment as usual. This analysis was limited to participants assigned to the intervention. Participants completed a survey and hemoglobin A1c (HbA1c) test and received REACH text messages, including self-care promotion texts, interactive texts asking about medication adherence, and adherence feedback texts. For the first 6 months, texts were sent daily, and half of the participants also received monthly phone coaching. After 6 months, coaching stopped, and participants had the option to receive fewer texts for the subsequent 6 months. We defined engagement via responses to the interactive texts and responses to a follow-up interview. We used regression models to analyze associations with response rate and thematic and structural analysis to understand participants’ reasons for responding to the texts and their preferred text frequency.

Results

The participants were, on average, aged 55.8 (SD 9.8) years, 55.2% (137/248) female, and 52.0% (129/248) non-White; 40.7% (101/248) had ≤ a high school education, and 40.7% (101/248) had an annual household income <US $25,000. The median response rate to interactive texts was 91% (IQR 75%-97%) over 12 months. Engagement gradually declined throughout the intervention but remained high. Engagement did not differ by age, gender, education, income, diabetes duration, insulin status, health literacy, or numeracy. Black race and worse baseline medication adherence and HbA1c were each associated with lower engagement, although the effects were small. Nearly half of the participants chose to continue receiving daily texts for the last 6 months of the intervention. Participants who continued daily text messages said they wanted to continue experiencing benefits to their health, whereas those who chose fewer texts said that the daily texts had helped them create routines and they no longer needed them as often. Engagement was not impacted by receiving coaching or by participants’ chosen text frequency.

Conclusions

Well-designed interactive text messages can engage diverse patients in a self-care intervention for at least 1 year. Variation in and reasons for frequency preference suggest that offering a frequency choice may be important to users’ engagement.

Engagement with a Web-Based Health Promotion Intervention among Vocational School Students: A Secondary User and Usage Analysis

Engagement with web-based interventions is both generally low and typically declining. Visits and revisits remain a challenge. Based on log data of a web-based cluster randomized controlled trial conducted in vocational schools, the present secondary analysis aimed to identify influencing factors on initially logging in to a health promotion platform among young adults and to examine the engagement over the course of an eight-week intervention. Data of 336 students (62.2% female, age span 18–25) from two intervention arms (web-based intervention and web-based intervention with an additional initial face-to-face contact) was included. Binary logistic regression and log-data visualization were performed. An additional initial face-to-face contact (odds ratio (OR) = 2.971, p = 0.005), female sex (OR = 2.237, p = 0.046) and the health-related skill “dealing with health information” (OR = 2.179, p = 0.030) significantly increased the likelihood of initially logging in. Other variables showed no influence. 16.6% of all potential users logged in at least once, of which 57.4% revisited the platform. Most logins were tracked at the beginning of the intervention and repeated engagement was low. To increase the engagement with web-based interventions, health-related skills should be fostered. In addition, a strategy could be to interlink comparable interventions in vocational schools more regularly with everyday teaching through multi-component interventions.

Factors Influencing Motivation and Engagement in Mobile Health Among Patients With Sickle Cell Disease in Low-Prevalence, High-Income Countries: Qualitative Exploration of Patient Requirements

Background

Sickle cell disease (SCD) is a hematological genetic disease affecting over 25 million people worldwide. The main clinical manifestations of SCD, hemolytic anemia and vaso-occlusion, lead to chronic pain and organ damages. With recent advances in childhood care, high-income countries have seen SCD drift from a disease of early childhood mortality to a neglected chronic disease of adulthood. In particular, coordinated, preventive, and comprehensive care for adults with SCD is largely underresourced. Consequently, patients are left to self-manage. Mobile health (mHealth) apps for chronic disease self-management are now flooding app stores. However, evidence remains unclear about their effectiveness, and the literature indicates low user engagement and poor adoption rates. Finally, few apps have been developed for people with SCD and none encompasses their numerous and complex self-care management needs.

Objective

This study aimed to identify factors that may influence the long-term engagement and user adoption of mHealth among the particularly isolated community of adult patients with SCD living in low-prevalence, high-income countries.

Methods

Semistructured interviews were conducted. Interviews were audiotaped, transcribed verbatim, and analyzed using thematic analysis. Analysis was informed by the Braun and Clarke framework and mapped to the COM-B model (capability, opportunity, motivation, and behavior). Results were classified into high-level functional requirements (FRs) and nonfunctional requirements (NFRs) to guide the development of future mHealth interventions.

Results

Overall, 6 males and 4 females were interviewed (aged between 21 and 55 years). Thirty FRs and 31 NFRs were extracted from the analysis. Most participants (8/10) were concerned about increasing their physical capabilities being able to stop pain symptoms quickly. Regarding the psychological capability aspects, all interviewees desired to receive trustworthy feedback on their self-care management practices. About their physical opportunities, most (7/10) expressed a strong desire to receive alerts when they would reach their own physiological limitations (ie, during physical activity). Concerning social opportunity, most (9/10) reported wanting to learn about the self-care practices of other patients. Relating to motivational aspects, many interviewees (6/10) stressed their need to learn how to avoid the symptoms and live as normal a life as possible. Finally, NFRs included inconspicuousness and customizability of user experience, automatic data collection, data shareability, and data privacy.

Conclusions

Our findings suggest that motivation and engagement with mHealth technologies among the studied population could be increased by providing features that clearly benefit them. Self-management support and self-care decision aid are patients’ major demands. As the complexity of SCD self-management requires a high cognitive load, pervasive health technologies such as wearable sensors, implantable devices, or inconspicuous conversational user interfaces should be explored to ease it. Some of the required technologies already exist but must be integrated, bundled, adapted, or improved to meet the specific needs of people with SCD.

Engaging Users in the Behavior Change Process With Digitalized Motivational Interviewing and Gamification: Development and Feasibility Testing of the Precious App

Background

Most adults do not engage in sufficient physical activity to maintain good health. Smartphone apps are increasingly used to support physical activity but typically focus on tracking behaviors with no support for the complex process of behavior change. Tracking features do not engage all users, and apps could better reach their targets by engaging users in reflecting their reasons, capabilities, and opportunities to change. Motivational interviewing supports this active engagement in self-reflection and self-regulation by fostering psychological needs proposed by the self-determination theory (ie, autonomy, competence, and relatedness). However, it is unknown whether digitalized motivational interviewing in a smartphone app engages users in this process.

Objective

This study aimed to describe the theory- and evidence-based development of the Precious app and to examine how digitalized motivational interviewing using a smartphone app engages users in the behavior change process. Specifically, we aimed to determine if use of the Precious app elicits change talk in participants and how they perceive autonomy support in the app.

Methods

A multidisciplinary team built the Precious app to support engagement in the behavior change process. The Precious app targets reflective processes with motivational interviewing and spontaneous processes with gamified tools, and builds on the principles of self-determination theory and control theory by using 7 relational techniques and 12 behavior change techniques. The feasibility of the app was tested among 12 adults, who were asked to interact with the prototype and think aloud. Semistructured interviews allowed participants to extend their statements. Participants’ interactions with the app were video recorded, transcribed, and analyzed with deductive thematic analysis to identify the theoretical themes related to autonomy support and change talk.

Results

Participants valued the autonomy supportive features in the Precious app (eg, freedom to pursue personally relevant goals and receive tailored feedback). We identified the following five themes based on the theory-based theme autonomy support: valuing the chance to choose, concern about lack of autonomy, expecting controlling features, autonomous goals, and autonomy supportive feedback. The motivational interviewing features actively engaged participants in reflecting their outcome goals and reasons for activity, producing several types of change talk and very little sustain talk. The types of change talk identified were desire, need, reasons, ability, commitment, and taking steps toward change.

Conclusions

The Precious app takes a unique approach to engage users in the behavior change process by targeting both reflective and spontaneous processes. It allows motivational interviewing in a mobile form, supports psychological needs with relational techniques, and targets intrinsic motivation with gamified elements. The motivational interviewing approach shows promise, but the impact of its interactive features and tailored feedback needs to be studied over time. The Precious app is undergoing testing in a series of n-of-1 randomized controlled trials.

Assessing the Psychometric Properties of the Digital Behavior Change Intervention Engagement Scale in Users of an App for Reducing Alcohol Consumption: Evaluation Study

BACKGROUND

The level and type of engagement with digital behavior change interventions (DBCIs) are likely to influence their effectiveness, but validated self-report measures of engagement are lacking. The DBCI Engagement Scale was designed to assess behavioral (ie, amount, depth of use) and experiential (ie, attention, interest, enjoyment) dimensions of engagement.

OBJECTIVE

We aimed to assess the psychometric properties of the DBCI Engagement Scale in users of a smartphone app for reducing alcohol consumption.

METHODS

Participants (N=147) were UK-based, adult, excessive drinkers recruited via an online research platform. Participants downloaded the Drink Less app and completed the scale immediately after their first login in exchange for a financial reward. Criterion variables included the objectively recorded amount of use, depth of use, and subsequent login. Five types of validity (ie, construct, criterion, predictive, incremental, divergent) were examined in exploratory factor, correlational, and regression analyses. The Cronbach alpha was calculated to assess the scale's internal reliability. Covariates included motivation to reduce alcohol consumption.

RESULTS

Responses on the DBCI Engagement Scale could be characterized in terms of two largely independent subscales related to experience and behavior. The experiential and behavioral subscales showed high (α=.78) and moderate (α=.45) internal reliability, respectively. Total scale scores predicted future behavioral engagement (ie, subsequent login) with and without adjusting for users' motivation to reduce alcohol consumption (adjusted odds ratio [ORadj]=1.14; 95% CI 1.03-1.27; P=.01), which was driven by the experiential (ORadj=1.19; 95% CI 1.05-1.34; P=.006) but not the behavioral subscale.

CONCLUSIONS

The DBCI Engagement Scale assesses behavioral and experiential aspects of engagement. The behavioral subscale may not be a valid indicator of behavioral engagement. The experiential subscale can predict subsequent behavioral engagement with an app for reducing alcohol consumption. Further refinements and validation of the scale in larger samples and across different DBCIs are needed.

The Service of Research Analytics to Optimize Digital Health Evidence Generation: Multilevel Case Study

Background

The widespread adoption of digital health interventions for chronic disease self-management has catalyzed a paradigm shift in the selection of methodologies used to evidence them. Recently, the application of digital health research analytics has emerged as an efficient approach to evaluate these data-rich interventions. However, there is a growing mismatch between the promising evidence base emerging from analytics mediated trials and the complexity of introducing these novel research methods into evaluative practice.

Objective

This study aimed to generate transferable insights into the process of implementing research analytics to evaluate digital health interventions. We sought to answer the following two research questions: (1) how should the service of research analytics be designed to optimize digital health evidence generation? and (2) what are the challenges and opportunities to scale, spread, and sustain this service in evaluative practice?

Methods

We conducted a qualitative multilevel embedded single case study of implementing research analytics in evaluative practice that comprised a review of the policy and regulatory climate in Ontario (macro level), a field study of introducing a digital health analytics platform into evaluative practice (meso level), and interviews with digital health innovators on their perceptions of analytics and evaluation (microlevel).

Results

The practice of research analytics is an efficient and effective means of supporting digital health evidence generation. The introduction of a research analytics platform to evaluate effective engagement with digital health interventions into a busy research lab was ultimately accepted by research staff, became routinized in their evaluative practice, and optimized their existing mechanisms of log data analysis and interpretation. The capacity for research analytics to optimize digital health evaluations is highest when there is (1) a collaborative working relationship between research client and analytics service provider, (2) a data-driven research agenda, (3) a robust data infrastructure with clear documentation of analytic tags, (4) in-house software development expertise, and (5) a collective tolerance for methodological change.

Conclusions

Scientific methods and practices that can facilitate the agile trials needed to iterate and improve digital health interventions warrant continued implementation. The service of research analytics may help to accelerate the pace of digital health evidence generation and build a data-rich research infrastructure that enables continuous learning and evaluation.

Data Mining in the Development of Mobile Health Apps: Assessing In-App Navigation Through Markov Chain Analysis

Background

Mobile apps generate vast amounts of user data. In the mobile health (mHealth) domain, researchers are increasingly discovering the opportunities of log data to assess the usage of their mobile apps. To date, however, the analysis of these data are often limited to descriptive statistics. Using data mining techniques, log data can offer significantly deeper insights.

Objective

The purpose of this study was to assess how Markov Chain and sequence clustering analysis can be used to find meaningful usage patterns of mHealth apps.

Methods

Using the data of a 25-day field trial (n=22) of the Start2Cycle app, an app developed to encourage recreational cycling in adults, a transition matrix between the different pages of the app was composed. From this matrix, a Markov Chain was constructed, enabling intuitive user behavior analysis.

Results

Through visual inspection of the transitions, 3 types of app use could be distinguished (route tracking, gamification, and bug reporting). Markov Chain–based sequence clustering was subsequently used to demonstrate how clusters of session types can otherwise be obtained.

Conclusions

Using Markov Chains to assess in-app navigation presents a sound method to evaluate use of mHealth interventions. The insights can be used to evaluate app use and improve user experience.

Is More Always Better?: Discovering Incentivized mHealth Intervention Engagement Related to Health Behavior Trends

Behavioral medicine is devoting increasing attention to the topic of participant engagement and its role in effective mobile health (mHealth) behavioral interventions. Several definitions of the term "engagement" have been proposed and discussed, especially in the context of digital health behavioral interventions. We consider that engagement refers to specific interaction and use patterns with the mHealth tools such as smartphone applications for intervention, whereas adherence refers to compliance with the directives of the health intervention, independent of the mHealth tools. Through our analysis of participant interaction and self-reported behavioral data in a college student health study with incentives, we demonstrate an example of measuring "effective engagement" as engagement behaviors that can be linked to the goals of the desired intervention. We demonstrate how clustering of one year of weekly health behavior self-reports generate four interpretable clusters related to participants' adherence to the desired health behaviors: healthy and steady, unhealthy and steady, decliners, and improvers. Based on the intervention goals of this study (health promotion and behavioral change), we show that not all app usage metrics are indicative of the desired outcomes that create effective engagement. As such, mHealth intervention design might consider eliciting not just more engagement or use overall, but rather, effective engagement defined by use patterns related to the desired behavioral outcome.