Targeting subjective engagement in experimental therapeutics for digital mental health interventions

Opportunities and challenges in leveraging digital technology for mental health system strengthening: a systematic review to inform interventions in the United Arab Emirates

Digital technology offers scalable, real-time interventions for mental health promotion and treatment. This systematic review explores the opportunities and challenges associated with the use of digital technology in mental health, with a focus on informing mental health system strengthening interventions in the United Arab Emirates (UAE). Following PRISMA guidelines, a systematic search of databases was conducted up to August 2023 and identified a total of 8479 citations of which 114 studies were included in the qualitative analysis. The included studies encompass diverse digital interventions, platforms, and modalities used across various mental health conditions. The review identifies feasible, acceptable, and efficacious interventions, ranging from telehealth and mobile apps to virtual reality and machine learning models. Opportunities for improving access to care, reducing patients' transfers, and utilizing real-world interaction data for symptom monitoring are highlighted. However, challenges such as digital exclusion, privacy concerns, and potential service replacement caution policymakers. This study serves as a valuable evidence base for policymakers and mental health stakeholders in the UAE to navigate the integration of digital technology in mental health services effectively.

RegulEm, an unified protocol based-app for the treatment of emotional disorders: a parallel mixed methods usability and quality study

BACKGROUND

Interest in mental health smartphone applications has grown in recent years. Despite their effectiveness and advantages, special attention needs to be paid to two aspects to ensure app engagement: to include patients and professionals in their design and to guarantee their usability. The aim of this study was to analyse the perceived usability and quality of the preliminary version of RegulEm, an app based in the Unified Protocol, as part of the second stage of the app development.

METHODS

A parallel mixed methods study was used with 7 professionals and 4 users who were previously involved in the first stage of the development of the app. MARS, uMARS and SUS scales were used, and two focus groups were conducted. Descriptive statistical analysis and a thematic content analysis were performed in order to gather as much information as possible on RegulEm's usability and quality as well as suggestions for improvement.

RESULTS

RegulEm's usability was perceived through the SUS scale scores as good by users (75 points) and excellent by professionals (84.64 points), while its quality was perceived through the uMARS and MARS scales as good by both groups, with 4 and 4.14 points out of 5. Different areas regarding RegulEm's usability and suggestions for improvement were identified in both focus groups and 20% of the suggestions proposed were implemented in the refined version of RegulEm.

CONCLUSION

RegulEm's usability and quality were perceived as good by users and professionals and different identified areas have contributed to its refinement. This study provides a more complete picture of RegulEm's usability and quality prior analysing its effectiveness, implementation and cost-effectiveness in Spanish public mental health units.

Research Into Digital Health Intervention for Mental Health: 25-Year Retrospective on the Ethical and Legal Challenges

Digital mental health interventions are routinely integrated into mental health services internationally and can contribute to reducing the global mental health treatment gap identified by the World Health Organization. Research teams designing and delivering evaluations frequently invest substantial effort in deliberating on ethical and legal challenges around digital mental health interventions. In this article, we reflect on our own research experience with digital mental health intervention design and evaluation to identify 8 of the most critical challenges that we or others have faced, and that have ethical or legal consequences. These include: (1) harm caused by online recruitment work; (2) monitoring of intervention safety; (3) exclusion of specific demographic or clinical groups; (4) inadequate robustness of effectiveness and cost-effectiveness findings; (5) adequately conceptualizing and supporting engagement and adherence; (6) structural barriers to implementation; (7) data protection and intellectual property; and (8) regulatory ambiguity relating to digital mental health interventions that are medical devices. As we describe these challenges, we have highlighted serious consequences that can or have occurred, such as substantial delays to studies if regulations around Software as a Medical Device (SaMD) are not fully understood, or if regulations change substantially during the study lifecycle. Collectively, the challenges we have identified highlight a substantial body of required knowledge and expertise, either within the team or through access to external experts. Ensuring access to knowledge requires careful planning and adequate financial resources (for example, paying public contributors to engage in debate on critical ethical issues or paying for legal opinions on regulatory issues). Access to such resources can be planned for on a per-study basis and enabled through funding proposals. However, organizations regularly engaged in the development and evaluation of digital mental health interventions should consider creating or supporting structures such as advisory groups that can retain necessary competencies, such as in medical device regulation.

Measuring digital intervention user experience with a novel ecological momentary assessment (EMA) method, CORTO

Digital interventions often suffer from low usage, which may reflect insufficient attention to user experience. Moreover, the existing evaluation methods have limited applicability in the remote study of user experience of complex interventions that have expansive content and that are used over an extensive period of time. To alleviate these challenges, we describe here a novel qualitative Ecological Momentary Assessment (EMA) method: the CORTO method (Contextual, One-item, Repeated, Timely, Open-ended). We used it to gather digital intervention user experience data from Finnish adults (n = 184) who lived with interview-confirmed major depressive disorder (MDD) and took part in a randomized controlled trial (RCT) that studied the efficacy of a novel 12-week game-based digital intervention for depression. A second dataset on user experience was gathered with retrospective interviews (n = 22). We inductively coded the CORTO method and retrospective interview data, which led to four user experience categories: (1) contextual use, (2) interaction-elicited emotional experience, (3) usability, and (4) technical issues. Then, we used the created user experience categories and Template Analysis to analyze both datasets together, and reported the results qualitatively. Finally, we compared the two datasets with each other. We found that the data generated with the CORTO method offered more insights into usability and technical categories than the interview data that particularly illustrated the contextual use. The emotional valence of the interview data was more positive compared with the CORTO data. Both the CORTO and interview data detected 55 % of the micro-level categories; 20 % of micro-level categories were only detected by the CORTO data and 25 % only by the interview data. We found that the during-intervention user experience measurement with the CORTO method can provide intervention-specific insights, and thereby further the iterative user-centered intervention development. Overall, these findings highlight the impact of evaluation methods on the categories and qualities of insights acquired in intervention research.

User Engagement Clusters of an 8-Week Digital Mental Health Intervention Guided by a Relational Agent (Woebot): Exploratory Study

BACKGROUND

With the proliferation of digital mental health interventions (DMHIs) guided by relational agents, little is known about the behavioral, cognitive, and affective engagement components associated with symptom improvement over time. Obtaining a better understanding could lend clues about recommended use for particular subgroups of the population, the potency of different intervention components, and the mechanisms underlying the intervention's success.

OBJECTIVE

This exploratory study applied clustering techniques to a range of engagement indicators, which were mapped to the intervention's active components and the connect, attend, participate, and enact (CAPE) model, to examine the prevalence and characterization of each identified cluster among users of a relational agent-guided DMHI.

METHODS

We invited adults aged 18 years or older who were interested in using digital support to help with mood management or stress reduction through social media to participate in an 8-week DMHI guided by a natural language processing-supported relational agent, Woebot. Users completed assessments of affective and cognitive engagement, working alliance as measured by goal and task working alliance subscale scores, and enactment (ie, application of therapeutic recommendations in real-world settings). The app passively collected data on behavioral engagement (ie, utilization). We applied agglomerative hierarchical clustering analysis to the engagement indicators to identify the number of clusters that provided the best fit to the data collected, characterized the clusters, and then examined associations with baseline demographic and clinical characteristics as well as mental health outcomes at week 8.

RESULTS

Exploratory analyses (n=202) supported 3 clusters: (1) "typical utilizers" (n=81, 40%), who had intermediate levels of behavioral engagement; (2) "early utilizers" (n=58, 29%), who had the nominally highest levels of behavioral engagement in week 1; and (3) "efficient engagers" (n=63, 31%), who had significantly higher levels of affective and cognitive engagement but the lowest level of behavioral engagement. With respect to mental health baseline and outcome measures, efficient engagers had significantly higher levels of baseline resilience (P<.001) and greater declines in depressive symptoms (P=.01) and stress (P=.01) from baseline to week 8 compared to typical utilizers. Significant differences across clusters were found by age, gender identity, race and ethnicity, sexual orientation, education, and insurance coverage. The main analytic findings remained robust in sensitivity analyses.

CONCLUSIONS

There were 3 distinct engagement clusters found, each with distinct baseline demographic and clinical traits and mental health outcomes. Additional research is needed to inform fine-grained recommendations regarding optimal engagement and to determine the best sequence of particular intervention components with known potency. The findings represent an important first step in disentangling the complex interplay between different affective, cognitive, and behavioral engagement indicators and outcomes associated with use of a DMHI incorporating a natural language processing-supported relational agent.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05672745; https://classic.clinicaltrials.gov/ct2/show/NCT05672745.

The Effect of a Digital Mental Health Program on Anxiety and Depression Symptoms: Retrospective Analysis of Clinical Severity

BACKGROUND

Evidence-based digital health programs have shown efficacy in being primary tools to improve emotional and mental health, as well as offering supplementary support to individuals undergoing psychotherapy for anxiety, depression, and other mental health disorders. However, information is lacking about the dose response to digital mental health interventions.

OBJECTIVE

The objective of the study was to examine the effect of time in program and program usage on symptom change among individuals enrolled in a real-world comprehensive digital mental health program (myStrength) who are experiencing severe anxiety or depression.

METHODS

Eligible participants (N=18,626) were adults aged 18 years and older who were enrolled in myStrength for at least four weeks as part of their employee wellness benefit program, who completed baseline, the 2-week, 2-month, and 6-month surveys querying symptoms of anxiety (Generalized Anxiety Disorder-7 [GAD-7]) and depression (Patient Health Questionnaire-9 [PHQ-9]). Linear growth curve models were used to analyze the effect of average weekly program usage on subsequent GAD-7 and PHQ-9 scores for participants with scores indicating severe anxiety (GAD-7≥15) or depression (PHQ-9≥15). All models were adjusted for baseline score and demographics.

RESULTS

Participants in the study (N=1519) were 77.4% female (1176/1519), had a mean age of 45 years (SD 14 years), and had an average enrollment time of 3 months. At baseline, participants reported an average of 9.39 (SD 6.04) on the GAD-7 and 11.0 (SD 6.6) on the PHQ-9. Those who reported 6-month results had an average of 8.18 (SD 6.15) on the GAD-7 and 9.18 (SD 6.79) on the PHQ-9. Participants with severe scores (n=506) experienced a significant improvement of 2.97 (SE 0.35) and 3.97 (SE 0.46) at each time point for anxiety and depression, respectively (t=-8.53 and t=-8.69, respectively; Ps<.001). Those with severe baseline scores also saw a reduction of 0.27 (SE 0.08) and 0.25 (SE 0.09) points in anxiety and depression, respectively, for each additional program activity per week (t=-3.47 and t=-2.66, respectively; Ps<.05).

CONCLUSIONS

For participants with severe baseline scores, the study found a clinically significant reduction of approximately 9 points for anxiety and 12 points for depression after 6 months of enrollment, suggesting that interventions targeting mental health must maintain active, ongoing engagement when symptoms are present and be available as a continuous resource to maximize clinical impact, specifically in those experiencing severe anxiety or depression. Moreover, a dosing effect was shown, indicating improvement in outcomes among participants who engaged with the program every other day for both anxiety and depression. This suggests that digital mental health programs that provide both interesting and evidence-based activities could be more successful in further improving mental health outcomes.

Measuring Engagement with Mental Health and Behavior Change Interventions: an Integrative Review of Methods and Instruments

BACKGROUND

Engagement is a complex construct consisting of behavioral, cognitive, and affective dimensions, making engagement a difficult construct to measure. This integrative review aims to (1) present a multidisciplinary overview of measurement methods that are currently used to measure engagement with adult mental health and behavior change interventions, delivered in-person, blended, or digitally, and (2) provide a set of recommendations and considerations for researchers wishing to study engagement.

METHODS

We used an integrative approach and identified original studies and reviews on engagement with mental health or behavior change interventions that were delivered in-person, digitally, or blended.

RESULTS

Forty articles were analyzed in this review. Common methods to assess engagement were through objective usage data, questionnaire-based data, and qualitative data, with objective usage data being used most frequently. Based on the synthesis of engagement measures, we advise researchers to (1) predefine the operationalization of engagement for their specific research context, (2) measure behavioral, cognitive, and affective dimensions of engagement in all cases, and (3) measure engagement over time.

CONCLUSIONS

Current literature shows a bias towards behavioral measures of engagement in research, as most studies measured engagement exclusively through objective usage data, without including cognitive and affective measures of engagement. We hope that our recommendations will help to reduce this bias and to steer engagement research towards an integrated approach.

Examining young adults daily perspectives on usage of anxiety apps: A user study

The growing number of mental health smartphone applications has led to increased interest in how these tools might support users in different models of care. However, research on the use of these interventions in real-world settings has been scarce. It is important to understand how apps are used in a deployment setting, especially among populations where such tools might add value to current models of care. The objective of this study is to explore the daily use of commercially-available mobile apps for anxiety that integrate CBT, with a focus on understanding reasons for and barriers for app use and engagement. This study recruited 17 young adults (age M = 24.17 years) while on a waiting list to receive therapy in a Student Counselling Service. Participants were asked to select up to two of a list of three selected apps (Wysa, Woebot, and Sanvello) and instructed to use the apps for two weeks. Apps were selected because they used techniques from cognitive behavioral therapy, and offer diverse functionality for anxiety management. Qualitative and quantitative data were gathered through daily questionnaires to capture participants' experiences with the mobile apps. In addition, eleven semi-structured interviews were conducted at the end of the study. We used descriptive statistics to analyze participants' interaction with different app features and used a general inductive approach to analyze the collected qualitative data. The results highlight that users form opinions about the apps during the first days of app use. A number of barriers to sustained use are identified including cost-related issues, inadequate content to support long-term use, and a lack of customization options for different app functions. The app features used differ among participants with self-monitoring and treatment elements being the most used features.

User engagement in mobile apps for people with schizophrenia: A scoping review

Over the past decade, there has been an increase in the number of mobile apps designed for mental health proposes and mHealth has been perceived as a promising approach to help people with schizophrenia to manage their condition. However, adoption rates are relatively low and long-term user engagement is a major issue. The aim of this study is to identify and better understand what strategies and factors may influence user engagement and facilitate prolonged use of apps for people with schizophrenia to better manage their illness. A scoping review was conducted in accordance with the Arksey and O'Malley scoping review framework and following PRISMA ScR guidelines. The sources consisted of searching four electronic databases. Rayyan software was used for this study selection process and a narrative approach was used to synthesize the extracted data. A total of 28 studies which met the inclusion criteria were identified. The engagement strategies included push notifications, message prompts, personalization, application customization, goal setting, game-like features, use of different multimedia formats, social connectedness, support (peers and professionals), reliability of content and quality of feedback received. Some demographic factors may influence adherence such as age, gender, education level and socioeconomic status. Other factors also may play a role impacting engagement: health status, data privacy and security, involvement in design process, incentives for participation, app usage fitting in the user routines, initial training, and constant technical support. Included studies present high heterogeneity in outcome measures and thresholds criteria to assess engagement. Understanding what influences engagement and how to measure it is essential to enhance the design of mobile apps and deliver scalable solutions to help people with schizophrenia better manage their illness in their real-world uptake.

Understanding users’ perspectives on mobile apps for anxiety management

Anxiety disorders are the most common type of mental health problem. The potential of apps to improve mental health has led to an increase in the number of anxiety apps available. Even though anxiety apps hold the potential to enhance mental health care for individuals, there is relatively little knowledge concerning users’ perspectives. This mixed-methods study aims to understand the nature of user burden and engagement with mental health apps (MHapps) targeting anxiety management, in order to identify ways to improve the design of these apps. Users’ perspectives on these apps were gathered by analyzing 600 reviews from 5 apps on the app stores (Study 1), and conducting 15 interviews with app users (Study 2). The results shed light on several barriers to adoption and sustained use. Users appreciate apps that offer content variation, customizability, and good interface design, and often requested an enhanced, personalized experience to improve engagement. We propose addressing the specific app quality issues identified through human-centered design, more personalized content delivery, and by improving features for social and therapeutic support.

Operationalizing Engagement With an Interpretation Bias Smartphone App Intervention: Case Series

BACKGROUND

Engagement with mental health smartphone apps is an understudied but critical construct to understand in the pursuit of improved efficacy.

OBJECTIVE

This study aimed to examine engagement as a multidimensional construct for a novel app called HabitWorks. HabitWorks delivers a personalized interpretation bias intervention and includes various strategies to enhance engagement such as human support, personalization, and self-monitoring.

METHODS

We examined app use in a pilot study (n=31) and identified 5 patterns of behavioral engagement: consistently low, drop-off, adherent, high diary, and superuser.

RESULTS

We present a series of cases (5/31, 16%) from this trial to illustrate the patterns of behavioral engagement and cognitive and affective engagement for each case. With rich participant-level data, we emphasize the diverse engagement patterns and the necessity of studying engagement as a heterogeneous and multifaceted construct.

CONCLUSIONS

Our thorough idiographic exploration of engagement with HabitWorks provides an example of how to operationalize engagement for other mental health apps.

User Experience, Engagement, and Popularity in Mental Health Apps: Secondary Analysis of App Analytics and Expert App Reviews

Background

User experience and engagement are critical elements of mental health apps’ abilities to support users. However, work examining the relationships among user experience, engagement, and popularity has been limited. Understanding how user experience relates to engagement with and popularity of mental health apps can demonstrate the relationship between subjective and objective measures of app use. In turn, this may inform efforts to develop more effective and appealing mental health apps and ensure that they reach wide audiences.

Objective

We aimed to examine the relationship among subjective measures of user experience, objective measures of popularity, and engagement in mental health apps.

Methods

We conducted a preregistered secondary data analysis in a sample of 56 mental health apps. To measure user experience, we used expert ratings on the Mobile App Rating Scale (MARS) and consumer ratings from the Apple App Store and Google Play. To measure engagement, we acquired estimates of monthly active users (MAU) and user retention. To measure app popularity, we used download count, total app revenue, and MAU again.

Results

MARS total score was moderately positively correlated with app-level revenue (Kendall rank [T]=0.30, P=.002), MAU (T=0.39, P<.001), and downloads (T=0.41, P<.001). However, the MARS total score and each of its subscales (Engagement, Functionality, Aesthetics, and Information) showed extremely small correlations with user retention 1, 7, and 30 days after downloading. Furthermore, the total MARS score only correlated with app store rating at T=0.12, which, at P=.20, did not meet our threshold for significance.

Conclusions

More popular mental health apps receive better ratings of user experience than less popular ones. However, user experience does not predict sustained engagement with mental health apps. Thus, mental health app developers and evaluators need to better understand user experience and engagement, as well as to define sustained engagement, what leads to it, and how to create products that achieve it. This understanding might be supported by better collaboration between industry and academic teams to advance a science of engagement.

Engagement with mobile health interventions for depression: A systematic review

BACKGROUND

Depressive disorders are a major public health problem, and many people face barriers to accessing evidence-based mental health treatment. Mobile health (mHealth) interventions may circumvent logistical barriers to in-person care (e.g., cost, transportation), however the symptoms of depression (low motivation, concentration difficulties) may make it difficult for people with the disorder to engage with mHealth.

OBJECTIVE

The aim of this systematic review is to examine assessment and reporting of engagement in clinical trials of mHealth interventions for depression, including objective engagement (e.g., number of times program is used), subjective engagement (e.g., qualitative data on users' experiences), and associations between engagement and other clinically important variables (e.g., symptom improvement, participant characteristics).

METHODS

Three electronic databases (PsycINFO, Web of Science, PubMed) were searched in February 2020 using search terms for mHealth and depression. Studies were included in the review if they tested a mHealth intervention designed for people with depressive disorders or elevated depression symptoms.

RESULTS

Thirty studies met inclusion criteria and were reviewed. Most studies reported objective engagement (N = 23, 76.7%), approximately half reported subjective engagement (N = 16, 53.3%), and relatively few examined associations between engagement and clinical improvement, participant characteristics, or other clinically relevant variables (N = 13, 43.3%).

CONCLUSIONS

Although most studies in this small but rapidly growing literature report at least one measure of engagement, there is substantial heterogeneity. Intentional, theory-driven, and consistent measurement of engagement with mHealth interventions for depression may advance the field's understanding of effective engagement to facilitate clinical improvement, identify dose-response relationships, and maximize generalizability for underserved populations.