Usage Metrics of Web-Based Interventions Evaluated in Randomized Controlled Trials: Systematic Review

Research Into Digital Health Intervention for Mental Health: 25-Year Retrospective on the Ethical and Legal Challenges

Digital mental health interventions are routinely integrated into mental health services internationally and can contribute to reducing the global mental health treatment gap identified by the World Health Organization. Research teams designing and delivering evaluations frequently invest substantial effort in deliberating on ethical and legal challenges around digital mental health interventions. In this article, we reflect on our own research experience with digital mental health intervention design and evaluation to identify 8 of the most critical challenges that we or others have faced, and that have ethical or legal consequences. These include: (1) harm caused by online recruitment work; (2) monitoring of intervention safety; (3) exclusion of specific demographic or clinical groups; (4) inadequate robustness of effectiveness and cost-effectiveness findings; (5) adequately conceptualizing and supporting engagement and adherence; (6) structural barriers to implementation; (7) data protection and intellectual property; and (8) regulatory ambiguity relating to digital mental health interventions that are medical devices. As we describe these challenges, we have highlighted serious consequences that can or have occurred, such as substantial delays to studies if regulations around Software as a Medical Device (SaMD) are not fully understood, or if regulations change substantially during the study lifecycle. Collectively, the challenges we have identified highlight a substantial body of required knowledge and expertise, either within the team or through access to external experts. Ensuring access to knowledge requires careful planning and adequate financial resources (for example, paying public contributors to engage in debate on critical ethical issues or paying for legal opinions on regulatory issues). Access to such resources can be planned for on a per-study basis and enabled through funding proposals. However, organizations regularly engaged in the development and evaluation of digital mental health interventions should consider creating or supporting structures such as advisory groups that can retain necessary competencies, such as in medical device regulation.

Evaluation of different landing pages on behavioural engagement with the CARA dashboard: A user research protocol

BACKGROUND

CARA set out to develop a data-visualisation platform to facilitate general practitioners to develop a deeper understanding of their patient population, disease management and prescribing through dashboards. To support the continued use and sustainability of the CARA dashboards, dashboard performance and user engagement have to be optimised. User research places people at the centre of the design process and aims to evaluate the needs, behaviours and attitudes of users to inform the design, development and impact of a product.

OBJECTIVE

To explore how different initial key messages impact the level of behavioural engagement with a CARA dashboard.

METHODS

Participating general practices can upload their practice data for analysis and visualisation in CARA dashboards. Practices will be randomised to one of three different initial landing pages: the full dashboard or one of two key messages: a between comparison (their practice prescribing with the average of all other practices) or within comparison (with practice data of the same month the previous year) with subsequent continuation to the full dashboard. Analysis will determine which of the three landing pages encourages user interaction, as measured by the number of 'clicks', 'viewings' and 'sessions'. Dashboard usage data will be collected through Google analytics.

DISCUSSION

This study will provide evidence of behavioural engagement and its metrics during the implementation of the CARA dashboards to optimise and sustain interaction.

TRIAL REGISTRATION

ISRCTN32783644 (Registration date: 02/01/2024).

Feasibility and preliminary efficacy of iConquerFear: a self-guided digital intervention for fear of cancer recurrence

PURPOSE

Approximately 50% of cancer survivors experience moderate-severe fear of cancer recurrence (FCR). Self-guided digital interventions have potential to address the high level of FCR-related unmet needs at scale, but existing digital interventions have demonstrated variable engagement and efficacy. This study aimed to evaluate the feasibility and preliminary efficacy of iConquerFear, a five-module self-guided digital FCR intervention.

METHODS

Eligible curatively treated breast cancer survivors were recruited. Participants reporting clinically significant FCR (≥ 13 on the Fear of Cancer Recurrence Inventory-Short Form; FCRI-SF) were given access to iConquerFear. Feasibility was indicated by > 50% of eligible participants enrolling in iConquerFear and recording moderate (≥ 120 min) or greater usage. Preliminary efficacy was evaluated via changes in self-reported FCR severity, anxiety, depression, intrusions and metacognitions from baseline to immediately and 3 months post-intervention.

RESULTS

Fifty-four (83%) of 65 eligible participants enrolled in iConquerFear; six subsequently withdrew. Thirty-nine (83%) participants recorded moderate (n = 24; 120-599 min) or high (n = 15; ≥ 600 min) usage. Engagement levels increased with participant age (p = 0.043), but were lower in participants with higher baseline FCR (p = 0.028). Qualitative feedback indicated engagement was sometimes limited by difficulties with navigation and relating to featured survivors. Participants reported significantly improved FCR (mean reduction (95%CI): baseline to post-intervention - 3.44 (- 5.18, - 1.71), baseline to 3-month follow-up - 4.52 (- 6.25, - 2.78), p =  < 0.001).

CONCLUSION

iConquerFear is a feasible and potentially efficacious intervention for reducing FCR in breast cancer survivors. Easier navigation and more relatable examples may enhance engagement.

IMPLICATIONS FOR CANCER SURVIVORS

iConquerFear may help address moderate but burdensome FCR levels in cancer survivors.

Effects of an online family-focused parenting support intervention on preterm infants' physical development and parents' sense of competence and care ability: A randomized controlled trial

BACKGROUND

Parenting of preterm infants presents unique challenges, particularly during their transition from hospital to home. Early discharge programs can assist parents during this critical period. However, the feasibility of delivering a family-focused online discharge program remains understudied.

OBJECTIVES

This study was conducted to investigate the impact of a family-focused online parenting support intervention on parents' sense of competence, depression, caregiving, social support, family functioning, and the weight and length of preterm infants.

METHODS

We conducted a single-blind, two-arm randomized controlled trial with a repeated-measures design. Participants were recruited from a specialized tertiary hospital in China between May and December 2022. Forty-five families were randomly assigned to the intervention group and 44 to the control group. The intervention included three components: (1) two one-hour group sessions and three 30-minute individual sessions of online parenting support; (2) two follow-up telephone calls (15-30 min each) post-discharge; and (3) access to online parenting resources. Primary outcomes included parenting sense of competence, parental care ability, and preterm infants' weight and length. Secondary outcomes included depression, social support, and family functioning. Measurements were taken at baseline, preterm infant discharge, one month post-discharge, and three months post-discharge. A generalized estimating equation model was employed based on the intention-to-treat principle for outcome comparison.

RESULTS

Parents in the intervention group showed significant improvements in parenting sense of competence (P < 0.001), parental care ability (P < 0.001), depression (P < 0.001), and social support (P = 0.002). However, no statistically significant differences were observed in preterm infants' weight and length, or in family functioning (P > 0.05).

CONCLUSIONS

Although the intervention did not affect preterm infants' outcomes, the positive enhancements in parenting sense of competence, care ability, depression, and social support indicate that online family-focused parenting support can effectively prepare parents for hospital discharge and the early transition period.

Real-World User Demographics of Three Web-Based Digital Mental Health Interventions Provided by the US Department of Veterans Affairs: Observational Study Using Web Analytics Data

BACKGROUND

Unguided digital mental health interventions (UDMHIs) have the potential to provide low-cost and effective mental health care at scale. Controlled trials have demonstrated the efficacy of UDMHIs to address mental health symptoms and conditions. However, few previous publications have described the demographics of real-world users of UDMHIs that are freely available to the public. The US Department of Veterans Affairs has created and hosts several UDMHIs on its Veteran Training Portal website. These web-based, free-to-use, and publicly available UDMHIs include Path to Better Sleep, Anger and Irritability Management Skills, and Moving Forward, which focus on insomnia, problematic anger, and depression symptoms, respectively.

OBJECTIVE

This study aimed to examine the user demographics of these 3 UDMHIs in the year 2021. In addition, it aimed to compare the age and gender distribution of the users of those 3 UDMHIs with one another and with the age and gender distribution of the total US veteran population.

METHODS

Google Analytics was used to collect user data for each of the 3 UDMHIs. The age and gender distribution of the users of each UDMHI was compared with that of the other UDMHIs as well as with that of the overall US veteran population using chi-square tests. Information on the total number of users, the country they were in, and the devices they used to access the UDMHIs was also collected and reported.

RESULTS

In 2021, the 3 UDMHIs together recorded 29,306 unique users. The estimated age range and gender were available for 24.12% (7068/29,306) of those users. Each UDMHI's age and gender distribution significantly differed from that of the other UDMHIs and from that of the overall US veteran population (P<.001 on all chi-square tests). Women and younger age groups were overrepresented among UDMHI users compared with the overall US veteran population. The majority of devices used to access the UDMHIs were desktop or laptop devices, although a substantial proportion of devices used were mobile devices (10,199/29,752, 34.28%). Most users (27,789/29,748, 93.41%) were located in the United States, with users from Canada, the United Kingdom, and Australia accounting for another 2.61% (775/29,748) of total users.

CONCLUSIONS

Our use of Google Analytics data provided useful information about the users of 3 free and publicly available UDMHIs provided by the US Department of Veterans Affairs. Although our findings should be considered in light of the limitations of autonomously collected web analytics data, they still offer useful information for health care policy makers, administrators, and UDMHI developers.

Web based research in sexual medicine: a position statement of the European Society for Sexual Medicine

Background

Although the use of the Web has brought major advances in every step of the research process, this also comes with several methodological challenges.

Aim

The article presents the European Society for Sexual Medicine's position statements on key methodological concerns relative to Web-based research in sexual medicine.

Methods

The authors conducted a systematic scoping review of articles using Web-based research methods in sexual medicine. For the creation of the statements, the authors processed the data from the methodology of the studies and formulated the final statements reaching 100% agreement in the group.

Outcomes

European Society for Sexual Medicine statements were provided on the following domains: definition of the population of interest, selection of the population of interest, data collection quality, response rate, self-reported questionnaire, consent, and legal obligations.

Results

Researchers should justify the relevance of the Internet population to the population of interest; should clearly describe how they identified study participants; should select and employ specific measures to avoid hoax; should explicitly describe the process of calculation of response and completion rates as well as the relative implications; should validate traditional sexual health questionnaires for online and, if possible, multilingual use; should not ignore consent in Web-based research; and need to be knowledgeable of the technical measures and legal obligations to protect anonymity.

Implications

Researchers are advised to include trained computer scientists in their group, have a good understanding of their legal obligations as to collecting, storing and disseminating personal data, and design their studies by taking into account the challenges of Web-based research.

Strengths and limitations

The heterogeneity of the included studies and methodological low quality of most of them was a limitation, which also shows the importance of this study and the need for guidelines regarding Web-based research.

Conclusion

Large uncontrolled samples could be a threat to the quality of the studies and increase bias if researchers are not mindful of the methodological challenges they would need to account for.

Internet-facilitated interventions for informal caregivers of patients with neurodegenerative disorders: Systematic review and meta-analysis

Objective

This systematic review explored the effectiveness of internet-delivered interventions in improving psychological outcomes of informal caregivers for neurodegenerative-disorder (ND) patients.

Methods

We searched seven databases for English-language papers published from 1999 to May 2021. Study-eligibility required that interventions used a minimum 50% internet-facilitation, targeting unpaid, adult informal caregivers of community-based ND-patients. We included randomised controlled trials (RCTs) and pre-post evaluative studies reporting outcomes for at least one-time point post-intervention. Independent quality checks on abstract and full-text screening were completed. Data extraction encompassed interventions' features, approaches, theoretical bases and delivery-modes. The Integrated quality Criteria for the Review Of Multiple Study designs (ICROMS) framework assessed risk of bias. Alongside narrative synthesis, we calculated meta-analyses on post-intervention using outcome measures from at least two RCTs to assess effectiveness.

Results

Searches yielded 51 eligible studies with 3180 participants. In 48 studies, caregivers supported a dementia-diagnosed individual. Intervention-durations encompassed four weeks to 12 months, with usage-frequency either prescribed or participant-determined. The most frequently-used approach was education, followed by social support. We calculated meta-analyses using data from 16 RCTs. Internet-delivered interventions were superior in improving mastery (g  =  1.17 [95% CI; 0.1 to 2.24], p  =  0.03) and reducing anxiety (g = -1.29 [95% CI; -1.56 to -1.01], p < 0.01), compared to all controls. Findings were equivocal for caregivers' quality of life, burden and other outcomes. High heterogeneity reflected the multifarious combinations of approaches and delivery-modes, precluding assessment of the most efficacious intervention features. Analyses using burden and self-efficacy outcomes' follow-up data were also non-significant compared to all comparator-types. Although 32 studies met the ICROMS threshold scores, we rated most studies' evidence quality as 'very-low'.

Conclusions

This review demonstrated some evidence for the efficacy of internet-delivered interventions targeting informal ND-caregivers. However, more rigorous studies, with longer follow-ups across outcomes and involving NDs other than dementia, are imperative to enhance the knowledge-base.

An Evaluation of 5-Year Web Analytics for HeadsUpGuys: A Men's Depression E-Mental Health Resource

The present study reports an evaluation of web analytics, over a 5-year period, for HeadsUpGuys.org, an eHealth resource for men with depression. Google Analytics, Search Console, and Tag Manager were used to monitor user activity over the course of the website's first 5 years (June 15, 2015-June 15, 2020). Through this period, HeadsUpGuys had a total of 1,665,356 unique users, amounting to 1,948,481 sessions and 3,328,258 page views. Organic traffic accounted for the highest proportion (53.44%; n = 1,041,277) of website sessions. Four of the top 10 Google search queries that brought users to the website related to suicidality. Three countries (United States, United Kingdom, Canada) accounted for almost three-quarters (71.10%; n = 1,385,485) of the site's traffic. Nearly three-quarters (73.35%; n = 1,429,285) of sessions occurred on a mobile device. The goal conversion rate for the Self Check was 60.27%. The average time on page was 2 min 53 s, with a bounce rate of 65.92%, and an exit rate of 57.20%. The goal conversion rate for the Stress Test was 52.89%. The average time on page was 4 min 8 s, with a bounce rate of 72.40% and an exit rate of 48.88%. The conversion rate for the final goal was 11.53%, indicating that approximately one in 10 visitors to the site had a session of at least 3 min. The findings illustrate the potential of eHealth resources to support men's mental health and provide a real-world benchmark to help advance the men's eHealth field.

Effectiveness of a kindergarten-based intervention to increase vegetable intake and reduce food neophobia amongst 1-year-old children: a cluster randomised controlled trial

Background

Children's first years of life are crucial to their future health. Studies show that a varied diet with a high intake of vegetables is positive in several domains of health. The present low vegetable intake amongst children is, therefore, a concern. Food neophobia is a common barrier to vegetable intake in children. As most Norwegian children attend kindergarten from an early age, kindergartens could contribute to the prevention of food neophobia and the promotion of vegetable intake.

Objective

The aim of this study was to assess the effect of a cluster randomised trial amongst 1-year-old children in kindergarten to reduce food neophobia and promote healthy eating.

Methods

Kindergartens were randomly allocated to either a control group or one of two intervention groups. Both intervention groups (diet and diet + Sapere-method) were served a warm lunch meal including three alternating intervention vegetables, whilst the intervention group 2 (diet + Sapere) in addition received tools for weekly sensory lessons. The intervention was digitally administered via information and recipes on a study website. The control group did not receive any information. Parents completed digitally distributed questionnaires addressing food neophobia and food habits at baseline and post-intervention.

Results

The parents of 144 1-year-old children in 46 kindergartens completed the questionnaires, which were included in the main analysis. The results suggested a higher intake of the intervention vegetables in group 2 (diet + Sapere) compared to the control group. The effect on total vegetable intake was inconclusive. No effect was observed on the level of food neophobia in either of the intervention group.

Conclusion

This digitally delivered dietary and sensory intervention promoted the intake of intervention-targeted vegetables with inconclusive effect on total vegetable intake due to large loss to follow-up. No effect on the level of food neophobia was detected.

Development, Implementation and First Evaluation of an Online Portal to Promote the Mental Health of University Students (me@JGU)

BACKGROUND

University students encounter various stressors such as exam preparation, workload and economic concerns. Having to deal with a multitude of stressors can lead to mental health problems and have a negative impact on academic outcomes in students attending university. This paper describes the development and usability evaluation of an open and easily accessible online portal (me@JGU) designed to help students build skills they need to cope with common stressors and manage their own mental health.

METHODS

We developed a website that addresses the most common stressors among university students and offers strategies for dealing with difficult situations. Initial evaluation results were collected using website statistics and a short anonymous survey regarding the attractiveness and usability of the website.

RESULTS

Over an eight-month period, there were 5739 visitors, a total of 16,495 page views and 3748 downloads. The survey results indicate that me@JGU covers relevant topics and that the students like the layout.

CONCLUSIONS

Online interventions promoting mental health of university populations are easily accessible and cost effective for large populations. They may prevent study difficulties, inform students about mental health and offer possible solutions. In addition, at-risk students can receive information about other relevant resources, and feel encouraged to access support and treatment.

Opportunities and challenges of delivering digital clinical trials: lessons learned from a randomised controlled trial of an online behavioural intervention for children and young people

BACKGROUND

Despite being the gold standard of research to determine effectiveness, randomised controlled trials (RCTs) often struggle with participant recruitment, engagement and retention. These issues may be exacerbated when recruiting vulnerable populations, such as participants with mental health issues. We aimed to update understanding of the scope of these problems in trials of health technology and identify possible solutions through reflecting on experiences from an exemplar trial (Online Remote Behavioural Intervention for Tics; ORBIT).

METHOD

We extracted anonymised data on recruitment, retention and requests for more funding and time from trials funded by the largest funder of health technology trials in the UK (the National Institute of Health Research Health Technology Assessment) between 2010 and 2020, and compared these with data from a recent, successful trial (ORBIT). ORBIT aimed to assess the clinical- and cost-effectiveness of blended online and human behavioural therapy for tics in young people. Many of the trial procedures, including recruitment, the intervention and data collection, were undertaken online.

RESULTS

Data were extracted on 51 trials conducted between 2010 and 2020. Sixty per cent of trials failed to reach their original recruitment target and only 44% achieved their follow-up in the specified time frame. In contrast, ORBIT recruited to target and achieved 90% follow-up. We posit that these achievements are related to (a) judicious use of digital technology for trial procedures and (b) adequate numbers of highly trained and motivated trial staff. We provide details of both these to help other research teams plan and cost for successful trials.

CONCLUSION

An approach combining human and online methods may be advantageous in facilitating trial delivery, particularly in paediatric mental health services. Given the importance of successful clinical trials in advancing healthcare delivery and the waste of human and economic resources associated with unsuccessfully delivered trials, it is imperative that trials are appropriately costed and future research focusses on improving trial design and delivery.

TRIAL REGISTRATION

The ORBIT trial is registered with ISRTCN ( ISRCTN70758207 ) Registered on March 20, 2018. and ClinicalTrials.gov ( NCT03483493 ). Registered on March 30, 2018.

Mobile Breast Cancer e-Support Program for Chinese Women With Breast Cancer Undergoing Chemotherapy (Part 3): Secondary Data Analysis

Background

Many app-based interventions targeting women with breast cancer have been developed and tested for effectiveness. However, information regarding the evaluation of the usage of these interventions is scarce. A better understanding of usage data is important to determine how women use apps and how these interventions affect health outcomes.

Objective

This study aimed to examine the usage duration and login frequency of an app-based intervention, the Breast Cancer e-Support (BCS) program, and to investigate the association between usage data and participants’ demographic and medical characteristics.

Methods

This study is a secondary data analysis of a randomized controlled trial assessing the effectiveness of the BCS program. The BCS program contains four modules: Learning Forum, Discussion Forum, Ask-the-Expert Forum, and Your Story Forum. A total of 57 women in the intervention group accessed the BCS program during their 12-week chemotherapy. The app’s background system tracked the usage duration and login frequency for each forum and the entire BCS program.

Results

The total usage duration per participant ranged from 0 to 9371 minutes, and the login frequency per participant ranged from 0 to 774 times. The Discussion Forum and the Learning Forum were the most frequently used modules. The general linear model showed that age, education, family monthly income, and employment were associated with BCS usage duration and/or login frequency. Age (F_1,45=10.09, P=.003, B=115.34, 95% CI 42.22-188.47) and education level (F_1,45=7.22, P=.01, B=1949.63, 95% CI 487.76-3411.50) were positively associated with the usage duration of the entire BCS program. Family monthly income was positively associated with the usage duration of the Learning Forum (F_1,45=11.85, P=.001, B=1488.55, 95% CI 617.58-2359.51) and the login frequency of the entire BCS program (F_1,45=4.47, P=.04, B=113.68, 95% CI 5.33-222.03). Employment was negatively associated with the usage duration of the Ask-the-expert Forum (F_1,45=4.50, P=.04, B=–971.87, 95% CI –1894.66 to –49.07) and the Your Story Forum (F_1,45=5.36, P=.03, B=–640.71, 95% CI –1198.30 to –83.11) and positively associated with the login frequency of the entire BCS program (F_1,45=10.86, P=.002, B=192.88, 95% CI 75.01-310.74). No statistical differences were found between BCS usage data and cancer stage, BMI, comorbidity, types of surgery, or cycles of chemotherapy.

Conclusions

Overall, this study found considerable variability in the usage of app-based interventions. When health care professionals incorporate app-based interventions into their routine care for women with breast cancer, the learning and discussion functions of apps should be strengthened to promote engagement. Additionally, characteristics of women with breast cancer, such as age, level of education, income, and employment status, should be taken in consideration to develop tailored apps that address their particular needs and therefore improve their engagement with the app.

Trial Registration

Australian New Zealand Clinical Trials Registry ACTRN12616000639426; http://www.ANZCTR.org.au/ACTRN12616000639426.aspx