Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach

Early Attrition Prediction for Web-Based Interpretation Bias Modification to Reduce Anxious Thinking: A Machine Learning Study

BACKGROUND

Digital mental health is a promising paradigm for individualized, patient-driven health care. For example, cognitive bias modification programs that target interpretation biases (cognitive bias modification for interpretation [CBM-I]) can provide practice thinking about ambiguous situations in less threatening ways on the web without requiring a therapist. However, digital mental health interventions, including CBM-I, are often plagued with lack of sustained engagement and high attrition rates. New attrition detection and mitigation strategies are needed to improve these interventions.

OBJECTIVE

This paper aims to identify participants at a high risk of dropout during the early stages of 3 web-based trials of multisession CBM-I and to investigate which self-reported and passively detected feature sets computed from the participants interacting with the intervention and assessments were most informative in making this prediction.

METHODS

The participants analyzed in this paper were community adults with traits such as anxiety or negative thinking about the future (Study 1: n=252, Study 2: n=326, Study 3: n=699) who had been assigned to CBM-I conditions in 3 efficacy-effectiveness trials on our team's public research website. To identify participants at a high risk of dropout, we created 4 unique feature sets: self-reported baseline user characteristics (eg, demographics), self-reported user context and reactions to the program (eg, state affect), self-reported user clinical functioning (eg, mental health symptoms), and passively detected user behavior on the website (eg, time spent on a web page of CBM-I training exercises, time of day during which the exercises were completed, latency of completing the assessments, and type of device used). Then, we investigated the feature sets as potential predictors of which participants were at high risk of not starting the second training session of a given program using well-known machine learning algorithms.

RESULTS

The extreme gradient boosting algorithm performed the best and identified participants at high risk with macro-F1-scores of .832 (Study 1 with 146 features), .770 (Study 2 with 87 features), and .917 (Study 3 with 127 features). Features involving passive detection of user behavior contributed the most to the prediction relative to other features. The mean Gini importance scores for the passive features were as follows: .033 (95% CI .019-.047) in Study 1; .029 (95% CI .023-.035) in Study 2; and .045 (95% CI .039-.051) in Study 3. However, using all features extracted from a given study led to the best predictive performance.

CONCLUSIONS

These results suggest that using passive indicators of user behavior, alongside self-reported measures, can improve the accuracy of prediction of participants at a high risk of dropout early during multisession CBM-I programs. Furthermore, our analyses highlight the challenge of generalizability in digital health intervention studies and the need for more personalized attrition prevention strategies.

Novel Multicomponent Digital Care Assistant and Support Program for People After Stroke or Transient Ischaemic Attack: A Pilot Feasibility Study

Evidence is increasing for digital health programs targeting the secondary prevention of stroke. We aimed to determine the feasibility of the novel Care Assistant and support Program for people after Stroke (CAPS) or transient ischaemic attack (TIA) by combining person-centred goal setting and risk-factor monitoring through a web-based clinician portal, SMS messages, a mobile application (app), and a wearable device. We conducted a 12-week mixed-methods, open-label feasibility study. Participants (6 months-3 years after stroke or TIA, access to the internet via a smartphone/tablet) were recruited via the Australian Stroke Clinical Registry. Participants set one or two secondary prevention goals with a researcher and provided access and training in technology use. Feasibility outcomes included recruitment, retention, usability, acceptability, and satisfaction. Secondary outcomes included goal attainment, health outcomes, and program costs. Following 600 invitations, 58 responded, 34/36 (94%) eligible participants commenced the program (one withdrawal; 97% retention), and 10 were interviewed. Participants (27% female, 33% TIA) generally rated the usability of the mobile application as 'Good' to 'Excellent' (System Usability Scale). Most (94%) agreed the program helped with engagement in health self-monitoring. Overall, 52 goals were set, predominantly regarding exercise (21/52), which were the most frequently achieved (9/21). At 12 weeks, participants reported significant improvements (p < 0.05) in self-efficacy (Cohen's d = 0.40), cardiovascular health (d = 0.71), and the mental health domain of the PROMIS GH (d = 0.63). CAPS was acceptable, with good retention and engagement of participants. Evaluation of this program in a randomised controlled trial is warranted.

A Technological Tool Aimed at Self-Care in Patients With Multimorbidity: Cross-Sectional Usability Study

BACKGROUND

Information and communication technologies (ICTs) have been positioned as useful tools to facilitate self-care. The interaction between a patient and technology, known as usability, is particularly important for achieving positive health outcomes. Specific characteristics of patients with chronic diseases, including multimorbidity, can affect their interaction with different technologies. Thus, studying the usability of ICTs in the field of multimorbidity has become a key element to ensure their relevant role in promoting self-care.

OBJECTIVE

The aim of this study was to analyze the usability of a technological tool dedicated to health and self-care in patients with multimorbidity in primary care.

METHODS

A descriptive observational cross-sectional usability study was performed framed in the clinical trial in the primary care health centers of Madrid Health Service of the TeNDER (Affective Based Integrated Care for Better Quality of Life) project. The TeNDER technological tool integrates sensors for monitoring physical and sleep activity along with a mobile app for consulting the data collected and working with self-management tools. This project included patients over 60 years of age who had one or more chronic diseases, at least one of which was mild-moderate cognitive impairment, Parkinson disease, or cardiovascular disease. From the 250 patients included in the project, 38 agreed to participate in the usability study. The usability variables investigated were effectiveness, which was determined by the degree of completion and the total number of errors per task; efficiency, evaluated as the average time to perform each task; and satisfaction, quantified by the System Usability Scale. Five tasks were evaluated based on real case scenarios. Usability variables were analyzed according to the sociodemographic and clinical characteristics of patients. A logistic regression model was constructed to estimate the factors associated with the type of support provided for task completion.

RESULTS

The median age of the 38 participants was 75 (IQR 72.0-79.0) years. There was a slight majority of women (20/38, 52.6%) and the participants had a median of 8 (IQR 7.0-11.0) chronic diseases. Thirty patients completed the usability study, with a usability effectiveness result of 89.3% (134/150 tasks completed). Among the 30 patients, 66.7% (n=20) completed all tasks and 56.7% (17/30) required personalized help on at least one task. In the multivariate analysis, educational level emerged as a facilitating factor for independent task completion (odds ratio 1.79, 95% CI 0.47-6.83). The median time to complete the total tasks was 296 seconds (IQR 210.0-397.0) and the median satisfaction score was 55 (IQR 45.0-62.5) out of 100.

CONCLUSIONS

Although usability effectiveness was high, the poor efficiency and usability satisfaction scores suggest that there are other factors that may interfere with the results. Multimorbidity was not confirmed to be a key factor affecting the usability of the technological tool.

TRIAL REGISTRATION

Clinicaltrials.gov NCT05681065; https://clinicaltrials.gov/study/NCT05681065.

Characterizing perceived usability and its correlation with smoking cessation: An analysis of user assessments of the smoking cessation app quitSTART

Background

As smartphone ownership has become common in all demographic groups in the United States, smartphone applications (apps) for smoking cessation have grown in popularity due to their potential for supporting the diverse populations in the United States who are attempting to quit smoking. Usability is commonly assessed in mobile health (mHealth) technology as an important aspect of the user experience that could influence users' adherence to a health app and health outcomes. However, the variation of perceived usability across demographic groups, and the implications of that variation for app success, have not been well studied.

Objective

The aims of this study were to characterize variation in the perceived usability of the National Cancer Institute Smokefree.gov Initiative smoking cessation app quitSTART across demographic groups, and to assess the correlation between perceived usability and short-term smoking cessation.

Methods

We conducted a secondary analysis of data from a randomized controlled trial conducted from 2020 to 2021, which used a 16-item modified version of the mHealth App Usability Questionnaire (MAUQ) to quantify perceived usability four weeks after app download among 131 smokers attempting to quit. Responses were coded on a 5-point Likert-type scale ranging from strongly disagree (1) to strongly agree (5) and total perceived usability was calculated as the sum of all 16 items (range: 16-80). Associations between participant demographic characteristics (gender, race, education level, age, etc.) and total usability were determined using an ANCOVA model. A multivariable logistic regression model was used to assess the association between usability and smoking cessation, also assessed 4 weeks after app download.

Results

The ANCOVA model demonstrated that race was associated with perceived usability, with participants from a racial minority group reporting higher total usability than White participants (p < 0.001). White participants had an adjusted mean total usability of 55.8 (95 % CI: 52.8, 58.8) while racial minority group participants had an adjusted mean total usability of 66.5 (95 % CI: 61.2, 71.6). Other participant demographic characteristics, such as gender and sexual minority status, were not associated with mean total usability. Total usability was positively associated with smoking cessation (OR: 1.04, 95 % CI: 1.00, 1.08, p = 0.031).

Conclusions

Total perceived usability of quitSTART was higher among adults from a racial minority group compared to White adults, and perceived usability was positively associated with cessation success. These findings emphasize the importance of ensuring high usability of mHealth smoking cessation apps for diverse populations.

Exploring the perspectives of primary care providers on use of the electronic Patient Reported Outcomes tool to support goal-oriented care: a qualitative study

BACKGROUND

Digital health technologies can support primary care delivery, but clinical uptake in primary care is limited. This study explores enablers and barriers experienced by primary care providers when adopting new digital health technologies, using the example of the electronic Patient Reported Outcome (ePRO) tool; a mobile application and web portal designed to support goal-oriented care. To better understand implementation drivers and barriers primary care providers' usage behaviours are compared to their perspectives on ePRO utility and fit to support care for patients with complex care needs.

METHODS

This qualitative sub-analysis was part of a larger trial evaluating the use of the ePRO tool in primary care. Qualitative interviews were conducted with providers at the midpoint (i.e. 4.5-6 months after ePRO implementation) and end-point (i.e. 9-12 months after ePRO implementation) of the trial. Interviews explored providers' experiences and perceptions of integrating the tool within their clinical practice. Interview data were analyzed using a hybrid thematic analysis and guided by the Technology Acceptance Model. Data from thirteen providers from three distinct primary care sites were included in the presented study.

RESULTS

Three core themes were identified: (1) Perceived usefulness: perceptions of the tool's alignment with providers' typical approach to care, impact and value and fit with existing workflows influenced providers' intention to use the tool and usage behaviour; (2) Behavioural intention: providers had a high or low behavioural intention, and for some, it changed over time; and (3) Improving usage behaviour: enabling external factors and enhancing the tool's perceived ease of use may improve usage behaviour.

CONCLUSIONS

Multiple refinements/iterations of the ePRO tool (e.g. enhancing the tool's alignment with provider workflows and functions) may be needed to enhance providers' usage behaviour, perceived usefulness and behavioural intention. Enabling external factors, such as organizational and IT support, are also necessary to increase providers' usage behaviour. Lessons from this study advance knowledge of technology implementation in primary care.

TRIAL REGISTRATION

Clinicaltrials.gov Identified NCT02917954. Registered September 2016, https://www.clinicaltrials.gov/ct2/show/study/NCT02917954.

Assessing the Implementation and Effectiveness of the Electronic Patient-Reported Outcome Tool for Older Adults With Complex Care Needs: Mixed Methods Study

Background

Goal-oriented care is being adopted to deliver person-centered primary care to older adults with multimorbidity and complex care needs. Although this model holds promise, its implementation remains a challenge. Digital health solutions may enable processes to improve adoption; however, they require evaluation to determine feasibility and impact.

Objective

This study aims to evaluate the implementation and effectiveness of the electronic Patient-Reported Outcome (ePRO) mobile app and portal system, designed to enable goal-oriented care delivery in interprofessional primary care practices. The research questions driving this study are as follows: Does ePRO improve quality of life and self-management in older adults with complex needs? What mechanisms are likely driving observed outcomes?

Methods

A multimethod, pragmatic randomized controlled trial using a stepped-wedge design and ethnographic case studies was conducted over a 15-month period in 6 comprehensive primary care practices across Ontario with a target enrollment of 176 patients. The 6 practices were randomized into either early (3-month control period; 12-month intervention) or late (6-month control period; 9-month intervention) groups. The primary outcome measure of interest was the Assessment of Quality of Life-4D (AQoL-4D). Data were collected at baseline and at 3 monthly intervals for the duration of the trial. Ethnographic data included observations and interviews with patients and providers at the midpoint and end of the intervention. Outcome data were analyzed using linear models conducted at the individual level, accounting for cluster effects at the practice level, and ethnographic data were analyzed using qualitative description and framework analysis methods.

Results

Recruitment challenges resulted in fewer sites and participants than expected; of the 176 target, only 142 (80.6%) patients were identified as eligible to participate because of lower-than-expected provider participation and fewer-than-expected patients willing to participate or perceived as ready to engage in goal-setting. Of the 142 patients approached, 45 (32%) participated. Patients set a variety of goals related to self-management, mental health, social health, and overall well-being. Owing to underpowering, the impact of ePRO on quality of life could not be definitively assessed; however, the intervention group, ePRO plus usual care (mean 15.28, SD 18.60) demonstrated a nonsignificant decrease in quality of life (t₂₄=−1.20; P=.24) when compared with usual care only (mean 21.76, SD 2.17). The ethnographic data reveal a complex implementation process in which the meaningfulness (or coherence) of the technology to individuals’ lives and work acted as a key driver of adoption and tool appraisal.

Conclusions

This trial experienced many unexpected and significant implementation challenges related to recruitment and engagement. Future studies could be improved through better alignment of the research methods and intervention to the complex and diverse clinical settings, dynamic goal-oriented care process, and readiness of provider and patient participants.

Trial Registration

ClinicalTrials.gov NCT02917954; https://clinicaltrials.gov/ct2/show/NCT02917954