Opening the Black Box of Electronic Health: Collecting, Analyzing, and Interpreting Log Data

A systematic review on eHealth technology personalization approaches

Despite the widespread use of personalization of eHealth technologies, there is a lack of comprehensive understanding regarding its application. This systematic review aims to bridge this gap by identifying and clustering different personalization approaches based on the type of variables used for user segmentation and the adaptations to the eHealth technology and examining the role of computational methods in the literature. From the 412 included reports, we identified 13 clusters of personalization approaches, such as behavior + channeling and environment + recommendations. Within these clusters, 10 computational methods were utilized to match segments with technology adaptations, such as classification-based methods and reinforcement learning. Several gaps were identified in the literature, such as the limited exploration of technology-related variables, the limited focus on user interaction reminders, and a frequent reliance on a single type of variable for personalization. Future research should explore leveraging technology-specific features to attain individualistic segmentation approaches.

Attrition of older adults in web-based health interventions: Survival analysis within an observational cohort study

To identify demographics and personal motivation types that predict dropping out of eHealth interventions among older adults. We conducted an observational cohort study. Participants completed a pre-test questionnaire and got access to an eHealth intervention, called Stranded, for 4 weeks. With survival and Cox-regression analyses, demographics and types of personal motivation were identified that affect drop-out. Ninety older adults started using Stranded. 45.6% participants continued their use for 4 weeks. 32.2% dropped out in the first week and 22.2% dropped out in the second or third week. The final multivariate Cox-regression model which predicts drop-out, consisted of the variables: perceived computer skills and level of external regulation. Predicting the chance of dropping out of an eHealth intervention is possible by using level of self-perceived computer skills and level of external regulation (externally controlled rewards or punishments direct behaviour). Anticipating to these factors can improve eHealth adoption.

Digital Smoking Cessation Intervention for Cancer Survivors: Analysis of Predictors and Moderators of Engagement and Outcome Alongside a Randomized Controlled Trial

BACKGROUND

Recent studies have shown positive, though small, clinical effects of digital smoking cessation (SC) interventions for cancer survivors. However, research on associations among participant characteristics, intervention engagement, and outcomes is limited.

OBJECTIVE

This study aimed to explore the predictors and moderators of engagement and outcome of MyCourse-Quit Smoking (in Dutch: "MijnKoers-Stoppen met Roken"), a digital minimally guided intervention for cancer survivors.

METHODS

A secondary analysis of data from the randomized controlled trial was performed. The number of cigarettes smoked in the past 7 days at 6-month follow-up was the primary outcome measure. We analyzed interactions among participant characteristics (11 variables), intervention engagement (3 variables), and outcome using robust linear (mixed) modeling.

RESULTS

In total, 165 participants were included in this study. Female participants accessed the intervention less often than male participants (B=-11.12; P=.004). A higher Alcohol Use Disorders Identification Test score at baseline was associated with a significantly higher number of logins (B=1.10; P<.001) and diary registrations (B=1.29; P<.001). A higher Fagerström Test for Nicotine Dependence score at baseline in the intervention group was associated with a significantly larger reduction in tobacco use after 6 months (B=-9.86; P=.002). No other associations and no moderating effects were found.

CONCLUSIONS

Overall, a limited number of associations was found between participant characteristics, engagement, and outcome, except for gender, problematic alcohol use, and nicotine dependence. Future studies are needed to shed light on how this knowledge can be used to improve the effects of digital SC programs for cancer survivors.

TRIAL REGISTRATION

Netherlands Trial register NTR6011/NL5434; https://onderzoekmetmensen.nl/nl/trial/22832.

A service-user digital intervention to collect real-time safety information on acute, adult mental health wards: the WardSonar mixed-methods study

Background

Acute inpatient mental health services report high levels of safety incidents. The application of patient safety theory has been sparse, particularly concerning interventions that proactively seek patient perspectives.

Objective(s)

Develop and evaluate a theoretically based, digital monitoring tool to collect real-time information from patients on acute adult mental health wards about their perceptions of ward safety.

Design

Theory-informed mixed-methods study. A prototype digital monitoring tool was developed from a co-design approach, implemented in hospital settings, and subjected to qualitative and quantitative evaluation.

Setting and methods

Phase 1: scoping review of the literature on patient involvement in safety interventions in acute mental health care; evidence scan of digital technology in mental health contexts; qualitative interviews with mental health patients and staff about perspectives on ward safety. This, alongside stakeholder engagement with advisory groups, service users and health professionals, informed the development processes. Most data collection was virtual. Phase 1 resulted in the technical development of a theoretically based digital monitoring tool that collected patient feedback for proactive safety monitoring. Phase 2: implementation of the tool in six adult acute mental health wards across two UK NHS trusts; evaluation via focused ethnography and qualitative interviews. Statistical analysis of WardSonar data and routine ward data involving construction of an hour-by-hour data set per ward, permitting detailed analysis of the use of the WardSonar tool.

Participants

A total of 8 patients and 13 mental health professionals participated in Phase 1 interviews; 33 staff and 34 patients participated in Phase 2 interviews.

Interventions

Patients could use a web application (the WardSonar tool) to record real-time perceptions of ward safety. Staff could access aggregated, anonymous data to inform timely interventions.

Results

Coronavirus disease 2019 restrictions greatly impacted the study. Stakeholder engagement permeated the project. Phase 1 delivered a theory-based, collaboratively designed digital tool for proactive patient safety monitoring. Phase 2 showed that the tool was user friendly and broadly acceptable to patients and staff. The aggregated safety data were infrequently used by staff. Feasibility depended on engaged staff and embedding use of the tool in ward routines. There is strong evidence that an incident leads to increased probability of further incidents within the next 4 hours. This puts a measure on the extent to which social/behavioural contagion persists. There is weak evidence to suggest that an incident leads to a greater use of the WardSonar tool in the following hour, but none to suggest that ward atmosphere predicts future incidents. Therefore, how often patients use the tool seems to send a stronger signal about potential incidents than patients' real-time reports about ward atmosphere.

Limitations

Implementation was limited to two NHS trusts. Coronavirus disease 2019 impacted design processes including stakeholder engagement; implementation; and evaluation of the monitoring tool in routine clinical practice. Higher uptake could enhance validity of the results.

Conclusions

WardSonar has the potential to provide a valuable route for patients to communicate safety concerns. The WardSonar monitoring tool has a strong patient perspective and uses proactive real-time safety monitoring rather than traditional retrospective data review.

Future work

The WardSonar tool can be refined and tested further in a post Coronavirus disease 2019 context.

Study registration

This study is registered as ISRCTN14470430.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: NIHR128070) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 14. See the NIHR Funding and Awards website for further award information.

Use and Engagement With Low-Intensity Cognitive Behavioral Therapy Techniques Used Within an App to Support Worry Management: Content Analysis of Log Data

BACKGROUND

Low-intensity cognitive behavioral therapy (LICBT) has been implemented by the Improving Access to Psychological Therapies services across England to manage excessive worry associated with generalized anxiety disorder and support emotional well-being. However, barriers to access limit scalability. A solution has been to incorporate LICBT techniques derived from an evidence-based protocol within the Iona Mind Well-being app for Worry management (IMWW) with support provided through an algorithmically driven conversational agent.

OBJECTIVE

This study aims to examine engagement with a mobile phone app to support worry management with specific attention directed toward interaction with specific LICBT techniques and examine the potential to reduce symptoms of anxiety.

METHODS

Log data were examined with respect to a sample of "engaged" users who had completed at least 1 lesson related to the Worry Time and Problem Solving in-app modules that represented the "minimum dose." Paired sample 2-tailed t tests were undertaken to examine the potential for IMWW to reduce worry and anxiety, with multivariate linear regressions examining the extent to which completion of each of the techniques led to reductions in worry and anxiety.

RESULTS

There was good engagement with the range of specific LICBT techniques included within IMWW. The vast majority of engaged users were able to interact with the cognitive behavioral therapy model and successfully record types of worry. When working through Problem Solving, the conversational agent was successfully used to support the user with lower levels of engagement. Several users engaged with Worry Time outside of the app. Forgetting to use the app was the most common reason for lack of engagement, with features of the app such as completion of routine outcome measures and weekly reflections having lower levels of engagement. Despite difficulties in the collection of end point data, there was a significant reduction in severity for both anxiety (t53=5.5; P<.001; 95% CI 2.4-5.2) and low mood (t53=2.3; P=.03; 95% CI 0.2-3.3). A statistically significant linear model was also fitted to the Generalized Anxiety Disorder-7 (F2,51=6.73; P<.001), while the model predicting changes in the Patient Health Questionnaire-8 did not reach significance (F2,51=2.33; P=.11). This indicates that the reduction in these measures was affected by in-app engagement with Worry Time and Problem Solving.

CONCLUSIONS

Engaged users were able to successfully interact with the LICBT-specific techniques informed by an evidence-based protocol although there were lower completion rates of routine outcome measures and weekly reflections. Successful interaction with the specific techniques potentially contributes to promising data, indicating that IMWW may be effective in the management of excessive worry. A relationship between dose and improvement justifies the use of log data to inform future developments. However, attention needs to be directed toward enhancing interaction with wider features of the app given that larger improvements were associated with greater engagement.

How Reflective Automated e-Coaching Can Help Employees Improve Their Capacity for Resilience: Mixed Methods Study

BACKGROUND

An eHealth tool that coaches employees through the process of reflection has the potential to support employees with moderate levels of stress to increase their capacity for resilience. Most eHealth tools that include self-tracking summarize the collected data for the users. However, users need to gain a deeper understanding of the data and decide upon the next step to take through self-reflection.

OBJECTIVE

In this study, we aimed to examine the perceived effectiveness of the guidance offered by an automated e-Coach during employees' self-reflection process in gaining insights into their situation and on their perceived stress and resilience capacities and the usefulness of the design elements of the e-Coach during this process.

METHODS

Of the 28 participants, 14 (50%) completed the 6-week BringBalance program that allowed participants to perform reflection via four phases: identification, strategy generation, experimentation, and evaluation. Data collection consisted of log data, ecological momentary assessment (EMA) questionnaires for reflection provided by the e-Coach, in-depth interviews, and a pre- and posttest survey (including the Brief Resilience Scale and the Perceived Stress Scale). The posttest survey also asked about the utility of the elements of the e-Coach for reflection. A mixed methods approach was followed.

RESULTS

Pre- and posttest scores on perceived stress and resilience were not much different among completers (no statistical test performed). The automated e-Coach did enable users to gain an understanding of factors that influenced their stress levels and capacity for resilience (identification phase) and to learn the principles of useful strategies to improve their capacity for resilience (strategy generation phase). Design elements of the e-Coach reduced the reflection process into smaller steps to re-evaluate situations and helped them to observe a trend (identification phase). However, users experienced difficulties integrating the chosen strategies into their daily life (experimentation phase). Moreover, the identified events related to stress and resilience were too specific through the guidance offered by the e-Coach (identification phase), and the events did not recur, which consequently left users unable to sufficiently practice (strategy generation phase), experiment (experimentation phase), and evaluate (evaluation phase) the techniques during meaningful events.

CONCLUSIONS

Participants were able to perform self-reflection under the guidance of the automated e-Coach, which often led toward gaining new insights. To improve the reflection process, more guidance should be offered by the e-Coach that would aid employees to identify events that recur in daily life. Future research could study the effects of the suggested improvements on the quality of reflection via an automated e-Coach.

Usage of digital, social and goal-setting functionalities to support health behavior change: A mixed methods study among disadvantaged women during or after pregnancy and their healthcare professionals

OBJECTIVE

We aimed to gain insight into how and to what extent social (i.e. private/group chat) and goal-setting (e.g. rewards) functionalities in digital interventions for health behavior change were used by clients and nurses in a preventive care program for disadvantaged women during or after pregnancy, and which factors influenced usage.

METHODS

We collected quantitative and qualitative data on usage of these functionalities in 'Kindle', a mHealth intervention to prepare for health behavior change.

RESULTS

We found that nurses (n = 5) and clients (n = 20) scarcely used both functionalities. They sent 862 messages in the social functionality whose security they appreciated, but habitually used WhatsApp likewise. Moreover, nurses were hesitant to let their clients interact in the group chat. Clients formulated 59 personal goals, which they found difficult to do. Nurses rewarded 846 points for clients' progress on goal attainment, but found it hard to determine how many points to reward. Clients and nurses indicated that the functionality made it more fun and easy to discuss clients' personal goals.

CONCLUSIONS

To conclude, digital, social and goal-setting functionalities were used to a limited extent by nurses and clients, and need optimization before implementation to support disadvantaged groups to change their health behavior.

Compas-Y: A mixed methods pilot evaluation of a mobile self-compassion training for people with newly diagnosed cancer

Objective

Compas-Y is a compassionate mind training app that was co-designed to be fully adapted to mobile technology and to people with newly diagnosed cancer. This study aimed to evaluate the use, appreciation and impact of the app.

Methods

Seventy-one people with cancer who created an app account were included (38% breast cancer, 72% diagnosed <4 months ago, 76% received chemotherapy). Participants had very high baseline scores of self-compassion. In a convergent mixed methods design, back-end log-data (n = 71), pre-post surveys (n = 34) and semi-structured interviews (n = 23) collected for >8 weeks and were concurrently analysed using joint displays.

Results

About half of the participants (45%) used 4 of the 6 modules. Compas-Y was highly appreciated, with all content considered relevant and a source of support. Experienced benefits related to improved mental health. Particularly, we found significant changes in anxiety, but not in depression or well-being. In the interviews, people reported experiencing more rest and more positive emotions due to using the app. Process benefits included significant reductions in self-criticism (inadequate self and self-blame), but not self-compassion. In the interviews, people reported improved self-compassion and less self-criticism, more self-awareness, recognition and support, and improved emotion regulation and coping. The surveys did not capture the full range of outcomes that participants reported in the interviews.

Conclusions

Compas-Y is a highly appreciated mobile intervention that supported users in aspects of their mental health. Findings are discussed in terms of reach and adherence, app functionalities, co-design and tailoring of cancer-related and compassion-based eHealth.

Predictors of adolescent engagement and outcomes - A cross-sectional study using the togetherall (formerly Big White Wall) digital mental health platform

BACKGROUND

Online mental health platforms can improve access to, and use of, mental health support for young people who may find it difficult to engage with face-to-face delivery.

OBJECTIVE

We modelled predictors of engagement and symptom change in adolescent users of the Togetherall (formerly "Big White Wall") anonymous digital mental health peer-support platform.

METHODS

We report a retrospective analysis of longitudinal user data from UK 16-18 year Togetherall users, referred from mental health services (N = 606). Baseline demographics were reported for participants who logged anxiety and depression measures. Number of log-ins, mean session duration, total usage time, number of guided support courses and self-help materials accessed were our usage metrics. Participant characteristics and symptoms were used to predict engagement. For n = 245 users with symptom measures at >1 timepoint we modelled the effect of predictors on symptom scores.

RESULTS

Mean logins was 5.11 and mean usage time was 64.22 mins. Participants with one log-in represented 33.5% of the sample. Total time accessing Togetherall predicated greater usage of self-help materials and courses. Females made greater use of materials and courses than males. In a subsample, higher baseline depression and anxiety, longer total usage time and mean session duration predicted final depression scores, whereas higher baseline depression and anxiety and greater accessed self-help materials predicted lower final anxiety scores.

LIMITATIONS

A naturalistic design was used and symptom modelling should be interpreted with caution.

CONCLUSIONS

Findings suggest adolescents can engage with the Togetherall platform. Baseline symptoms and characteristics can inform user engagement with digital platforms.

Clients and professionals elicit long-term care preferences by using 'What matters to me': A process evaluation in the Netherlands

BACKGROUND

'What matters to me' is a five-category preference elicitation tool to assist clients and professionals in choosing long-term care. This study aimed to evaluate the use of and experiences with this tool.

METHODS

A mixed-method process evaluation was applied. Participants were 71 clients or relatives, and 12 professionals. They were all involved in decision-making on long-term care. Data collection comprised online user activity logs (N = 71), questionnaires (N = 38) and interviews (N = 20). Descriptive statistics was used for quantitative data, and a thematic analysis for qualitative data.

RESULTS

Sixty-nine per cent of participants completed one or more categories in an average time of 6.9 (±0.03) minutes. The tool was rated 6.63 (±0.88) of 7 in the Post-Study System Usability Questionnaire (PSSUQ). Ninety-five per cent experienced the tool as useful in practice. Suggestions for improvement included a separate version for relatives and a non-digital version. Although professionals thought the potentially extended consultation time could be problematic, all participants would recommend the tool to others.

CONCLUSION

'What matters to me' seems useful to assist clients and professionals with preference elicitation in long-term care. Evaluation of the impact on consultations between clients and professionals by using 'What matters to me' is needed.

Combining Persuasive System Design Principles and Behavior Change Techniques in Digital Interventions Supporting Long-term Weight Loss Maintenance: Design and Development of eCHANGE

BACKGROUND

Long-term weight maintenance after weight loss is challenging, and innovative solutions are required. Digital technologies can support behavior change and, therefore, have the potential to be an effective tool for weight loss maintenance. However, to create meaningful and effective digital behavior change interventions that support end user values and needs, a combination of persuasive system design (PSD) principles and behavior change techniques (BCTs) might be needed.

OBJECTIVE

This study aimed to investigate how an evidence-informed digital behavior change intervention can be designed and developed by combining PSD principles and BCTs into design features to support end user values and needs for long-term weight loss maintenance.

METHODS

This study presents a concept for how PSD principles and BCTs can be translated into design features by combining design thinking and Agile methods to develop and deliver an evidence-informed digital behavior change intervention aimed at supporting weight maintenance. Overall, 45 stakeholders participated in the systematic and iterative development process comprising co-design workshops, prototyping, Agile development, and usability testing. This included prospective end users (n=17, 38%; ie, people with obesity who had lost ≥8% of their weight), health care providers (n=9, 20%), healthy volunteers (n=4, 9%), a service designer (n=1, 2%), and stakeholders from the multidisciplinary research and development team (n=14, 31%; ie, software developers; digital designers; and eHealth, behavior change, and obesity experts). Stakeholder input on how to operationalize the design features and optimize the technology was examined through formative evaluation and qualitative analyses using rapid and in-depth analysis approaches.

RESULTS

A total of 17 design features combining PSD principles and BCTs were identified as important to support end user values and needs based on stakeholder input during the design and development of eCHANGE, a digital intervention to support long-term weight loss maintenance. The design features were combined into 4 main intervention components: Week Plan, My Overview, Knowledge and Skills, and Virtual Coach and Smart Feedback System. To support a healthy lifestyle and continued behavior change to maintain weight, PSD principles such as tailoring, personalization, self-monitoring, reminders, rewards, rehearsal, praise, and suggestions were combined and implemented into the design features together with BCTs from the clusters of goals and planning, feedback and monitoring, social support, repetition and substitution, shaping knowledge, natural consequences, associations, antecedents, identity, and self-belief.

CONCLUSIONS

Combining and implementing PSD principles and BCTs in digital interventions aimed at supporting sustainable behavior change may contribute to the design of engaging and motivating interventions in line with end user values and needs. As such, the design and development of the eCHANGE intervention can provide valuable input for future design and tailoring of evidence-informed digital interventions, even beyond digital interventions in support of health behavior change and long-term weight loss maintenance.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04537988; https://clinicaltrials.gov/ct2/show/NCT04537988.

Methods for Human-Centered eHealth Development: Narrative Scoping Review

Background

Thorough holistic development of eHealth can contribute to a good fit among the technology, its users, and the context. However, despite the availability of frameworks, not much is known about specific research activities for different aims, phases, and settings. This results in researchers having to reinvent the wheel. Consequently, there is a need to synthesize existing knowledge on research activities for participatory eHealth development processes.

Objective

The 3 main goals of this review are to create an overview of the development strategies used in studies based on the CeHRes (Center for eHealth Research) Roadmap, create an overview of the goals for which these methods can be used, and provide insight into the lessons learned about these methods.

Methods

We included eHealth development studies that were based on the phases and/or principles of the CeHRes Roadmap. This framework was selected because of its focus on participatory, iterative eHealth design in context and to limit the scope of this review. Data were extracted about the type of strategy used, rationale for using the strategy, research questions, and reported information on lessons learned. The most frequently mentioned lessons learned were summarized using a narrative, inductive approach.

Results

In the included 160 papers, a distinction was made between overarching development methods (n=10) and products (n=7). Methods are used to gather new data, whereas products can be used to synthesize previously collected data and support the collection of new data. The identified methods were focus groups, interviews, questionnaires, usability tests, literature studies, desk research, log data analyses, card sorting, Delphi studies, and experience sampling. The identified products were prototypes, requirements, stakeholder maps, values, behavior change strategies, personas, and business models. Examples of how these methods and products were applied in the development process and information about lessons learned were provided.

Conclusions

This study shows that there is a plethora of methods and products that can be used at different points in the development process and in different settings. To do justice to the complexity of eHealth development, it seems that multiple strategies should be combined. In addition, we found no evidence for an optimal single step-by-step approach to develop eHealth. Rather, researchers need to select the most suitable research methods for their research objectives, the context in which data are collected, and the characteristics of the participants. This study serves as a first step toward creating a toolkit to support researchers in applying the CeHRes Roadmap to practice. In this way, they can shape the most suitable and efficient eHealth development process.

A Digital Intervention for Respiratory Tract Infections (Internet Dr): Process Evaluation to Understand How to Support Self-care for Minor Ailments

Background

Approximately 57 million physician appointments annually in the United Kingdom are for minor ailments. These illnesses could be self-cared for, which would potentially lower patients’ anxiety, increase their confidence, and be more convenient. In a randomized controlled trial of the Internet Dr digital intervention, patients with access to the intervention had fewer consultations for respiratory tract infections (RTIs). Having established intervention efficacy, further examination of trial data is required to understand how the intervention works.

Objective

This paper reports a process evaluation of Internet Dr usage by the intervention group. The evaluation aims to demonstrate how meaningful usage metrics (ie, interactions that are specific and relevant to the intervention) can be derived from the theoretical principles underlying the intervention, then applied to examine whether these interactions are effective in supporting self-care for RTIs, for whom, and at what time.

Methods

The Internet Dr trial recorded patients’ characteristics and usage data over 24 weeks. At follow-up, users reported whether their levels of enablement to cope with their illness changed over the trial period. The Medical Research Council process evaluation guidance and checklists from the framework for Analyzing and Measuring Usage and Engagement Data were applied to structure research questions examining associations between usage and enablement.

Results

Viewing pages containing advice on caring for RTIs were identified as a meaningful metric for measuring intervention usage. Almost half of the users (616/1491, 42.31%) viewed at least one advice page, with most people (478/616, 77.6%) accessing them when they initially enrolled in the study. Users who viewed an advice page reported increased enablement to cope with their illness as a result of having participated in the study compared with users who did not (mean 2.12, SD 2.92 vs mean 1.65, SD 3.10; mean difference 0.469, 95% CI 0.082-0.856). The target population was users who had visited their general practitioners for an RTI in the year before the trial, and analyses revealed that this group was more likely to access advice pages (odds ratio 1.35, 95% CI 1.159-1.571; P<.001).

Conclusions

The process evaluation identifies viewing advice pages as associated with increased enablement to self-care, even when accessed in the absence of a RTI, meaning that dissemination activities need not be restricted to targeting users who are ill. The intervention was effective at reaching the target population of users who had previously consulted their general practitioners. However, attrition before reaching advice pages was high, highlighting the necessity of prioritizing access during the design phase. These findings provide guidance on how the intervention may be improved and disseminated and have wider implications for minor ailment interventions.

Outpatient portal use in prenatal care: differential use by race, risk, and area social determinants of health

To report the relationship of outpatient portal (OPP) use with clinical risk, area social determinants of health (SDoH), and race/ethnicity among pregnant women. Regression models predicting overall and individual portal feature use (main effects and interactions) based on key variables were specified using log files and clinical data. Overall OPP use among non-Hispanic Black women or patients who lived in lower SDoH neighborhoods were significantly less. High-risk pregnancy patients were likely to use the OPP more than those with normal-risk pregnancy. We found similar associations with individual OPP features, like Visit (scheduling) and My Record (test results). We also found significant interactive associations between race/ethnicity, clinical risk, and SDoH. Non-Hispanic Black women and those living in lower SDoH areas used OPP less than non-Hispanic White women from similar or affluent areas. More research must be conducted to learn of OPP use implications for pregnant women with specific clinical diagnoses.

Predicting Success of a Digital Self-Help Intervention for Alcohol and Substance Use With Machine Learning

Background

Digital self-help interventions for reducing the use of alcohol tobacco and other drugs (ATOD) have generally shown positive but small effects in controlling substance use and improving the quality of life of participants. Nonetheless, low adherence rates remain a major drawback of these digital interventions, with mixed results in (prolonged) participation and outcome. To prevent non-adherence, we developed models to predict success in the early stages of an ATOD digital self-help intervention and explore the predictors associated with participant's goal achievement.

Methods

We included previous and current participants from a widely used, evidence-based ATOD intervention from the Netherlands (Jellinek Digital Self-help). Participants were considered successful if they completed all intervention modules and reached their substance use goals (i.e., stop/reduce). Early dropout was defined as finishing only the first module. During model development, participants were split per substance (alcohol, tobacco, cannabis) and features were computed based on the log data of the first 3 days of intervention participation. Machine learning models were trained, validated and tested using a nested k-fold cross-validation strategy.

Results

From the 32,398 participants enrolled in the study, 80% of participants did not complete the first module of the intervention and were excluded from further analysis. From the remaining participants, the percentage of success for each substance was 30% for alcohol, 22% for cannabis and 24% for tobacco. The area under the Receiver Operating Characteristic curve was the highest for the Random Forest model trained on data from the alcohol and tobacco programs (0.71 95%CI 0.69-0.73) and (0.71 95%CI 0.67-0.76), respectively, followed by cannabis (0.67 95%CI 0.59-0.75). Quitting substance use instead of moderation as an intervention goal, initial daily consumption, no substance use on the weekends as a target goal and intervention engagement were strong predictors of success.

Discussion

Using log data from the first 3 days of intervention use, machine learning models showed positive results in identifying successful participants. Our results suggest the models were especially able to identify participants at risk of early dropout. Multiple variables were found to have high predictive value, which can be used to further improve the intervention.

Reasons for not reaching or using web-based self-management applications, and the use and evaluation of Oncokompas among cancer survivors, in the context of a randomised controlled trial

INTRODUCTION

The web-based self-management application Oncokompas was developed to support cancer survivors to monitor health-related quality of life and symptoms (Measure) and to provide tailored information (Learn) and supportive care options (Act). In a previously reported randomised controlled trial (RCT), 68% of 655 recruited survivors were eligible, and of those 45% participated in the RCT. Among participants of the RCT that were randomised to the intervention group, 52% used Oncokompas as intended. The aim of this study was to explore reasons for not participating in the RCT, and reasons for not using Oncokompas among non-users, and the use and evaluation of Oncokompas among users.

METHODS

Reasons for not participating were assessed with a study-specific questionnaire among 243 survivors who declined participation. Usage was investigated among 320 participants randomised to the intervention group of the RCT via system data and a study-specific questionnaire that was assessed during the 1 week follow-up (T1) assessment.

RESULTS

Main reasons for not participating were not interested in participation in scientific research (40%) and not interested in scientific research and Oncokompas (28%). Main reasons for not being interested in Oncokompas were wanting to leave the period of being ill behind (29%), no symptom burden (23%), or lacking internet skills (18%). Out of the 320 participants in the intervention group 167 (52%) used Oncokompas as intended. Among 72 non-users, main reasons for not using Oncokompas were no symptom burden (32%) or lack of time (26%). Among 248 survivors that activated their account, satisfaction and user-friendliness were rated with a 7 (scale 0-10). Within 3 (IQR 1-4) sessions, users selected 32 (IQR 6-37) topics. Main reasons for not using healthcare options in Act were that the information in Learn was already sufficient (44%) or no supportive care needs (32%).

DISCUSSION

Main reasons for not reaching or using Oncokompas were no symptom burden, no supportive care needs, or lack of time. Users selected many cancer-generic and tumour-specific topics to address, indicating added value of the wide range of available topics.

The formation of patient trust and its transference to online health services: the case of a Dutch online patient portal for rehabilitation care

Background

Trust is widely recognized as a crucial factor in successful physician–patient communication and patient engagement in treatment. However, with the rise of eHealth technologies, such as online patient portals, the role of trust and the factors that influence it need to be reconsidered. In this study, we aim to identify the factors that contribute to trust in an eHealth service and we aim to identify the consequences of trust in an eHealth service in terms of use.

Methods

The Patient Trust Assessment Tool was provided to new outpatients of a rehabilitation center in the Netherlands, that were expected to use the center’s online patient portal. Via this tool, we assessed five trust-related factors. This data was supplemented by questions about demographics (age, gender, rehabilitation treatment) and data about use (number of sessions, total time spent in sessions), derived from data logs. Data was analyzed via Partial Least Squares Structural Equation Modelling.

Results

In total, 93 patients participated in the study. Out of these participants, 61 used the portal at least once. The measurement model was considered good. Trust in the organization was found to affect trust in the care team (β = .63), trust in the care team affected trust in the treatment (β = .60). Both, trust in the care team and trust in the treatment influenced trust in the technology (β = .42 and .30, respectively). Trust in the technology affected the holistic concept trust in the service (β = .78). This holistic trust in the service finally, did not affect use.

Conclusions

This study shows that the formation of this trust is not unidimensional, but consists of different, separate factors (trust in the care organization, trust in the care team and trust in the treatment). Trust transfer does take place from offline to online health services. However, trust in the service does not directly affect the use of the eHealth technology.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-021-01552-4.

Commencement of and Retention in Web-Based Interventions and Response to Prompts and Reminders: Longitudinal Observational Study Based on Two Randomized Controlled Trials

BACKGROUND

Web-based interventions are effective for several psychological problems. However, recruitment, adherence, and missing data are challenges when evaluating these interventions.

OBJECTIVE

This study aimed to describe the use patterns during the commencement phase, possible retention patterns (continuation of data provision), and responses to prompts and reminders among participants in 2 randomized controlled trials (RCTs) evaluating web-based interventions.

METHODS

Data on use patterns logged in 2 RCTs aiming to reduce symptoms of anxiety and depression among adult patients recently diagnosed with cancer (AdultCan RCT) and patients with a recent myocardial infarction (Heart RCT) were analyzed. The web-based intervention in the AdultCan trial consisted of unguided self-help and psychoeducation and that in the Heart trial consisted of therapist-supported cognitive behavioral therapy. In total, 2360 participants' use patterns at first log-in, including data collection at baseline (ie, commencement) and at 2 follow-ups, were analyzed. Both the intervention and comparison groups were analyzed.

RESULTS

At commencement, 70.85% (909/1283) and 86.82% (935/1077) of the participants in AdultCan and Heart RCTs, respectively, logged in and completed baseline data collection after receiving a welcome email with log-in credentials. The median duration of the first log-in was 44 minutes and 38 minutes in AdultCan and Heart RCTs, respectively. Slightly less than half of the participants' first log-ins were completed outside standard office hours. More than 80% (92/114 and 103/111) of the participants in both trials explored the intervention within 2 weeks of being randomized to the treatment group, with a median duration of 7 minutes and 47 minutes in AdultCan and Heart RCTs, respectively. There was a significant association between intervention exploration time during the first 2 weeks and retention in the Heart trial but not in the AdultCan trial. However, the control group was most likely to retain and provide complete follow-up data. Across the 3 time points of data collection explored in this study, the proportion of participants responding to all questionnaires within 1 week from the prompt, without a reminder, varied between 35.45% (413/1165) and 66.3% (112/169). After 2 reminders, up to 97.6% (165/169) of the participants responded.

CONCLUSIONS

Most participants in both RCTs completed the baseline questionnaires within 1 week of receiving the welcome email. Approximately half of them answered questions at baseline data collection outside office hours, suggesting that the time flexibility inherent in web-based interventions contributes to commencement and use. In contrast to what was expected, the intervention groups generally had lower completion rates than the comparison groups. About half of the participants completed the questionnaires without a reminder, but thereafter, reminders contributed to both baseline and follow-up retention, suggesting they were effective. Strategies to increase commencement of and retention in eHealth interventions are important for the future development of effective interventions and relevant research.

e-Mental Health Program Usage Patterns in Randomized Controlled Trials and in the General Public to Inform External Validity Considerations: Sample Groupings Using Cluster Analyses

Background

Randomized controlled trials (RCTs) with vigorous study designs are vital for determining the efficacy of treatments. Despite the high internal validity attributed to RCTs, external validity concerns limit the generalizability of results to the general population. Bias can be introduced, for example, when study participants who self-select into a trial are more motivated to comply with study conditions than are other individuals. These external validity considerations extend to e-mental health (eMH) research, especially when eMH tools are designed for public access and provide minimal or no supervision.

Objective

Clustering techniques were employed to identify engagement profiles of RCT participants and community users of a self-guided eMH program. This exploratory approach inspected actual, not theorized, RCT participant and community user engagement patterns. Both samples had access to the eMH program over the same time period and received identical usage recommendations on the eMH program website. The aim of this study is to help gauge expectations of similarities and differences in usage behaviors of an eMH tool across evaluation and naturalistic contexts.

Methods

Australian adults signed up to myCompass, a self-guided online treatment program created to reduce mild to moderate symptoms of negative emotions. They did so either by being part of an RCT onboarding (160/231, 69.6% female) or by accessing the program freely on the internet (5563/8391, 66.30% female) between October 2011 and October 2012. During registration, RCT participants and community users provided basic demographic information. Usage metrics (number of logins, trackings, and learning activities) were recorded by the system.

Results

Samples at sign-up differed significantly in age (P=.003), with community users being on average 3 years older (mean 41.78, SD 13.64) than RCT participants (mean 38.79, SD 10.73). Furthermore, frequency of program use was higher for RCT participants on all usage metrics compared to community users through the first 49 days after registration (all P values <.001). Two-step cluster analyses revealed 3 user groups in the RCT sample (Nonstarters, 10-Timers, and 30+-Timers) and 2 user groups in the community samples (2-Timers and 20-Timers). Groups seemed comparable in patterns of use but differed in magnitude, with RCT participant usage groups showing more frequent engagement than community usage groups. Only the high-usage group among RCT participants approached myCompass usage recommendations.

Conclusions

Findings suggested that external validity concerns of RCT designs may arise with regards to the predicted magnitude of eMH program use rather than overall usage styles. Following up RCT nonstarters may help provide unique insights into why individuals choose not to engage with an eMH program despite generally being willing to participate in an eMH evaluation study. Overestimating frequency of engagement with eMH tools may have theoretical implications and potentially impact economic considerations for plans to disseminate these tools to the general public.

Engagement With a Behavior Change App for Alcohol Reduction: Data Visualization for Longitudinal Observational Study

Background

Behavior change apps can develop iteratively, where the app evolves into a complex, dynamic, or personalized intervention through cycles of research, development, and implementation. Understanding how existing users engage with an app (eg, frequency, amount, depth, and duration of use) can help guide further incremental improvements. We aim to explore how simple visualizations can provide a good understanding of temporal patterns of engagement, as usage data are often longitudinal and rich.

Objective

This study aims to visualize behavioral engagement with Drink Less, a behavior change app to help reduce hazardous and harmful alcohol consumption in the general adult population of the United Kingdom.

Methods

We explored behavioral engagement among 19,233 existing users of Drink Less. Users were included in the sample if they were from the United Kingdom; were 18 years or older; were interested in reducing their alcohol consumption; had a baseline Alcohol Use Disorders Identification Test score of 8 or above, indicative of excessive drinking; and had downloaded the app between May 17, 2017, and January 22, 2019 (615 days). Measures of when sessions begin, length of sessions, time to disengagement, and patterns of use were visualized with heat maps, timeline plots, k-modes clustering analyses, and Kaplan-Meier plots.

Results

The daily 11 AM notification is strongly associated with a change in engagement in the following hour; reduction in behavioral engagement over time, with 50.00% (9617/19,233) of users disengaging (defined as no use for 7 or more consecutive days) 22 days after download; identification of 3 distinct trajectories of use, namely engagers (4651/19,233, 24.18% of users), slow disengagers (3679/19,233, 19.13% of users), and fast disengagers (10,903/19,233, 56.68% of users); and limited depth of engagement with 85.076% (7,095,348/8,340,005) of screen views occurring within the Self-monitoring and Feedback module. In addition, a peak of both frequency and amount of time spent per session was observed in the evenings.

Conclusions

Visualizations play an important role in understanding engagement with behavior change apps. Here, we discuss how simple visualizations helped identify important patterns of engagement with Drink Less. Our visualizations of behavioral engagement suggest that the daily notification substantially impacts engagement. Furthermore, the visualizations suggest that a fixed notification policy can be effective for maintaining engagement for some users but ineffective for others. We conclude that optimizing the notification policy to target both effectiveness and engagement is a worthwhile investment. Our future goal is to both understand the causal effect of the notification on engagement and further optimize the notification policy within Drink Less by tailoring to contextual circumstances of individuals over time. Such tailoring will be informed from the findings of our micro-randomized trial (MRT), and these visualizations were useful in both gaining a better understanding of engagement and designing the MRT.

Identifying Persuasive Design Principles and Behavior Change Techniques Supporting End User Values and Needs in eHealth Interventions for Long-Term Weight Loss Maintenance: Qualitative Study

BACKGROUND

An increasing number of eHealth interventions aim to support healthy behaviors that facilitate weight loss. However, there is limited evidence of the effectiveness of the interventions and little focus on weight loss maintenance. Knowledge about end user values and needs is essential to create meaningful and effective eHealth interventions, and to identify persuasive system design (PSD) principles and behavior change techniques (BCTs) that may contribute to the behavior change required for successful long-term weight loss maintenance.

OBJECTIVE

This study aimed to provide insight into the design of eHealth interventions supporting behavior change for long-term weight maintenance. The study sought to identify the values and needs of people with obesity aiming to maintain weight after weight loss, and to identify PSD principles, BCTs, and design requirements that potentially enable an eHealth intervention to meet end user values and needs.

METHODS

This study presents the concept of integrating PSD principles and BCTs into the design process of eHealth interventions to meet user values and needs. In this study, individual interviews and focus groups were conducted with people with obesity (n=23) and other key stakeholders (n=27) to explore end user values and needs related to weight loss maintenance. Design thinking methods were applied during the focus group sessions to identify design elements and to explore how eHealth solutions can support the needs to achieve sustainable weight loss maintenance. The PSD model and behavior change taxonomy by Michie were used to identify PSD principles and BCT clusters to meet end user values and needs.

RESULTS

A total of 8 key end user values were identified, reflecting user needs for weight loss maintenance support: self-management, personalized care, autonomy, feel supported, positive self-image, motivation, happiness, and health. Goals and planning, feedback and monitoring, repetition and substitution, shaping knowledge, social support, identity, and self-belief were some of the BCT clusters identified to address these concepts, together with PSD principles such as personalization, tailoring, self-monitoring, praise, and suggestions.

CONCLUSIONS

The process of translating end user values and needs into design elements or features of eHealth technologies is an important part of the design process. To our knowledge, this is the first study to explore how PSD principles and BCTs can be integrated when designing eHealth self-management interventions for long-term weight loss maintenance. End users and other key stakeholders highlighted important factors to be considered in the design of eHealth interventions supporting sustained behavior change. The PSD principles and BCTs identified provide insights and suggestions about design elements and features to include for supporting weight loss maintenance. The findings indicate that a combination of BCTs and PSD principles may be needed in evidence-based eHealth interventions to stimulate motivation and adherence to support healthy behaviors and sustained weight loss maintenance.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04537988; https://clinicaltrials.gov/ct2/show/NCT04537988.

An Evaluation of Biometric Monitoring Technologies for Vital Signs in the Era of COVID-19

The novel coronavirus disease 2019 (COVID-19) global pandemic has shifted how many patients receive outpatient care. Telehealth and remote monitoring have become more prevalent, and measurements taken in a patient's home using biometric monitoring technologies (BioMeTs) offer convenient opportunities to collect vital sign data. Healthcare providers may lack prior experience using BioMeTs in remote patient care, and, therefore, may be unfamiliar with the many versions of BioMeTs, novel data collection protocols, and context of the values collected. To make informed patient care decisions based on the biometric data collected remotely, it is important to understand the engineering solutions embedded in the products, data collection protocols, form factors (physical size and shape), data quality considerations, and availability of validation information. This article provides an overview of BioMeTs available for collecting vital signs (temperature, heart rate, blood pressure, oxygen saturation, and respiratory rate) and discusses the strengths and limitations of continuous monitoring. We provide considerations for remote data collection and sources of validation information to guide BioMeT use in the era of COVID-19 and beyond.

Acceptance and Potential Impact of the eWALL Platform for Health Monitoring and Promotion in Persons with a Chronic Disease or Age-Related Impairment

Pervasive health technologies can increase the effectiveness of personal health monitoring and training, but more user studies are necessary to understand the interest for these technologies, and how they should be designed and implemented. In the present study, we evaluated eWALL, a user-centered pervasive health technology consisting of a platform that monitors users’ physical and cognitive behavior, providing feedback and motivation via an easy-to-use, touch-based user interface. The eWALL was placed for one month in the home of 48 subjects with a chronic condition (chronic obstructive pulmonary disease—COPD or mild cognitive impairment—MCI) or with an age-related impairment. User acceptance, platform use, and potential clinical effects were evaluated using surveys, data logs, and clinical scales. Although some features of the platform need to be improved before reaching technical maturity and making a difference in patients’ lives, the real-life evaluation of eWALL has shown how some features may influence patients’ intention to use this promising technology. Furthermore, this study made it clear how the free use of different health apps is modulated by the real needs of the patient and by their usefulness in the context of the patient’s clinical status.

Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study

BACKGROUND

Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase.

OBJECTIVE

The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase.

METHODS

Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an "eHealth evaluation cycle" and subsequently compose the online "eHealth methodology guide."

RESULTS

The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an "eHealth evaluation cycle" was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the "eHealth methodology guide" was composed by assigning the 75 evaluation approaches to the specific study phases of the "eHealth evaluation cycle."

CONCLUSIONS

Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online "eHealth methodology guide." By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the "eHealth evaluation cycle," the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study

BACKGROUND

Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase.

OBJECTIVE

The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase.

METHODS

Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an "eHealth evaluation cycle" and subsequently compose the online "eHealth methodology guide."

RESULTS

The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an "eHealth evaluation cycle" was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the "eHealth methodology guide" was composed by assigning the 75 evaluation approaches to the specific study phases of the "eHealth evaluation cycle."

CONCLUSIONS

Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online "eHealth methodology guide." By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the "eHealth evaluation cycle," the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Assistive technology designed to support self-management of people with dementia: user involvement, dissemination, and adoption. A scoping review

BACKGROUND

Assistive technology is advocated as a key solution to the need for support among people living with dementia. There is growing awareness of the benefits of user involvement in the design and test of these technologies and the need to identifying applicable and effective methods for implementation. The aim of this review was to explore and synthesize research addressing assistive technology designed to be used by people with dementia for self-management. Further research aims were to explore if and how user involvement, dissemination, and adoption of assistive technology were addressed.

METHOD

Electronic databases were searched using specified search terms. Key publications and grey literature sources were hand-searched. Materials published until year end 2018 were included. The results were summarized according to the research aims.

RESULTS

Eleven papers derived from eight studies were included. The studies presented data from prototype design and testing, and the review showed great variation in study scope, design, and methodology. User involvement varied from extensive involvement to no user involvement. Methods for adoption also varied widely and only targeted prototype testing. None of the studies addressed dissemination.

CONCLUSION

The results of this review underline the need for well-designed high-quality research into all the aspects that are essential to deliver applicable, effective, and sustainable assistive technology to support self-management of people with dementia. There is a need for evidence-based methods to promote and qualify user involvement, dissemination, and adoption. The results also point to the need for standardized outcome measures and standards for conducting and reporting research to improve its quality and impact.

Metrics for Outpatient Portal Use Based on Log File Analysis: Algorithm Development

BACKGROUND

Web-based outpatient portals help patients engage in the management of their health by allowing them to access their medical information, schedule appointments, track their medications, and communicate with their physicians and care team members. Initial studies have shown that portal adoption positively affects health outcomes; however, early studies typically relied on survey data. Using data from health portal applications, we conducted systematic assessments of patients' use of an outpatient portal to examine how patients engage with the tool.

OBJECTIVE

This study aimed to document the functionality of an outpatient portal in the context of outpatient care by mining portal usage data and to provide insights into how patients use this tool.

METHODS

Using audit log files from the outpatient portal associated with the electronic health record system implemented at a large multihospital academic medical center, we investigated the behavioral traces of a study population of 2607 patients who used the portal between July 2015 and February 2019. Patient portal use was defined as having an active account and having accessed any portal function more than once during the study time frame.

RESULTS

Through our analysis of audit log file data of the number and type of user interactions, we developed a taxonomy of functions and actions and computed analytic metrics, including frequency and comprehensiveness of use. We additionally documented the computational steps required to diagnose artifactual data and arrive at valid usage metrics. Of the 2607 patients in our sample, 2511 were active users of the patients portal where the median number of sessions was 94 (IQR 207). Function use was comprehensive at the patient level, while each session was instead limited to the use of one specific function. Only 17.45% (78,787/451,762) of the sessions were linked to activities involving more than one portal function.

CONCLUSIONS

In discussing the full methodological choices made in our analysis, we hope to promote the replicability of our study at other institutions and contribute to the establishment of best practices that can facilitate the adoption of behavioral metrics that enable the measurement of patient engagement based on the outpatient portal use.

Integrating People, Context, and Technology in the Implementation of a Web-Based Intervention in Forensic Mental Health Care: Mixed-Methods Study

Background

While eMental health interventions can have many potential benefits for mental health care, implementation outcomes are often disappointing. In order to improve these outcomes, there is a need for a better understanding of complex, dynamic interactions between a broad range of implementation-related factors. These interactions and processes should be studied holistically, paying attention to factors related to context, technology, and people.

Objective

The main objective of this mixed-method study was to holistically evaluate the implementation strategies and outcomes of an eMental health intervention in an organization for forensic mental health care.

Methods

First, desk research was performed on 18 documents on the implementation process. Second, the intervention’s use by 721 patients and 172 therapists was analyzed via log data. Third, semistructured interviews were conducted with all 18 therapists of one outpatient clinic to identify broad factors that influence implementation outcomes. The interviews were analyzed via a combination of deductive analysis using the nonadoption, abandonment, scale-up, spread, and sustainability framework and inductive, open coding.

Results

The timeline generated via desk research showed that implementation strategies focused on technical skills training of therapists. Log data analyses demonstrated that 1019 modules were started, and 18.65% (721/3865) of patients of the forensic hospital started at least one module. Of these patients, 18.0% (130/721) completed at least one module. Of the therapists using the module, 54.1% (93/172 sent at least one feedback message to a patient. The median number of feedback messages sent per therapist was 1, with a minimum of 0 and a maximum of 460. Interviews showed that therapists did not always introduce the intervention to patients and using the intervention was not part of their daily routine. Also, therapists indicated patients often did not have the required conscientiousness and literacy levels. Furthermore, they had mixed opinions about the design of the intervention. Important organization-related factors were the need for more support and better integration in organizational structures. Finally, therapists stated that despite its current low use, the intervention had the potential to improve the quality of treatment.

Conclusions

Synthesis of different types of data showed that implementation outcomes were mostly disappointing. Implementation strategies focused on technical training of therapists, while little attention was paid to changes in the organization, design of the technology, and patient awareness. A more holistic approach toward implementation strategies—with more attention to the organization, patients, technology, and training therapists—might have resulted in better implementation outcomes. Overall, adaptivity appears to be an important concept in eHealth implementation: a technology should be easily adaptable to an individual patient, therapists should be trained to deal flexibly with an eMental health intervention in their treatment, and organizations should adapt their implementation strategies and structures to embed a new eHealth intervention.

Frequent Mobile Electronic Medical Records Users Respond More Quickly to Emergency Department Consultation Requests: Retrospective Quantitative Study

Background

Specialty consultation is a critical aspect of emergency department (ED) practice, and a delay in providing consultation might have a significant clinical effect and worsen ED overcrowding. Although mobile electronic medical records (EMR) are being increasingly used and are known to improve the workflow of health care providers, limited studies have evaluated their effectiveness in real-life clinical scenarios.

Objective

For this study, we aimed to determine the association between response duration to an ED specialty consultation request and the frequency of mobile EMR use.

Methods

This retrospective study was conducted in an academic ED in Seoul, South Korea. We analyzed EMR and mobile EMR data from May 2018 to December 2018. Timestamps of ED consultation requests were retrieved from a PC-based EMR, and the response interval was calculated. Doctors’ log frequencies were obtained from the mobile EMR, and we merged data using doctors’ deidentification numbers. Pearson’s product-moment correlation was performed to identify this association. The primary outcome was the relationship between the frequency of mobile EMR usage and the time interval from ED request to consultation completion by specialty doctors. The secondary outcome was the relationship between the frequency of specialty doctors’ mobile EMR usage and the response time to consultation requests.

Results

A total of 25,454 consultations requests were made for 15,555 patients, and 252 specialty doctors provided ED specialty consultations. Of the 742 doctors who used the mobile EMR, 208 doctors used it for the specialty consultation process. After excluding the cases lacking essential information, 21,885 consultations with 208 doctors were included for analysis. According to the mobile EMR usage pattern, the average usage frequency of all users was 13.3 logs/day, and the average duration of the completion of the specialty consultation was 51.7 minutes. There was a significant inverse relationship between the frequency of mobile EMR usage and time interval from ED request to consultation completion by specialty doctors (coefficient=–0.19; 95% CI –0.32 to –0.06; P=.005). Secondary analysis with the response time was done. There was also a significant inverse relationship between the frequency of specialty doctors’ mobile EMR usage and the response time to consultation requests (coefficient=–0.18; 95% CI –0.30 to –0.04; P=.009).

Conclusions

Our findings suggest that frequent mobile EMR usage is associated with quicker response time to ED consultation requests.

Developing Embodied Conversational Agents for Coaching People in a Healthy Lifestyle: Scoping Review

BACKGROUND

Embodied conversational agents (ECAs) are animated computer characters that simulate face-to-face counseling. Owing to their capacity to establish and maintain an empathic relationship, they are deemed to be a promising tool for starting and maintaining a healthy lifestyle.

OBJECTIVE

This review aimed to identify the current practices in designing and evaluating ECAs for coaching people in a healthy lifestyle and provide an overview of their efficacy (on behavioral, knowledge, and motivational parameters) and use (on usability, usage, and user satisfaction parameters).

METHODS

We used the Arksey and O'Malley framework to conduct a scoping review. PsycINFO, Medical Literature Analysis and Retrieval System Online, and Scopus were searched with a combination of terms related to ECA and lifestyle. Initially, 1789 unique studies were identified; 20 studies were included.

RESULTS

Most often, ECAs targeted physical activity (n=16) and had the appearance of a middle-aged African American woman (n=13). Multiple behavior change techniques (median=3) and theories or principles (median=3) were applied, but their interpretation and application were usually not reported. ECAs seemed to be designed for the end user rather than with the end user. Stakeholders were usually not involved. A total of 7 out of 15 studies reported better efficacy outcomes for the intervention group, and 5 out of 8 studies reported better use-related outcomes, as compared with the control group.

CONCLUSIONS

ECAs are a promising tool for persuasive communication in the health domain. This review provided valuable insights into the current developmental processes, and it recommends the use of human-centered, stakeholder-inclusive design approaches, along with reporting on the design activities in a systematic and comprehensive manner. The gaps in knowledge were identified on the working mechanisms of intervention components and the right timing and frequency of coaching.

Exploring How Professionals Within Agile Health Care Informatics Perceive Visualizations of Log File Analyses: Observational Study Followed by a Focus Group Interview

BACKGROUND

An increasing number of software companies work according to the agile software development method, which is difficult to integrate with user-centered design (UCD) practices. Log file analysis may provide opportunities for integrating UCD practices in the agile process. However, research within health care information technology mostly has a theoretical approach and is often focused on the researcher's interpretation of log file analyses.

OBJECTIVE

We aimed to propose a systematic approach to log file analysis in this study and present this to developers to explore how they react and interpret this approach in the context of a real-world health care information system, in an attempt to answer the following question: How may log file analyses contribute to increasing the match between the health care system and its users, within the agile development method, according to agile team members?

METHODS

This study comprised 2 phases to answer the research question. In the first phase, log files were collected from a health care information system and subsequently analyzed (summarizing sequential patterns, heat mapping, and clustering). In the second phase, the results of these analyses are presented to agile professionals during a focus group interview. The interpretations of the agile professionals are analyzed by open axial coding.

RESULTS

Log file data of 17,924 user sessions and, in total, 176,678 activities were collected. We found that the Patient Timeline is mainly visited, with 23,707 (23,707/176,678; 13.42%) visits in total. The main unique user session occurred in 5.99% (1074/17,924) of all user sessions, and this comprised Insert Measurement Values for Patient and Patient Timeline, followed by the page Patient Settings and, finally, Patient Treatment Plan. In the heat map, we found that users often navigated to the pages Insert Measurement Values and Load Messages Collaborate. Finally, in the cluster analysis, we found 5 clusters, namely, the Information-seeking cluster, the Collaborative cluster, the Mixed cluster, the Administrative cluster, and the Patient-oriented cluster. We found that the interpretations of these results by agile professionals are related to stating hypotheses (n=34), comparing paths (n=31), benchmarking (n=22), and prioritizing (n=17).

CONCLUSIONS

We found that analyzing log files provides agile professionals valuable insights into users' behavior. Therefore, we argue that log file analyses should be used within agile development to inform professionals about users' behavior. In this way, further UCD research can be informed by these results, making the methods less labor intensive. Moreover, we argue that these translations to an approach for further UCD research will be carried out by UCD specialists, as they are able to infer which goals the user had when going through these paths when looking at the log data.

Effects of the Proactive Interdisciplinary Self-Management (PRISMA) Program on Online Care Platform Usage in Patients with Type 2 Diabetes in Primary Care: A Randomized Controlled Trial

Online care platforms can support patients with type 2 diabetes (T2DM) in managing their health. However, in the use of eHealth, a low participation rate is common. The Proactive Interdisciplinary Self-Management (PRISMA) program, aimed at improving patients' self-management skills, was expected to encourage patients to manage their disease through the use of an online platform. Therefore, the objective of the current study was to investigate whether a group education program can improve the use of an online care platform in patients with T2DM treated by primary care providers in the Netherlands. In a randomized controlled trial, patients with T2DM received either PRISMA with usual care or usual care only. During a six-month follow-up period in 2014-2015, usage (number of log-ons and time spent per session) of an online care platform (e-Vita) aimed at improving T2DM self-management was assessed. A training about the functionalities of e-Vita was offered. The sample consisted of 203 patients. No differences were found between the intervention and control groups in the number of patients who attended the platform training (interested patients) (X 2(1) = 0.58; p = 0.45), and the number of patients who logged on at least once (platform users) (X 2(1) = 0.46; p = 0.50). In addition, no differences were found between the groups in the type of users-patients who logged on twice or more (active users) or patients who logged on once (nonactive users) (X 2(1) = 0.56; p = 0.45). The PRISMA program did not change platform usage in patients with T2DM. In addition, only a small proportion of the patients logged on twice or more. Patients probably need other encouragements to manage their condition using an online platform.

Using Cluster Analysis to Explore Engagement and e-Attainment as Emergent Behavior in Electronic Mental Health

BACKGROUND

In most e-mental health (eMH) research to date, adherence is defined according to a trial protocol. However, adherence to a study protocol may not completely capture a key aspect of why participants engage with eMH tools, namely, to achieve personal mental health goals. As a consequence, trial attrition reported as non-adherence or dropout may reflect e-attainment, the discontinuation of eMH engagement after personal goals have been met. Clarifying engagement patterns, such as e-attainment, and how these align with mental health trajectories, may help optimize eMH design and implementation science.

OBJECTIVE

This study aimed to use clustering techniques to identify real-world engagement profiles in a community of eMH users and examine if such engagement profiles are associated with different mental health outcomes. The novelty of this approach was our attempt to identify actual user engagement behaviors, as opposed to employing engagement benchmarks derived from a trial protocol. The potential of this approach is to link naturalistic behaviors to beneficial mental health outcomes, which would be especially informative when designing eMH programs for the general public.

METHODS

Between May 2013 and June 2018, Australian adults (N=43,631) signed up to myCompass, a self-guided eMH program designed to help alleviate mild to moderate symptoms of depression, anxiety, and stress. Recorded usage data included number of logins, frequency of mood tracking, number of started and completed learning activities, and number of tracking reminders set. A subset of users (n=168) completed optional self-assessment mental health questionnaires (Patient Health Questionnaire-9 item, PHQ-9; Generalized Anxiety Disorder Questionnaire-7 item, GAD-7) at registration and at 28 and 56 days after sign-up. Another subset of users (n=861) completed the PHQ-9 and GAD-7 at registration and at 28 days.

RESULTS

Two-step cluster analyses revealed 3 distinct usage patterns across both subsamples: moderates, trackers, and super users, signifying differences both in the frequency of use as well as differences in preferences for program functionalities. For both subsamples, repeated measures analysis of variances showed significant decreases over time in PHQ-9 and GAD-7 scores. Time-by-cluster interactions, however, did not yield statistical significance in both subsamples, indicating that clusters did not predict symptom reduction over time. Interestingly, users who completed the self-assessment questionnaires twice had slightly but significantly lower depression and anxiety levels at sign-up compared with users who completed the questionnaires a third time at 56 days.

CONCLUSIONS

Findings suggested that although users engaged with myCompass in different but measurable ways, those different usage patterns evoked equivalent mental health benefits. Furthermore, the randomized controlled trial paradigm may unintentionally limit the scope of eMH engagement research by mislabeling early mental health goal achievers as dropouts. More detailed and naturalistic approaches to study engagement with eMH technologies may improve program design and, ultimately, program effectiveness.

Evaluating the implementation of a personal health record for chronic primary and secondary care: a mixed methods approach

BACKGROUND

Personal health records (PHRs) provide the opportunity for self-management support, enhancing communication between patients and caregivers, and maintaining and/or improving the quality of chronic disease management. Their implementation is a multi-level and complex process, requiring a holistic approach that takes into account the technology, its users and the context of implementation. The aim of this research is to evaluate the fidelity of a PHR in chronic care (the degree to which it was implemented as intended) in order to explain the found effects.

METHODS

A convergent parallel mixed methods design was used, where qualitative and quantitative data were collected in parallel, analyzed separately, and finally merged. Log data of 536 users were used to gain insight into the actual long-term use of the PHR (the dose). Focus group meetings among caregivers (n = 13) were conducted to assess program differentiation (or intended use). Interviews with caregivers (n = 28) and usability tests with potential end-users (n = 13) of the PHR were used to understand the responsiveness and the differences and similarities between the intended and actual use of the PHR.

RESULTS

The results of the focus groups showed that services for coaching are strongly associated with monitoring health values and education. However, the PHR was not used that way during the study period. In the interviews, caregivers indicated that they were ignorant on how to deploy the PHR in current working routines. Therefore, they find it difficult to motivate their patients in using the PHR. Participants in the usability study indicate that they would value a PHR in the future, given that the usability will be improved and that the caregivers will use it in daily practice as well.

CONCLUSIONS

In this study, actual use of the PHRs by patients was influenced by the responsiveness of caregivers. This responsiveness is likely to be strongly influenced by the perceived support when defining the differentiation and delivery of the PHR. A mixed-methods approach to understand intervention fidelity was of added value in providing explanations for the found effects that could not be revealed by solely focusing on the effectiveness of the technology in an experimental trial.

The Service of Research Analytics to Optimize Digital Health Evidence Generation: Multilevel Case Study

Background

The widespread adoption of digital health interventions for chronic disease self-management has catalyzed a paradigm shift in the selection of methodologies used to evidence them. Recently, the application of digital health research analytics has emerged as an efficient approach to evaluate these data-rich interventions. However, there is a growing mismatch between the promising evidence base emerging from analytics mediated trials and the complexity of introducing these novel research methods into evaluative practice.

Objective

This study aimed to generate transferable insights into the process of implementing research analytics to evaluate digital health interventions. We sought to answer the following two research questions: (1) how should the service of research analytics be designed to optimize digital health evidence generation? and (2) what are the challenges and opportunities to scale, spread, and sustain this service in evaluative practice?

Methods

We conducted a qualitative multilevel embedded single case study of implementing research analytics in evaluative practice that comprised a review of the policy and regulatory climate in Ontario (macro level), a field study of introducing a digital health analytics platform into evaluative practice (meso level), and interviews with digital health innovators on their perceptions of analytics and evaluation (microlevel).

Results

The practice of research analytics is an efficient and effective means of supporting digital health evidence generation. The introduction of a research analytics platform to evaluate effective engagement with digital health interventions into a busy research lab was ultimately accepted by research staff, became routinized in their evaluative practice, and optimized their existing mechanisms of log data analysis and interpretation. The capacity for research analytics to optimize digital health evaluations is highest when there is (1) a collaborative working relationship between research client and analytics service provider, (2) a data-driven research agenda, (3) a robust data infrastructure with clear documentation of analytic tags, (4) in-house software development expertise, and (5) a collective tolerance for methodological change.

Conclusions

Scientific methods and practices that can facilitate the agile trials needed to iterate and improve digital health interventions warrant continued implementation. The service of research analytics may help to accelerate the pace of digital health evidence generation and build a data-rich research infrastructure that enables continuous learning and evaluation.

Exploring the Relationship Between Usage and Outcomes of an Internet-Based Intervention for Individuals With Depressive Symptoms: Secondary Analysis of Data From a Randomized Controlled Trial

Background

Internet interventions can easily generate objective data about program usage. Increasingly, more studies explore the relationship between usage and outcomes, but they often report different metrics of use, and the findings are mixed. Thus, current evaluations fail to demonstrate which metrics should be considered and how these metrics are related to clinically meaningful change.

Objective

This study aimed to explore the relationship between several usage metrics and outcomes of an internet-based intervention for depression.

Methods

This is a secondary analysis of data from a randomized controlled trial that examined the efficacy of an internet-based cognitive behavioral therapy for depression (Space from Depression) in an adult community sample. All participants who enrolled in the intervention, regardless of meeting the inclusion criteria, were included in this study. Space from Depression is a 7-module supported intervention, delivered over a period of 8 weeks. Different usage metrics (ie, time spent, modules and activities completed, and percentage of program completion) were automatically collected by the platform, and composite variables from these (eg, activities per session) were computed. A breakdown of the usage metrics was obtained by weeks. For the analysis, the sample was divided into those who obtained a reliable change (RC)—and those who did not.

Results

Data from 216 users who completed pre- and posttreatment outcomes were included in the analyses. A total of 89 participants obtained an RC, and 127 participants did not obtain an RC. Those in the RC group significantly spent more time, had more log-ins, used more tools, viewed a higher percentage of the program, and got more reviews from their supporter compared with those who did not obtain an RC. Differences between groups in usage were observed from the first week in advance across the different metrics, although they vanished over time. In the RC group, the usage was higher during the first 4 weeks, and then a significant decrease was observed. Our results showed that specific levels of platform usage, 7 hours total time spent, 15 sessions, 30 tools used, and 50% of program completion, were associated with RC.

Conclusions

Overall, the results showed that those individuals who obtained an RC after the intervention had higher levels of exposure to the platform. The usage during the first half of the intervention was higher, and differences between groups were observed from the first week. This study also showed specific usage levels associated with outcomes that could be tested in controlled studies to inform the minimal usage to establish adherence. These results will help to better understand how to use internet-based interventions and what optimal level of engagement can most affect outcomes.

Trial Registration

ISRCTN Registry ISRCTN03704676; http://www.isrctn.com/ISRCTN03704676

International Registered Report Identifier (IRRID)

RR2-10.1186/1471-244X-14-147

Persuasive System Design Principles and Behavior Change Techniques to Stimulate Motivation and Adherence in Electronic Health Interventions to Support Weight Loss Maintenance: Scoping Review

BACKGROUND

Maintaining weight after weight loss is a major health challenge, and eHealth (electronic health) solutions may be a way to meet this challenge. Application of behavior change techniques (BCTs) and persuasive system design (PSD) principles in eHealth development may contribute to the design of technologies that positively influence behavior and motivation to support the sustainable health behavior change needed.

OBJECTIVE

This review aimed to identify BCTs and PSD principles applied in eHealth interventions to support weight loss and weight loss maintenance, as well as techniques and principles applied to stimulate motivation and adherence for long-term weight loss maintenance.

METHODS

A systematic literature search was conducted in PsycINFO, Ovid MEDLINE (including PubMed), EMBASE, Scopus, Web of Science, and AMED, from January 1, 2007 to June 30, 2018. Arksey and O'Malley's scoping review methodology was applied. Publications on eHealth interventions were included if focusing on weight loss or weight loss maintenance, in combination with motivation or adherence and behavior change.

RESULTS

The search identified 317 publications, of which 45 met the inclusion criteria. Of the 45 publications, 11 (24%) focused on weight loss maintenance, and 34 (76%) focused on weight loss. Mobile phones were the most frequently used technology (28/45, 62%). Frequently used wearables were activity trackers (14/45, 31%), as well as other monitoring technologies such as wireless or digital scales (8/45, 18%). All included publications were anchored in behavior change theories. Feedback and monitoring and goals and planning were core behavior change technique clusters applied in the majority of included publications. Social support and associations through prompts and cues to support and maintain new habits were more frequently used in weight loss maintenance than weight loss interventions. In both types of interventions, frequently applied persuasive principles were self-monitoring, goal setting, and feedback. Tailoring, reminders, personalization, and rewards were additional principles frequently applied in weight loss maintenance interventions. Results did not reveal an ideal combination of techniques or principles to stimulate motivation, adherence, and weight loss maintenance. However, the most frequently mentioned individual techniques and principles applied to stimulate motivation were, personalization, simulation, praise, and feedback, whereas associations were frequently mentioned to stimulate adherence. eHealth interventions that found significant effects for weight loss maintenance all applied self-monitoring, feedback, goal setting, and shaping knowledge, combined with a human social support component to support healthy behaviors.

CONCLUSIONS

To our knowledge, this is the first review examining key BCTs and PSD principles applied in weight loss maintenance interventions compared with those of weight loss interventions. This review identified several techniques and principles applied to stimulate motivation and adherence. Future research should aim to examine which eHealth design combinations can be the most effective in support of long-term behavior change and weight loss maintenance.

Data Mining in the Development of Mobile Health Apps: Assessing In-App Navigation Through Markov Chain Analysis

Background

Mobile apps generate vast amounts of user data. In the mobile health (mHealth) domain, researchers are increasingly discovering the opportunities of log data to assess the usage of their mobile apps. To date, however, the analysis of these data are often limited to descriptive statistics. Using data mining techniques, log data can offer significantly deeper insights.

Objective

The purpose of this study was to assess how Markov Chain and sequence clustering analysis can be used to find meaningful usage patterns of mHealth apps.

Methods

Using the data of a 25-day field trial (n=22) of the Start2Cycle app, an app developed to encourage recreational cycling in adults, a transition matrix between the different pages of the app was composed. From this matrix, a Markov Chain was constructed, enabling intuitive user behavior analysis.

Results

Through visual inspection of the transitions, 3 types of app use could be distinguished (route tracking, gamification, and bug reporting). Markov Chain–based sequence clustering was subsequently used to demonstrate how clusters of session types can otherwise be obtained.

Conclusions

Using Markov Chains to assess in-app navigation presents a sound method to evaluate use of mHealth interventions. The insights can be used to evaluate app use and improve user experience.

Text Message Responsivity in a 2-Way Short Message Service Pilot Intervention With Adolescent and Young Adult Survivors of Cancer

OBJECTIVE

Within a 2-way text messaging study in AYAs who recently completed treatment for cancer, we sought to evaluate text message responsivity across different types of text messages.

METHODS

AYAs who recently completed treatment for cancer (n=26; mean age=16 years; 62% female, 16/26 participants) received 2-way text messages about survivorship health topics over a 16-week period. Using participants' text message log data, we coded responsivity to text messages and evaluated trends in responsivity to unprompted text messages and prompted text messages of varying content (eg, medication reminders, appointment reminders, and texts about personal experiences as a cancer survivor).

RESULTS

Across prompted and unprompted text messages, responsivity rapidly decreased (P ≤.001 and =.01, respectively) and plateaued by the third week of the intervention. However, participants were more responsive to prompted text messages (mean responsivity=46% by week 16) than unprompted messages (mean responsivity=10% by week 16). They also demonstrated stable responsivity to certain prompted content: medication reminders, appointment reminders, goal motivation, goal progress, and patient experience texts.

CONCLUSIONS

Our methodology of evaluating text message responsivity revealed important patterns of engagement in a 2-way text message intervention for AYA cancer survivors.

Patient Engagement as Measured by Inpatient Portal Use: Methodology for Log File Analysis

Background

Inpatient portals (IPPs) have the potential to increase patient engagement and satisfaction with their health care. An IPP provides a hospitalized patient with similar functions to those found in outpatient portals, including the ability to view vital signs, laboratory results, and medication information; schedule appointments; and communicate with their providers. However, IPPs may offer additional functions such as meal planning, real-time messaging with the inpatient care team, daily schedules, and access to educational materials relevant to their specific condition. In practice, IPPs have been developed as websites and tablet apps, with hospitals providing the required technology as a component of care during the patient’s stay.

Objective

This study aimed to describe how inpatients are using IPPs at the first academic medical center to implement a system-wide IPP and document the challenges and choices associated with this analytic process.

Methods

We analyzed the audit log files of IPP users hospitalized between January 2014 and January 2016. Data regarding the date/time and duration of interactions with each of the MyChart Bedside modules (eg, view lab results or medications and patient schedule) and activities (eg, messaging the provider and viewing educational videos) were captured as part of the system audit logs. The development of a construct to describe the length of time associated with a single coherent use of the tool—which we call a session—provides a foundational unit of analysis. We defined frequency as the number of sessions a patient has during a given provision day. We defined comprehensiveness in terms of the percentage of functions that an individual uses during a given provision day.

Results

The analytic process presented data challenges such as length of stay and tablet-provisioning factors. This study presents data visualizations to illustrate a series of data-cleaning issues. In the presence of these robust approaches to data cleaning, we present the baseline usage patterns associated with our patient panel. In addition to frequency and comprehensiveness, we present considerations of median data to mitigate the effect of outliers.

Conclusions

Although other studies have published usage data associated with IPPs, most have not explicated the challenges and choices associated with the analytic approach deployed within each study. Our intent in this study was to be somewhat exhaustive in this area, in part, because replicability requires common metrics. Our hope is that future researchers in this area will avail themselves of these perspectives to engage in critical assessment moving forward.

A Framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) in Digital Interventions: Viewpoint

Trials of digital interventions can yield extensive, in-depth usage data, yet usage analyses tend to focus on broad descriptive summaries of how an intervention has been used by the whole sample. This paper proposes a novel framework to guide systematic, fine-grained usage analyses that better enables understanding of how an intervention works, when, and for whom. The framework comprises three stages to assist in the following: (1) familiarization with the intervention and its relationship to the captured data, (2) identification of meaningful measures of usage and specifying research questions to guide systematic analyses of usage data, and (3) preparation of datasheets and consideration of available analytical methods with which to examine the data. The framework can be applied to inform data capture during the development of a digital intervention and/or in the analysis of data after the completion of an evaluation trial. We will demonstrate how the framework shaped preparation and aided efficient data capture for a digital intervention to lower transmission of cold and flu viruses in the home, as well as how it informed a systematic, in-depth analysis of usage data collected from a separate digital intervention designed to promote self-management of colds and flu. The Analyzing and Measuring Usage and Engagement Data (AMUsED) framework guides systematic and efficient in-depth usage analyses that will support standardized reporting with transparent and replicable findings. These detailed findings may also enable examination of what constitutes effective engagement with particular interventions.

The GET READY relapse prevention programme for anxiety and depression: a mixed-methods study protocol

BACKGROUND

Since anxiety and depressive disorders often recur, self-management competencies are crucial for improving the long-term course of anxiety and depressive disorders. However, few relapse prevention programmes are available that focus on improving self-management. E-health combined with personal contact with a mental health professional in general practice might be a promising approach for relapse prevention. In this protocol, the GET READY (Guided E-healTh for RElapse prevention in Anxiety and Depression) study will be described in which a relapse prevention programme is developed, implemented and evaluated. The aim of the study is to determine patients' usage of the programme and the associated course of their symptoms, to examine barriers and facilitators of implementation, and to assess patients' satisfaction with the programme.

METHODS

Participants are discharged from mental healthcare services, and are in complete or partial remission. They receive access to an E-health platform, combined with regular contact with a mental health professional in general practices. Online questionnaires will be completed at baseline and after 3, 6 and 9 months. Also, semi-structured qualitative individual interviews and focus group interviews will be conducted with patients and mental health professionals.

DISCUSSION

This mixed-methods observational cohort study will provide insights into the use of a relapse prevention programme in relation to the occurrence of symptoms, as well as in its implementation and evaluation. Using the results of this study, the relapse prevention programme can be adapted in accordance with the needs of patients and mental health professionals. If this programme is shown to be acceptable, a randomized controlled trial may be conducted to test its efficacy.

TRIAL REGISTRATION

Retrospectively registered in the Netherlands Trial Register ( NTR7574 ; 25 October 2018).

A Library of Analytic Indicators to Evaluate Effective Engagement with Consumer mHealth Apps for Chronic Conditions: Scoping Review

Background

There is mixed evidence to support current ambitions for mobile health (mHealth) apps to improve chronic health and well-being. One proposed explanation for this variable effect is that users do not engage with apps as intended. The application of analytics, defined as the use of data to generate new insights, is an emerging approach to study and interpret engagement with mHealth interventions.

Objective

This study aimed to consolidate how analytic indicators of engagement have previously been applied across clinical and technological contexts, to inform how they might be optimally applied in future evaluations.

Methods

We conducted a scoping review to catalog the range of analytic indicators being used in evaluations of consumer mHealth apps for chronic conditions. We categorized studies according to app structure and application of engagement data and calculated descriptive data for each category. Chi-square and Fisher exact tests of independence were applied to calculate differences between coded variables.

Results

A total of 41 studies met our inclusion criteria. The average mHealth evaluation included for review was a two-group pretest-posttest randomized controlled trial of a hybrid-structured app for mental health self-management, had 103 participants, lasted 5 months, did not provide access to health care provider services, measured 3 analytic indicators of engagement, segmented users based on engagement data, applied engagement data for descriptive analyses, and did not report on attrition. Across the reviewed studies, engagement was measured using the following 7 analytic indicators: the number of measures recorded (76%, 31/41), the frequency of interactions logged (73%, 30/41), the number of features accessed (49%, 20/41), the number of log-ins or sessions logged (46%, 19/41), the number of modules or lessons started or completed (29%, 12/41), time spent engaging with the app (27%, 11/41), and the number or content of pages accessed (17%, 7/41). Engagement with unstructured apps was mostly measured by the number of features accessed (8/10, P=.04), and engagement with hybrid apps was mostly measured by the number of measures recorded (21/24, P=.03). A total of 24 studies presented, described, or summarized the data generated from applying analytic indicators to measure engagement. The remaining 17 studies used or planned to use these data to infer a relationship between engagement patterns and intended outcomes.

Conclusions

Although researchers measured on average 3 indicators in a single study, the majority reported findings descriptively and did not further investigate how engagement with an app contributed to its impact on health and well-being. Researchers are gaining nuanced insights into engagement but are not yet characterizing effective engagement for improved outcomes. Raising the standard of mHealth app efficacy through measuring analytic indicators of engagement may enable greater confidence in the causal impact of apps on improved chronic health and well-being.

An Analytics Platform to Evaluate Effective Engagement With Pediatric Mobile Health Apps: Design, Development, and Formative Evaluation

BACKGROUND

Mobile health (mHealth) apps for pediatric chronic conditions are growing in availability and challenge investigators to conduct rigorous evaluations that keep pace with mHealth innovation. Traditional research methods are poorly suited to operationalize the agile, iterative trials required to evidence and optimize these digitally mediated interventions.

OBJECTIVE

We sought to contribute a resource to support the quantification, analysis, and visualization of analytic indicators of effective engagement with mHealth apps for chronic conditions.

METHODS

We applied user-centered design methods to design and develop an Analytics Platform to Evaluate Effective Engagement (APEEE) with consumer mHealth apps for chronic conditions and implemented the platform to analyze both retrospective and prospective data generated from a smartphone-based pain self-management app called iCanCope for young people with chronic pain.

RESULTS

Through APEEE, we were able to automate the process of defining, operationalizing, and evaluating effective engagement with iCanCope. Configuring the platform to integrate with the app was feasible and provided investigators with a resource to consolidate, analyze, and visualize engagement data generated by participants in real time. Preliminary efforts to evaluate APEEE showed that investigators perceived the platform to be an acceptable evaluative resource and were satisfied with its design, functionality, and performance. Investigators saw potential in APEEE to accelerate and augment evidence generation and expressed enthusiasm for adopting the platform to support their evaluative practice once fully implemented.

CONCLUSIONS

Dynamic, real-time analytic platforms may provide investigators with a powerful means to characterize the breadth and depth of mHealth app engagement required to achieve intended health outcomes. Successful implementation of APEEE into evaluative practice may contribute to the realization of effective and evidence-based mHealth care.

Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence

BACKGROUND

In electronic health (eHealth) evaluations, there is increasing attention for studying the actual usage of a technology in relation to the outcomes found, often by studying the adherence to the technology. On the basis of the definition of adherence, we suggest that the following three elements are necessary to determine adherence to eHealth technology: (1) the ability to measure the usage behavior of individuals; (2) an operationalization of intended use; and (3) an empirical, theoretical, or rational justification of the intended use. However, to date, little is known on how to operationalize the intended usage of and the adherence to different types of eHealth technology.

OBJECTIVE

The study aimed to improve eHealth evaluations by gaining insight into when, how, and by whom the concept of adherence has been used in previous eHealth evaluations and finding a concise way to operationalize adherence to and intended use of different eHealth technologies.

METHODS

A systematic review of eHealth evaluations was conducted to gain insight into how the use of the technology was measured, how adherence to different types of technologies was operationalized, and if and how the intended use of the technology was justified. Differences in variables between the use of the technology and the operationalization of adherence were calculated using a chi-square test of independence.

RESULTS

In total, 62 studies were included in this review. In 34 studies, adherence was operationalized as "the more use, the better," whereas 28 studies described a threshold for intended use of the technology as well. Out of these 28, only 6 reported a justification for the intended use. The proportion of evaluations of mental health technologies reporting a justified operationalization of intended use is lagging behind compared with evaluations of lifestyle and chronic care technologies. The results indicated that a justification of intended use does not require extra measurements to determine adherence to the technology.

CONCLUSIONS

The results of this review showed that to date, justifications for intended use are often missing in evaluations of adherence. Evidently, it is not always possible to estimate the intended use of a technology. However, such measures do not meet the definition of adherence and should therefore be referred to as the actual usage of the technology. Therefore, it can be concluded that adherence to eHealth technology is an underdeveloped and often improperly used concept in the existing body of literature. When defining the intended use of a technology and selecting valid measures for adherence, the goal or the assumed working mechanisms should be leading. Adherence can then be standardized, which will improve the comparison of adherence rates to different technologies with the same goal and will provide insight into how adherence to different elements contributed to the outcomes.