Analyzing engagement in a web-based intervention platform through visualizing log-data

Measuring Engagement in Provider-Guided Digital Health Interventions With a Conceptual and Analytical Framework Using Nurse WRITE as an Exemplar: Exploratory Study With an Iterative Approach

BACKGROUND

Limited guidance exists for analyzing participant engagement in provider-guided digital health interventions (DHIs). System usage is commonly assessed, with acknowledged limitations in measuring socio-affective and cognitive aspects of engagement. Nurse WRITE, an 8-week web-based nurse-guided DHI for managing symptoms among women with recurrent ovarian cancer, offers an opportunity to develop a framework for assessing multidimensional engagement.

OBJECTIVE

This study aims to develop a conceptual and analytic framework to measure socio-affective, cognitive, and behavioral engagement with provider-guided DHIs. We then illustrate the framework's ability to describe and categorize engagement using Nurse WRITE as an example.

METHODS

A sample of 68 participants from Nurse WRITE who posted on the message boards were included. We adapted a prior framework for conceptualizing and operationalizing engagement across 3 dimensions and finalized a set of 6 distinct measures. Using patients' posts, we created 2 socio-affective engagement measures-total count of socio-affective engagement classes (eg, sharing personal experience) and total word count-and 2 cognitive engagement measures-total count of cognitive engagement classes (eg, asking information-seeking questions) and average question completion percentage. Additionally, we devised behavioral engagement measures using website data-the total count of symptom care plans and plan reviews. k-Means clustering categorized the participants into distinct groups based on levels of engagement across 3 dimensions. Descriptive statistics and narratives were used to describe engagement in 3 dimensions.

RESULTS

On average, participants displayed socio-affective engagement 34.7 times, writing 14,851 words. They showed cognitive engagement 19.4 times, with an average of 78.3% completion of nurses' inquiries. Participants also submitted an average of 1.6 symptom care plans and 0.7 plan reviews. Participants were clustered into high (n=13), moderate (n=17), and low engagers (n=38) based on the 6 measures. High engagers wrote a median of 36,956 (IQR 26,199-46,265) words. They demonstrated socio-affective engagement approximately 81 times and cognitive engagement around 46 times, approximately 6 times that of the low engagers and twice that of the moderate engagers. High engagers had a median of 91.7% (IQR 82.2%-93.7%) completion of the nurses' queries, whereas moderate engagers had 86.4% (IQR 80%-96.4%), and low engagers had 68.3% (IQR 60.1%-79.6%). High engagers completed a median of 3 symptom care plans and 2 reviews, while moderate engagers completed 2 plans and 1 review. Low engagers completed a median of 1 plan with no reviews.

CONCLUSIONS

This study developed and reported an engagement framework to guide behavioral intervention scientists in understanding and analyzing participants' engagement with provider-guided DHIs. Significant variations in engagement levels across 3 dimensions highlight the importance of measuring engagement with provider-guided DHIs in socio-affective, cognitive, and behavioral dimensions. Future studies should validate the framework with other DHIs, explore the influence of patient and provider factors on engagement, and investigate how engagement influences intervention efficacy.

Examining young adults daily perspectives on usage of anxiety apps: A user study

The growing number of mental health smartphone applications has led to increased interest in how these tools might support users in different models of care. However, research on the use of these interventions in real-world settings has been scarce. It is important to understand how apps are used in a deployment setting, especially among populations where such tools might add value to current models of care. The objective of this study is to explore the daily use of commercially-available mobile apps for anxiety that integrate CBT, with a focus on understanding reasons for and barriers for app use and engagement. This study recruited 17 young adults (age M = 24.17 years) while on a waiting list to receive therapy in a Student Counselling Service. Participants were asked to select up to two of a list of three selected apps (Wysa, Woebot, and Sanvello) and instructed to use the apps for two weeks. Apps were selected because they used techniques from cognitive behavioral therapy, and offer diverse functionality for anxiety management. Qualitative and quantitative data were gathered through daily questionnaires to capture participants' experiences with the mobile apps. In addition, eleven semi-structured interviews were conducted at the end of the study. We used descriptive statistics to analyze participants' interaction with different app features and used a general inductive approach to analyze the collected qualitative data. The results highlight that users form opinions about the apps during the first days of app use. A number of barriers to sustained use are identified including cost-related issues, inadequate content to support long-term use, and a lack of customization options for different app functions. The app features used differ among participants with self-monitoring and treatment elements being the most used features.

Interactivity, Quality, and Content of Websites Promoting Health Behaviors During Infancy: 6-Year Update of the Systematic Assessment

BACKGROUND

As of 2021, 89% of the Australian population are active internet users. Although the internet is widely used, there are concerns about the quality, accuracy, and credibility of health-related websites. A 2015 systematic assessment of infant feeding websites and apps available in Australia found that 61% of websites were of poor quality and readability, with minimal coverage of infant feeding topics and lack of author credibility.

OBJECTIVE

We aimed to systematically assess the quality, interactivity, readability, and comprehensibility of information targeting infant health behaviors on websites globally and provide an update of the 2015 systematic assessment.

METHODS

Keywords related to infant milk feeding behaviors, solid feeding behaviors, active play, screen time, and sleep were used to identify websites targeting infant health behaviors on the Google search engine on Safari. The websites were assessed by a subset of the authors using predetermined criteria between July 2021 and February 2022 and assessed for information content based on the Australian Infant Feeding Guidelines and National Physical Activity Recommendations. The Suitability Assessment of Materials, Quality Component Scoring System, the Health-Related Website Evaluation Form, and the adherence to the Health on the Net code were used to evaluate the suitability and quality of information. Readability was assessed using 3 web-based readability tools.

RESULTS

Of the 450 websites screened, 66 were included based on the selection criteria and evaluated. Overall, the quality of websites was mostly adequate. Media-related sources, nongovernmental organizations, hospitals, and privately owned websites had the highest median quality scores, whereas university websites received the lowest median score (35%). The information covered within the websites was predominantly poor: 91% (60/66) of the websites received an overall score of ≤74% (mean 53%, SD 18%). The suitability of health information was mostly rated adequate for literacy demand, layout, and learning and motivation of readers. The median readability score for the websites was grade 8.5, which is higher than the government recommendations (

CONCLUSIONS

Quality, content, readability, and interactivity of websites promoting health behaviors during infancy ranged between poor and adequate. Since the 2015 systematic assessment, there was a slight improvement in the quality of websites but no difference in the Suitability Assessment of Materials rating and readability of information. There is a need for researchers and health care providers to leverage innovative web-based platforms to provide culturally competent evidence-based information based on government guidelines that are accessible to those with limited English proficiency.

Collapse

Key Words

• accuracy
• babies
• baby
• bottle feeding
• breastfeeding
• comprehensibility
• consumer
• eating
• feeding
• food
• grading
• health behavior
• health information
• health website
• infant
• infant food
• interactivity
• nutrition
• quality
• rating
• readability
• screen time
• sleep
• solid food
• web-based information
• web-based platform
• websites

Collapse

MESH Headings

• Australia
• Comprehension
• Consumer Health Information
• Health Behavior
• Humans
• Internet
• Search Engine

Collapse

Grants Collapse

Affiliation(s)

Danielle Jawad Sydney School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia Health Promotion Unit, Population Health Research & Evaluation Hub, Sydney Local Health District, Sydney, Australia National Health and Medical Research Council Centre of Research Excellence in the Early Prevention of Obesity in Childhood - Translate, The University of Sydney, Sydney, Australia
Heilok Cheng National Health and Medical Research Council Centre of Research Excellence in the Early Prevention of Obesity in Childhood - Translate, The University of Sydney, Sydney, Australia Susan Wakil School of Nursing and Midwifery, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia Sydney Institute for Women, Children and their Families, Sydney Local Health District, Sydney, Australia
Li Ming Wen Sydney School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia Health Promotion Unit, Population Health Research & Evaluation Hub, Sydney Local Health District, Sydney, Australia National Health and Medical Research Council Centre of Research Excellence in the Early Prevention of Obesity in Childhood - Translate, The University of Sydney, Sydney, Australia Sydney Institute for Women, Children and their Families, Sydney Local Health District, Sydney, Australia
Chris Rissel Sydney School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia College of Medicine and Public Health, Rural and Remote Health South Australia and Northern Territory, Flinders University, Darwin, Australia
Louise Baur Sydney School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia National Health and Medical Research Council Centre of Research Excellence in the Early Prevention of Obesity in Childhood - Translate, The University of Sydney, Sydney, Australia Specialty of Child and Adolescent Health, Sydney Medical School, The University of Sydney, Sydney, Australia
Seema Mihrshahi Department of Health Sciences, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, Australia
Sarah Taki Sydney School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia Health Promotion Unit, Population Health Research & Evaluation Hub, Sydney Local Health District, Sydney, Australia National Health and Medical Research Council Centre of Research Excellence in the Early Prevention of Obesity in Childhood - Translate, The University of Sydney, Sydney, Australia Sydney Institute for Women, Children and their Families, Sydney Local Health District, Sydney, Australia

Collapse

Understanding users’ perspectives on mobile apps for anxiety management

Anxiety disorders are the most common type of mental health problem. The potential of apps to improve mental health has led to an increase in the number of anxiety apps available. Even though anxiety apps hold the potential to enhance mental health care for individuals, there is relatively little knowledge concerning users’ perspectives. This mixed-methods study aims to understand the nature of user burden and engagement with mental health apps (MHapps) targeting anxiety management, in order to identify ways to improve the design of these apps. Users’ perspectives on these apps were gathered by analyzing 600 reviews from 5 apps on the app stores (Study 1), and conducting 15 interviews with app users (Study 2). The results shed light on several barriers to adoption and sustained use. Users appreciate apps that offer content variation, customizability, and good interface design, and often requested an enhanced, personalized experience to improve engagement. We propose addressing the specific app quality issues identified through human-centered design, more personalized content delivery, and by improving features for social and therapeutic support.

Is Fun For Wellness Engaging? Evaluation of User Experience of an Online Intervention to Promote Well-Being and Physical Activity

Online well-being interventions demonstrate great promise in terms of both engagement and outcomes. Fun For Wellness (FFW) is a novel online intervention grounded in self-efficacy theory and intended to improve multidimensional well-being and physical activity through multi-modal methods. These strategies include capability-enhancing opportunities, learning experiences such as games, video vignettes, and self-assessments. RCT studies have suggested that FFW is efficacious in improving subjective and domain-specific well-being, and effective in improving mental health, physical health, physical activity, and self-efficacy in United States. adults who are overweight and in the general population. The present study uses qualitative and quantitative user experience data collected during two RCT trials to understand and evaluate engagement with FFW, its drivers, and its outcomes. Results suggest that FFW is enjoyable, moderately engaging, and easy to use; and contributes to positive outcomes including skill development and enhanced confidence, for both overweight individuals and the general adult population. Drivers of engagement appear to include rewards, gamification, scenario-based learning, visual tracking for self-monitoring, ease of use and simple communications, and the entertaining, interactive nature of program activities. Findings indicate that there are opportunities to streamline and simplify the experience. These results can help improve FFW and contribute to the science of engagement with online interventions designed to improve well-being.

The promise of machine learning in predicting treatment outcomes in psychiatry

For many years, psychiatrists have tried to understand factors involved in response to medications or psychotherapies, in order to personalize their treatment choices. There is now a broad and growing interest in the idea that we can develop models to personalize treatment decisions using new statistical approaches from the field of machine learning and applying them to larger volumes of data. In this pursuit, there has been a paradigm shift away from experimental studies to confirm or refute specific hypotheses towards a focus on the overall explanatory power of a predictive model when tested on new, unseen datasets. In this paper, we review key studies using machine learning to predict treatment outcomes in psychiatry, ranging from medications and psychotherapies to digital interventions and neurobiological treatments. Next, we focus on some new sources of data that are being used for the development of predictive models based on machine learning, such as electronic health records, smartphone and social media data, and on the potential utility of data from genetics, electrophysiology, neuroimaging and cognitive testing. Finally, we discuss how far the field has come towards implementing prediction tools in real-world clinical practice. Relatively few retrospective studies to-date include appropriate external validation procedures, and there are even fewer prospective studies testing the clinical feasibility and effectiveness of predictive models. Applications of machine learning in psychiatry face some of the same ethical challenges posed by these techniques in other areas of medicine or computer science, which we discuss here. In short, machine learning is a nascent but important approach to improve the effectiveness of mental health care, and several prospective clinical studies suggest that it may be working already.

Ethical Issues in Democratizing Digital Phenotypes and Machine Learning in the Next Generation of Digital Health Technologies

Digital phenotyping is the term given to the capturing and use of user log data from health and wellbeing technologies used in apps and cloud-based services. This paper explores ethical issues in making use of digital phenotype data in the arena of digital health interventions. Products and services based on digital wellbeing technologies typically include mobile device apps as well as browser-based apps to a lesser extent, and can include telephony-based services, text-based chatbots, and voice-activated chatbots. Many of these digital products and services are simultaneously available across many channels in order to maximize availability for users. Digital wellbeing technologies offer useful methods for real-time data capture of the interactions of users with the products and services. It is possible to design what data are recorded, how and where it may be stored, and, crucially, how it can be analyzed to reveal individual or collective usage patterns. The paper also examines digital phenotyping workflows, before enumerating the ethical concerns pertaining to different types of digital phenotype data, highlighting ethical considerations for collection, storage, and use of the data. A case study of a digital health app is used to illustrate the ethical issues. The case study explores the issues from a perspective of data prospecting and subsequent machine learning. The ethical use of machine learning and artificial intelligence on digital phenotype data and the broader issues in democratizing machine learning and artificial intelligence for digital phenotype data are then explored in detail.

Engagement With a Behavior Change App for Alcohol Reduction: Data Visualization for Longitudinal Observational Study

Background

Behavior change apps can develop iteratively, where the app evolves into a complex, dynamic, or personalized intervention through cycles of research, development, and implementation. Understanding how existing users engage with an app (eg, frequency, amount, depth, and duration of use) can help guide further incremental improvements. We aim to explore how simple visualizations can provide a good understanding of temporal patterns of engagement, as usage data are often longitudinal and rich.

Objective

This study aims to visualize behavioral engagement with Drink Less, a behavior change app to help reduce hazardous and harmful alcohol consumption in the general adult population of the United Kingdom.

Methods

We explored behavioral engagement among 19,233 existing users of Drink Less. Users were included in the sample if they were from the United Kingdom; were 18 years or older; were interested in reducing their alcohol consumption; had a baseline Alcohol Use Disorders Identification Test score of 8 or above, indicative of excessive drinking; and had downloaded the app between May 17, 2017, and January 22, 2019 (615 days). Measures of when sessions begin, length of sessions, time to disengagement, and patterns of use were visualized with heat maps, timeline plots, k-modes clustering analyses, and Kaplan-Meier plots.

Results

The daily 11 AM notification is strongly associated with a change in engagement in the following hour; reduction in behavioral engagement over time, with 50.00% (9617/19,233) of users disengaging (defined as no use for 7 or more consecutive days) 22 days after download; identification of 3 distinct trajectories of use, namely engagers (4651/19,233, 24.18% of users), slow disengagers (3679/19,233, 19.13% of users), and fast disengagers (10,903/19,233, 56.68% of users); and limited depth of engagement with 85.076% (7,095,348/8,340,005) of screen views occurring within the Self-monitoring and Feedback module. In addition, a peak of both frequency and amount of time spent per session was observed in the evenings.

Conclusions

Visualizations play an important role in understanding engagement with behavior change apps. Here, we discuss how simple visualizations helped identify important patterns of engagement with Drink Less. Our visualizations of behavioral engagement suggest that the daily notification substantially impacts engagement. Furthermore, the visualizations suggest that a fixed notification policy can be effective for maintaining engagement for some users but ineffective for others. We conclude that optimizing the notification policy to target both effectiveness and engagement is a worthwhile investment. Our future goal is to both understand the causal effect of the notification on engagement and further optimize the notification policy within Drink Less by tailoring to contextual circumstances of individuals over time. Such tailoring will be informed from the findings of our micro-randomized trial (MRT), and these visualizations were useful in both gaining a better understanding of engagement and designing the MRT.

Engagement with a Web-Based Health Promotion Intervention among Vocational School Students: A Secondary User and Usage Analysis

Engagement with web-based interventions is both generally low and typically declining. Visits and revisits remain a challenge. Based on log data of a web-based cluster randomized controlled trial conducted in vocational schools, the present secondary analysis aimed to identify influencing factors on initially logging in to a health promotion platform among young adults and to examine the engagement over the course of an eight-week intervention. Data of 336 students (62.2% female, age span 18–25) from two intervention arms (web-based intervention and web-based intervention with an additional initial face-to-face contact) was included. Binary logistic regression and log-data visualization were performed. An additional initial face-to-face contact (odds ratio (OR) = 2.971, p = 0.005), female sex (OR = 2.237, p = 0.046) and the health-related skill “dealing with health information” (OR = 2.179, p = 0.030) significantly increased the likelihood of initially logging in. Other variables showed no influence. 16.6% of all potential users logged in at least once, of which 57.4% revisited the platform. Most logins were tracked at the beginning of the intervention and repeated engagement was low. To increase the engagement with web-based interventions, health-related skills should be fostered. In addition, a strategy could be to interlink comparable interventions in vocational schools more regularly with everyday teaching through multi-component interventions.

Exploring the Relationship Between Usage and Outcomes of an Internet-Based Intervention for Individuals With Depressive Symptoms: Secondary Analysis of Data From a Randomized Controlled Trial

Background

Internet interventions can easily generate objective data about program usage. Increasingly, more studies explore the relationship between usage and outcomes, but they often report different metrics of use, and the findings are mixed. Thus, current evaluations fail to demonstrate which metrics should be considered and how these metrics are related to clinically meaningful change.

Objective

This study aimed to explore the relationship between several usage metrics and outcomes of an internet-based intervention for depression.

Methods

This is a secondary analysis of data from a randomized controlled trial that examined the efficacy of an internet-based cognitive behavioral therapy for depression (Space from Depression) in an adult community sample. All participants who enrolled in the intervention, regardless of meeting the inclusion criteria, were included in this study. Space from Depression is a 7-module supported intervention, delivered over a period of 8 weeks. Different usage metrics (ie, time spent, modules and activities completed, and percentage of program completion) were automatically collected by the platform, and composite variables from these (eg, activities per session) were computed. A breakdown of the usage metrics was obtained by weeks. For the analysis, the sample was divided into those who obtained a reliable change (RC)—and those who did not.

Results

Data from 216 users who completed pre- and posttreatment outcomes were included in the analyses. A total of 89 participants obtained an RC, and 127 participants did not obtain an RC. Those in the RC group significantly spent more time, had more log-ins, used more tools, viewed a higher percentage of the program, and got more reviews from their supporter compared with those who did not obtain an RC. Differences between groups in usage were observed from the first week in advance across the different metrics, although they vanished over time. In the RC group, the usage was higher during the first 4 weeks, and then a significant decrease was observed. Our results showed that specific levels of platform usage, 7 hours total time spent, 15 sessions, 30 tools used, and 50% of program completion, were associated with RC.

Conclusions

Overall, the results showed that those individuals who obtained an RC after the intervention had higher levels of exposure to the platform. The usage during the first half of the intervention was higher, and differences between groups were observed from the first week. This study also showed specific usage levels associated with outcomes that could be tested in controlled studies to inform the minimal usage to establish adherence. These results will help to better understand how to use internet-based interventions and what optimal level of engagement can most affect outcomes.

Trial Registration

ISRCTN Registry ISRCTN03704676; http://www.isrctn.com/ISRCTN03704676

International Registered Report Identifier (IRRID)

RR2-10.1186/1471-244X-14-147

Unraveling the Black Box: Exploring Usage Patterns of a Blended Treatment for Depression in a Multicenter Study

BACKGROUND

Blended treatments, combining digital components with face-to-face (FTF) therapy, are starting to find their way into mental health care. Knowledge on how blended treatments should be set up is, however, still limited. To further explore and optimize blended treatment protocols, it is important to obtain a full picture of what actually happens during treatments when applied in routine mental health care.

OBJECTIVE

The aims of this study were to gain insight into the usage of the different components of a blended cognitive behavioral therapy (bCBT) for depression and reflect on actual engagement as compared with intended application, compare bCBT usage between primary and specialized care, and explore different usage patterns.

METHODS

Data used were collected from participants of the European Comparative Effectiveness Research on Internet-Based Depression Treatment project, a European multisite randomized controlled trial comparing bCBT with regular care for depression. Patients were recruited in primary and specialized routine mental health care settings between February 2015 and December 2017. Analyses were performed on the group of participants allocated to the bCBT condition who made use of the Moodbuster platform and for whom data from all blended components were available (n=200). Included patients were from Germany, Poland, the Netherlands, and France; 64.5% (129/200) were female and the average age was 42 years (range 18-74 years).

RESULTS

Overall, there was a large variability in the usage of the blended treatment. A clear distinction between care settings was observed, with longer treatment duration and more FTF sessions in specialized care and a more active and intensive usage of the Web-based component by the patients in primary care. Of the patients who started the bCBT, 89.5% (179/200) also continued with this treatment format. Treatment preference, educational level, and the number of comorbid disorders were associated with bCBT engagement.

CONCLUSIONS

Blended treatments can be applied to a group of patients being treated for depression in routine mental health care. Rather than striving for an optimal blend, a more personalized blended care approach seems to be the most suitable. The next step is to gain more insight into the clinical and cost-effectiveness of blended treatments and to further facilitate uptake in routine mental health care.

A multi-faceted approach to characterizing user behavior and experience in a digital mental health intervention

Digital interventions offer great promise for supporting health-related behavior change. However, there is much that we have yet to learn about how people respond to them. In this study, we present a novel mixed-methods approach to analysis of the complex and rich data that digital interventions collect. We perform secondary analysis of IntelliCare, an intervention in which participants are able to try 14 different mental health apps over the course of eight weeks. The goal of our analysis is to characterize users' app use behavior and experiences, and is rooted in theoretical conceptualizations of engagement as both usage and user experience. In the first aim, we employ cluster analysis to identify subgroups of participants that share similarities in terms of the frequency of their usage of particular apps, and then employ other engagement measures to compare the clusters. We identified four clusters with different app usage patterns: Low Usage, High Usage, Daily Feats Users, and Day to Day users. Each cluster was distinguished by its overall frequency of app use, or the main app that participants used. In the second aim, we developed a computer-assisted text analysis and visualization method - message highlighting - to facilitate comparison of the clusters. Last, we performed a qualitative analysis using participant messages to better understand the mechanisms of change and usability of salient apps from the cluster analysis. Our novel approach, integrating text and visual analytics with more traditional qualitative analysis techniques, can be used to generate insights concerning the behavior and experience of users in digital health contexts, for subsequent personalization and to identify areas for improvement of intervention technologies.

A Framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) in Digital Interventions: Viewpoint

Trials of digital interventions can yield extensive, in-depth usage data, yet usage analyses tend to focus on broad descriptive summaries of how an intervention has been used by the whole sample. This paper proposes a novel framework to guide systematic, fine-grained usage analyses that better enables understanding of how an intervention works, when, and for whom. The framework comprises three stages to assist in the following: (1) familiarization with the intervention and its relationship to the captured data, (2) identification of meaningful measures of usage and specifying research questions to guide systematic analyses of usage data, and (3) preparation of datasheets and consideration of available analytical methods with which to examine the data. The framework can be applied to inform data capture during the development of a digital intervention and/or in the analysis of data after the completion of an evaluation trial. We will demonstrate how the framework shaped preparation and aided efficient data capture for a digital intervention to lower transmission of cold and flu viruses in the home, as well as how it informed a systematic, in-depth analysis of usage data collected from a separate digital intervention designed to promote self-management of colds and flu. The Analyzing and Measuring Usage and Engagement Data (AMUsED) framework guides systematic and efficient in-depth usage analyses that will support standardized reporting with transparent and replicable findings. These detailed findings may also enable examination of what constitutes effective engagement with particular interventions.

Measuring the Implementation of Behavioral Intervention Technologies: Recharacterization of Established Outcomes

Behavioral intervention technologies (BITs) are websites, software, mobile apps, and sensors designed to help users address or change behaviors, cognitions, and emotional states. BITs have the potential to transform health care delivery, and early research has produced promising findings of efficacy. BITs also favor new models of health care delivery and provide novel data sources for measurement. However, there are few examples of successful BIT implementation and a lack of consensus on as well as inadequate descriptions of BIT implementation measurement. The aim of this viewpoint paper is to provide an overview and characterization of implementation outcomes for the study of BIT use in routine practice settings. Eight outcomes for the evaluation of implementation have been previously described: acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability. In a proposed recharacterization of these outcomes with respect to BIT implementation, definitions are clarified, expansions to the level of analysis are identified, and unique measurement characteristics are discussed. Differences between BIT development and implementation, an increased focus on consumer-level outcomes, the expansion of providers who support BIT use, and the blending of BITs with traditional health care services are specifically discussed. BITs have the potential to transform health care delivery. Realizing this potential, however, will hinge on high-quality research that consistently and accurately measures how well such technologies have been integrated into health services. This overview and characterization of implementation outcomes support BIT research by identifying and proposing solutions for key theoretical and practical measurement challenges.

A Library of Analytic Indicators to Evaluate Effective Engagement with Consumer mHealth Apps for Chronic Conditions: Scoping Review

Background

There is mixed evidence to support current ambitions for mobile health (mHealth) apps to improve chronic health and well-being. One proposed explanation for this variable effect is that users do not engage with apps as intended. The application of analytics, defined as the use of data to generate new insights, is an emerging approach to study and interpret engagement with mHealth interventions.

Objective

This study aimed to consolidate how analytic indicators of engagement have previously been applied across clinical and technological contexts, to inform how they might be optimally applied in future evaluations.

Methods

We conducted a scoping review to catalog the range of analytic indicators being used in evaluations of consumer mHealth apps for chronic conditions. We categorized studies according to app structure and application of engagement data and calculated descriptive data for each category. Chi-square and Fisher exact tests of independence were applied to calculate differences between coded variables.

Results

A total of 41 studies met our inclusion criteria. The average mHealth evaluation included for review was a two-group pretest-posttest randomized controlled trial of a hybrid-structured app for mental health self-management, had 103 participants, lasted 5 months, did not provide access to health care provider services, measured 3 analytic indicators of engagement, segmented users based on engagement data, applied engagement data for descriptive analyses, and did not report on attrition. Across the reviewed studies, engagement was measured using the following 7 analytic indicators: the number of measures recorded (76%, 31/41), the frequency of interactions logged (73%, 30/41), the number of features accessed (49%, 20/41), the number of log-ins or sessions logged (46%, 19/41), the number of modules or lessons started or completed (29%, 12/41), time spent engaging with the app (27%, 11/41), and the number or content of pages accessed (17%, 7/41). Engagement with unstructured apps was mostly measured by the number of features accessed (8/10, P=.04), and engagement with hybrid apps was mostly measured by the number of measures recorded (21/24, P=.03). A total of 24 studies presented, described, or summarized the data generated from applying analytic indicators to measure engagement. The remaining 17 studies used or planned to use these data to infer a relationship between engagement patterns and intended outcomes.

Conclusions

Although researchers measured on average 3 indicators in a single study, the majority reported findings descriptively and did not further investigate how engagement with an app contributed to its impact on health and well-being. Researchers are gaining nuanced insights into engagement but are not yet characterizing effective engagement for improved outcomes. Raising the standard of mHealth app efficacy through measuring analytic indicators of engagement may enable greater confidence in the causal impact of apps on improved chronic health and well-being.

Developing Typologies of User Engagement With the BRANCH Alcohol-Harm Reduction Smartphone App: Qualitative Study

BACKGROUND

Understanding how users engage with electronic screening and brief intervention (eSBI) is a critical research objective to improve effectiveness of app-based interventions to reduce harmful alcohol consumption. Although quantitative measures of engagement provide a strong indicator of how the user engages with an app at the group level, they do not elucidate finer-grained details of how apps function from an individual, experiential perspective and why, or how, users engage with an intervention in a particular manner.

OBJECTIVE

The aim of this study was to (1) understand why and how participants engaged with the BRANCH app, (2) explore facilitators and barriers to engagement with app features, (3) explore how the BRANCH app impacted drinking behavior, (4) use these data to identify typologies of users of the BRANCH app in terms of engagement behaviors, and (5) identify future eSBI app design implications.

METHODS

In total, 20 one-to-one semistructured telephone interviews were conducted with participants recruited from a randomized controlled trial, which evaluated the effectiveness of engagement-promoting strategies in the BRANCH app targeting harmful drinking in young adults (aged 18-30 years). The topic guide explored users' current engagement levels with existing health promotion apps, their views toward the effectiveness of such apps, and what they liked and disliked about BRANCH, specifically focusing on how they engaged with the app. Framework analysis was used to develop typologies of user engagement.

RESULTS

The study identified 3 typologies of engagers. Trackers were defined by their motivations to use health-tracking apps to monitor and understand quantified self-data. They did not have intentions necessarily to cut down and predominantly used only the drinking diary. Cut-downers were motivated to use the app because they wanted to reduce their alcohol consumption Unlike Trackers, they did not use a range of different health apps daily, but saw the BRANCH app as an opportunity to test out a different method of trying to cut down their alcohol use. This typology used more features than Trackers, such as the goal setting function. Noncommitters were characterized as a group of users who were initially enthusiastic about using the app; however, this enthusiasm quickly waned and they gained no benefit from it.

CONCLUSIONS

This was the first study to identify typologies of user engagement with eSBI apps. Although in need of replication, it provides a first step in understanding independent categories of eSBI users, who may benefit from apps tailored to a user's typology or motivation. It also provides new evidence to suggest that apps may be used more effectively as a tool to raise awareness of drinking, instead of reducing alcohol use, and be a step in the care pathway, identifying at-risk individuals and signposting them to more intensive treatment.

TRIAL REGISTRATION

International Standard Randomised Controlled Trial Number ISRCTN70980706; http://www.isrctn.com /ISRCTN70980706 (Archived by WebCite at http://www.webcitation.org/73vfDXYEZ).

Promoting Psychological Well-Being at Work by Reducing Stress and Improving Sleep: Mixed-Methods Analysis

Background

Workplace programs designed to improve the health and psychological well-being of employees are becoming increasingly popular. However, there are mixed reports regarding the effectiveness of such programs and little analysis of what helps people to engage with such programs.

Objective

This evaluation of a particularly broad, team-based, digital health and well-being program uses mixed methods to identify the elements of the program that reduce work stress and promote psychological well-being, sleep quality, and productivity of employees.

Methods

Participation in the Virgin Pulse Global Challenge program during May to September 2016 was studied. Self-reported stress, sleep quality, productivity, and psychological well-being data were collected both pre- and postprogram. Participant experience data were collected through a third final survey. However, the response rates for the last 2 surveys were only 48% and 10%, respectively. A random forest was used to estimate the probability of the completion of the last 2 surveys based on the preprogram assessment data and the demographic data for the entire sample (N=178,350). The inverse of these estimated probabilities were used as weights in hierarchical linear models in an attempt to address any estimation bias caused by the low response rates. These linear models described changes in psychological well-being, stress, sleep, and productivity over the duration of the program in relation to gender and age, engagement with each of the modules, each of the program features, and participant descriptions of the Virgin Pulse Global Challenge. A 0.1% significance level was used due to the large sample size for the final survey (N=18,653).

Results

The final analysis suggested that the program is more beneficial for older people, with 2.9% greater psychological well-being improvements observed on average in the case of women than men (P<.001). With one exception, all the program modules contributed significantly to the outcome measures with the following average improvements observed: psychological well-being, 4.1%-6.0%; quality of sleep, 3.2%-6.9%; work-related stress, 1.7%-6.8%; and productivity, 1.9%-4.2%. However, only 4 of the program features were found to have significant associations with the outcome measures with the following average improvements observed: psychological well-being, 3.7%-5.6%; quality of sleep, 3.4%-6.5%; work-related stress, 4.1%-6.4%; and productivity, 1.6%-3.2%. Finally, descriptions of the Virgin Pulse Global Challenge produced 5 text topics that were related to the outcome measures. Healthy lifestyle descriptions showed a positive association with outcomes, whereas physical activity and step count tracking descriptions showed a negative association with outcomes.

Conclusions

The complementary use of qualitative and quantitative survey data in a mixed-methods analysis provided rich information that will inform the development of this and other programs designed to improve employee health. However, the low response rates and the lack of a control group are limitations, despite the attempts to address these problems in the analysis.

Content-Sensitive Characterization of Peer Interactions of Highly Engaged Users in an Online Community for Smoking Cessation: Mixed-Methods Approach for Modeling User Engagement in Health Promotion Interventions

Background

Online communities provide affordable venues for behavior change. However, active user engagement holds the key to the success of these platforms. In order to enhance user engagement and in turn, health outcomes, it is essential to offer targeted interventional and informational support.

Objective

In this paper, we describe a content plus frequency framework to enable the characterization of highly engaged users in online communities and study theoretical techniques employed by these users through analysis of exchanged communication.

Methods

We applied the proposed methodology for analysis of peer interactions within QuitNet, an online community for smoking cessation. Firstly, we identified 144 highly engaged users based on communication frequency within QuitNet over a period of 16 years. Secondly, we used the taxonomy of behavior change techniques, text analysis methods from distributional semantics, machine learning, and sentiment analysis to assign theory-driven labels to content. Finally, we extracted content-specific insights from peer interactions (n=159,483 messages) among highly engaged QuitNet users.

Results

Studying user engagement using our proposed framework led to the definition of 3 user categories—conversation initiators, conversation attractors, and frequent posters. Specific behavior change techniques employed by top tier users (threshold set at top 3) within these 3 user groups were found to be goal setting, social support, rewards and threat, and comparison of outcomes. Engagement-specific trends within sentiment manifestations were also identified.

Conclusions

Use of content-inclusive analytics has offered deep insight into specific behavior change techniques employed by highly engaged users within QuitNet. Implications for personalization and active user engagement are discussed.

A randomised controlled trial of the effect of providing online risk information and lifestyle advice for the most common preventable cancers: study protocol

BACKGROUND

Cancer is a leading cause of mortality and morbidity worldwide. Prevention is recognised by many, including the World Health Organization, to offer the most cost-effective long-term strategy for the control of cancer. One approach that focuses on individuals is the provision of personalised risk information. However, whether such information motivates behaviour change and whether the effect is different with varying formats of risk presentation is unclear. We aim to assess the short-term effect of providing information about personalised risk of cancer in three different formats alongside lifestyle advice on health-related behaviours, risk perception and risk conviction.

METHODS

In a parallel group, randomised controlled trial 1000 participants will be recruited through the online platform Prolific. Participants will be allocated to either a control group receiving cancer-specific lifestyle advice alone or one of three intervention groups receiving the same lifestyle advice alongside their estimated 10-year risk of developing one of the five most common preventable cancers, calculated from self-reported modifiable behavioural risk factors, in one of three different formats (bar chart, pictograph or qualitative scale). The primary outcome is change from baseline in computed risk relative to an individual with a recommended lifestyle at three months. Secondary outcomes include: perceived risk of cancer; anxiety; cancer-related worry; intention to change behaviour; and awareness of cancer risk factors.

DISCUSSION

This study will provide evidence on the short-term effect of providing online information about personalised risk of cancer alongside lifestyle advice on risk perception and health-related behaviours and inform the development of interventions.

TRIAL REGISTRATION

ISRCTN17450583. Registered 30 January 2018.

Comparing usage of a web and app stress management intervention: An observational study

Choices in the design and delivery of digital health behaviour interventions may have a direct influence on subsequent usage and engagement. Few studies have been able to make direct, detailed comparisons of differences in usage between interventions that are delivered via web or app. This study compared the usage of two versions of a digital stress management intervention, one delivered via a website (Healthy Paths) and the other delivered via an app (Healthy Mind). Design modifications were introduced within Healthy Mind to take account of reported differences in how individuals engage with websites compared to apps and mobile phones. Data were collected as part of an observational study nested within a broader exploratory trial of Healthy Mind. Objective usage of Healthy Paths and Healthy Mind were automatically recorded, including frequency and duration of logins, access to specific components within the intervention and order of page/screen visits. Usage was compared for a two week period following initial registration. In total, 381 participants completed the registration process for Healthy Paths (web) and 162 participants completed the registration process for Healthy Mind (app). App users logged in twice as often (Mdn = 2.00) as web users (Mdn = 1.00), U = 13,059.50, p ≤ 0.001, but spent half as much time (Mdn = 5.23 min) on the intervention compared to web users (Mdn = 10.52 min), U = 19,740.00, p ≤ 0.001. Visual exploration of usage patterns over time revealed that a significantly higher proportion of app users (n = 126, 82.35%) accessed both types of support available within the intervention (i.e. awareness and change-focused tools) compared to web users (n = 92, 40.17%), χ2(1, n = 382) = 66.60, p < 0.001. This study suggests that the digital platform used to deliver an intervention (i.e. web versus app) and specific design choices (e.g. navigation, length and volume of content) may be associated with differences in how the intervention content is used. Broad summative usage data (e.g. total time spent on the intervention) may mask important differences in how an intervention is used by different user groups if it is not complemented by more fine-grained analyses of usage patterns over time.

TRIAL REGISTRATION NUMBER

ISRCTN67177737.

Using an Analysis of Behavior Change to Inform Effective Digital Intervention Design: How Did the PRIMIT Website Change Hand Hygiene Behavior Across 8993 Users?

Background

In designing digital interventions for healthcare, it is important to understand not just whether interventions work but also how and for whom—including whether individual intervention components have different effects, whether a certain usage threshold is required to change behavior in each intervention and whether usage differs across population subgroups.

Purpose

We investigated these questions using data from a large trial of the digital PRimary care trial of a website based Infection control intervention to Modify Influenza-like illness and respiratory tract infection Transmission) (PRIMIT) intervention, which aimed to reduce respiratory tract infections (RTIs) by increasing hand hygiene behavior.

Method

Baseline and follow-up questionnaires measured behaviors, intentions and attitudes in hand hygiene. In conjunction with objective measures of usage of the four PRIMIT sessions, we analysed these observational data to examine mechanisms of behavior change in 8993 intervention users.

Results

We found that the PRIMIT intervention changed behavior, intentions and attitudes, and this change was associated with reduced RTIs. The largest hand hygiene change occurred after the first session, with incrementally smaller changes after each subsequent session, suggesting that engagement with the core behavior change techniques included in the first session was necessary and sufficient for behavior change. The intervention was equally effective for men and women, older and younger people and was particularly effective for those with lower levels of education.

Conclusions

Our well-powered analysis has implications for intervention development. We were able to determine a ‘minimum threshold’ of intervention engagement that is required for hand hygiene change, and we discuss the potential implications this (and other analyses of this type) may have for further intervention development. We also discuss the application of similar analyses to other interventions.

Factors Associated With Engagement With a Web-Based Lifestyle Intervention Following Provision of Coronary Heart Disease Risk: Mixed Methods Study

Background

Web-based interventions provide the opportunity to combine the tailored approach of face-to-face interventions with the scalability and cost-effectiveness of public health interventions. This potential is often limited by low engagement. A number of studies have described the characteristics of individuals who engage more in Web-based interventions but few have explored the reasons for these variations.

Objective

We aimed to explore individual-level factors associated with different degrees of engagement with a Web-based behavior change intervention following provision of coronary heart disease (CHD) risk information, and the barriers and facilitators to engagement.

Methods

This study involved the secondary analysis of data from the Information and Risk Modification Trial, a randomized controlled trial of a Web-based lifestyle intervention alone, or alongside information on estimated CHD risk. The intervention consisted of three interactive sessions, each lasting up to 60 minutes, delivered at monthly intervals. Participants were characterized as high engagers if they completed all three sessions. Thematic analysis of qualitative data from interviews with 37 participants was combined with quantitative data on usage of the Web-based intervention using a mixed-methods matrix, and data on the views of the intervention itself were analyzed across all participants.

Results

Thirteen participants were characterized as low engagers and 24 as high engagers. There was no difference in age (P=.75), gender (P=.95), or level of risk (P=.65) between the groups. Low engagement was more often associated with: (1) reporting a negative emotional reaction in response to the risk score (P=.029), (2) perceiving that the intervention did not provide any new lifestyle information (P=.011), and (3) being less likely to have reported feeling an obligation to complete the intervention as part of the study (P=.019). The mixed-methods matrix suggested that there was also an association between low engagement and less success with previous behavior change attempts, but the statistical evidence for this association was weak (P=.16). No associations were seen between engagement and barriers or facilitators to health behavior change, or comments about the design of the intervention itself. The most commonly cited barriers related to issues with access to the intervention itself: either difficulties remembering the link to the site or passwords, a perceived lack of flexibility within the website, or lack of time. Facilitators included the nonjudgmental presentation of lifestyle information, the use of simple language, and the personalized nature of the intervention.

Conclusions

This study shows that the level of engagement with a Web-based intervention following provision of CHD risk information is not influenced by the level of risk but by the individual’s response to the risk information, their past experiences of behavior change, the extent to which they consider the lifestyle information helpful, and whether they felt obliged to complete the intervention as part of a research study. A number of facilitators and barriers to Web-based interventions were also identified, which should inform future interventions.

The implementation of web-based cognitive rehabilitation in adult cancer survivors: examining participant engagement, attrition and treatment fidelity

PURPOSE

Low engagement and high attrition are common challenges in web-based interventions. Typical measures of engagement reported in the literature are not meaningful for describing participant activity within the intervention and can be misleading. This research aimed to develop a more meaningful method of measuring engagement in an online cognitive rehabilitation program whilst monitoring treatment fidelity.

METHODS

A pilot study and randomised controlled trial (RCT) were conducted. Data from 60 participants were analysed from three intervention groups: pilot cancer group, pilot non-cancer group and RCT cancer group. Groups completed the 4-week eReCog program comprised of four online modules. Engagement scores were calculated based on activities completed in each module. Attrition, interaction with the program facilitator and correlations with outcome measures were analysed.

RESULTS

Overall engagement in the intervention was high. The non-cancer group participated significantly less than the cancer groups (p = < 0.001), whereby the percentage of activity items completed was 92, 87 and 78% in the pilot cancer, RCT cancer and pilot non-cancer groups, respectively. Attrition was higher in the pilot non-cancer group (24%) compared to the pilot cancer group (8%) and the RCT cancer group (16%). Total engagement was correlated with fewer prospective memory problems on instrumental activities of daily living (p = 0.018).

CONCLUSIONS

Measuring completed activities in online interventions appears a more meaningful measure of engagement than other conventional methods described in the literature and has the potential to increase treatment fidelity in web-based research.

Addressing engagement in technology-based behavioural HIV interventions through paradata metrics

PURPOSE OF REVIEW

The goal of this review was to examine how often researchers report participants' online engagement using paradata (i.e. intervention usage metrics) when describing the outcomes of online behavioural HIV prevention and care interventions. We also highlight the utility of paradata collection and analysis in future technology-based trials.

RECENT FINDINGS

We focused on studies indexed on PubMed and published between 1 January 2016 and 31 March 2017 that reported the development and testing of online behavioural interventions for HIV prevention and/or care. Of the 705 extracted citations, six met study criteria.

SUMMARY

Only one study reported paradata reflecting participants' engagement with a technology-based intervention. Researchers should systematically collect and analyse paradata to strengthen the evidence base for technology-based interventions (do they work?), advance the use of behaviour change theory across modalities and platforms (how/why do they work?) and inform reach and scale-up efforts (for whom do they work?). Researchers may also rely on paradata to examine dose-response relationships due to user engagement, to identify replicable core components linked to behaviour change outcomes, to allocate resources judiciously and drive down development costs, and to pool these metrics for use in future meta-analyses.

Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis

"Engagement" with digital behaviour change interventions (DBCIs) is considered important for their effectiveness. Evaluating engagement is therefore a priority; however, a shared understanding of how to usefully conceptualise engagement is lacking. This review aimed to synthesise literature on engagement to identify key conceptualisations and to develop an integrative conceptual framework involving potential direct and indirect influences on engagement and relationships between engagement and intervention effectiveness. Four electronic databases (Ovid MEDLINE, PsycINFO, ISI Web of Knowledge, ScienceDirect) were searched in November 2015. We identified 117 articles that met the inclusion criteria: studies employing experimental or non-experimental designs with adult participants explicitly or implicitly referring to engagement with DBCIs, digital games or technology. Data were synthesised using principles from critical interpretive synthesis. Engagement with DBCIs is conceptualised in terms of both experiential and behavioural aspects. A conceptual framework is proposed in which engagement with a DBCI is influenced by the DBCI itself (content and delivery), the context (the setting in which the DBCI is used and the population using it) and the behaviour that the DBCI is targeting. The context and "mechanisms of action" may moderate the influence of the DBCI on engagement. Engagement, in turn, moderates the influence of the DBCI on those mechanisms of action. In the research literature, engagement with DBCIs has been conceptualised in terms of both experience and behaviour and sits within a complex system involving the DBCI, the context of use, the mechanisms of action of the DBCI and the target behaviour.

Gamification and Adherence to Web-Based Mental Health Interventions: A Systematic Review

BACKGROUND

Adherence to effective Web-based interventions for common mental disorders (CMDs) and well-being remains a critical issue, with clear potential to increase effectiveness. Continued identification and examination of "active" technological components within Web-based interventions has been called for. Gamification is the use of game design elements and features in nongame contexts. Health and lifestyle interventions have implemented a variety of game features in their design in an effort to encourage engagement and increase program adherence. The potential influence of gamification on program adherence has not been examined in the context of Web-based interventions designed to manage CMDs and well-being.

OBJECTIVE

This study seeks to review the literature to examine whether gaming features predict or influence reported rates of program adherence in Web-based interventions designed to manage CMDs and well-being.

METHODS

A systematic review was conducted of peer-reviewed randomized controlled trials (RCTs) designed to manage CMDs or well-being and incorporated gamification features. Seven electronic databases were searched.

RESULTS

A total of 61 RCTs met the inclusion criteria and 47 different intervention programs were identified. The majority were designed to manage depression using cognitive behavioral therapy. Eight of 10 popular gamification features reviewed were in use. The majority of studies utilized only one gamification feature (n=58) with a maximum of three features. The most commonly used feature was story/theme. Levels and game leaders were not used in this context. No studies explicitly examined the role of gamification features on program adherence. Usage data were not commonly reported. Interventions intended to be 10 weeks in duration had higher mean adherence than those intended to be 6 or 8 weeks in duration.

CONCLUSIONS

Gamification features have been incorporated into the design of interventions designed to treat CMD and well-being. Further research is needed to improve understanding of gamification features on adherence and engagement in order to inform the design of future Web-based health interventions in which adherence to treatment is of concern. Conclusions were limited by varied reporting of adherence and usage data.

Using mixed methods evaluation to assess the feasibility of online clinical training in evidence based interventions: a case study of cognitive behavioural treatment for low back pain

BACKGROUND

Cognitive behavioural (CB) approaches are effective in the management of non-specific low back pain (LBP). We developed the CB Back Skills Training programme (BeST) and previously provided evidence of clinical and cost effectiveness in a large pragmatic trial. However, practice change is challenged by a lack of treatment guidance and training for clinicians. We aimed to explore the feasibility and acceptability of an online programme (iBeST) for providing training in a CB approach.

METHODS

This mixed methods study comprised an individually randomised controlled trial of 35 physiotherapists and an interview study of 8 physiotherapists. Participants were recruited from 8 National Health Service departments in England and allocated by a computer generated randomisation list to receive iBeST (n = 16) or a face-to-face workshop (n = 19). Knowledge (of a CB approach), clinical skills (unblinded assessment of CB skills in practice), self-efficacy (reported confidence in using new skills), attitudes (towards LBP management), and satisfaction were assessed after training. Engagement with iBeST was assessed with user analytics. Interviews explored acceptability and experiences with iBeST. Data sets were analysed independently and jointly interpreted.

RESULTS

Fifteen (94 %) participants in the iBeST group and 16 (84 %) participants in the workshop group provided data immediately after training. We observed similar scores on knowledge (MD (95 % CI): 0.97 (-1.33, 3.26)), and self-efficacy to deliver the majority of the programme (MD (95 % CI) 0.25 (-1.7; 0.7)). However, the workshop group showed greater reduction in biomedical attitudes to LBP management (MD (95 % CI): -7.43 (-10.97, -3.89)). Clinical skills were assessed in 5 (33 %) iBeST participants and 7 (38 %) workshop participants within 6 months of training and were similar between groups (MD (95 % CI): 0.17(-0.2; 0.54)). Interviews highlighted that while initially sceptical, participants found iBeST acceptable. A number of strategies were identified to enhance future versions of iBeST such as including more skills practice.

CONCLUSIONS

Combined quantitative and qualitative data indicated that online training was an acceptable and promising method for providing training in an evidence based complex intervention. With future enhancement, the potential reach of this training method may facilitate evidence-based practice through large scale upskilling of the workforce.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN82203145 (registered prospectively on 03.09.2012).

Initial Field Trial of a Coach-Supported Web-Based Depression Treatment

Early web-based depression treatments were often self-guided and included few interactive elements, instead focusing mostly on delivering informational content online. Newer programs include many more types of features. As such, trials should analyze the ways in which people use these sites in order to inform the design of subsequent sites and models of support. The current study describes of a field trial consisting of 9 patients with major depressive disorder who completed a 12-week program including weekly coach calls. Patients usage varied widely, however, patients who formed regular patterns tended to persist with the program for the longest. Future sites might be able to facilitate user engagement by designing features to support regular use and to use coaches to help establish patterns to increase long-term use and benefit.

Usability and Acceptability of ASSESS MS: Assessment of Motor Dysfunction in Multiple Sclerosis Using Depth-Sensing Computer Vision

Background

Sensor-based recordings of human movements are becoming increasingly important for the assessment of motor symptoms in neurological disorders beyond rehabilitative purposes. ASSESS MS is a movement recording and analysis system being developed to automate the classification of motor dysfunction in patients with multiple sclerosis (MS) using depth-sensing computer vision. It aims to provide a more consistent and finer-grained measurement of motor dysfunction than currently possible.

Objective

To test the usability and acceptability of ASSESS MS with health professionals and patients with MS.

Methods

A prospective, mixed-methods study was carried out at 3 centers. After a 1-hour training session, a convenience sample of 12 health professionals (6 neurologists and 6 nurses) used ASSESS MS to capture recordings of standardized movements performed by 51 volunteer patients. Metrics for effectiveness, efficiency, and acceptability were defined and used to analyze data captured by ASSESS MS, video recordings of each examination, feedback questionnaires, and follow-up interviews.

Results

All health professionals were able to complete recordings using ASSESS MS, achieving high levels of standardization on 3 of 4 metrics (movement performance, lateral positioning, and clear camera view but not distance positioning). Results were unaffected by patients’ level of physical or cognitive disability. ASSESS MS was perceived as easy to use by both patients and health professionals with high scores on the Likert-scale questions and positive interview commentary. ASSESS MS was highly acceptable to patients on all dimensions considered, including attitudes to future use, interaction (with health professionals), and overall perceptions of ASSESS MS. Health professionals also accepted ASSESS MS, but with greater ambivalence arising from the need to alter patient interaction styles. There was little variation in results across participating centers, and no differences between neurologists and nurses.

Conclusions

In typical clinical settings, ASSESS MS is usable and acceptable to both patients and health professionals, generating data of a quality suitable for clinical analysis. An iterative design process appears to have been successful in accounting for factors that permit ASSESS MS to be used by a range of health professionals in new settings with minimal training. The study shows the potential of shifting ubiquitous sensing technologies from research into the clinic through a design approach that gives appropriate attention to the clinic environment.

A Visualization Tool to Analyse Usage of Web-Based Interventions: The Example of Positive Online Weight Reduction (POWeR)

Background

Attrition is a significant problem in Web-based interventions. Consequently, this research aims to identify the relation between Web usage and benefit from such interventions. A visualization tool has been developed that enables researchers to more easily examine large datasets on intervention usage that can be difficult to make sense of using traditional descriptive or statistical techniques alone.

Objective

This paper demonstrates how the visualization tool was used to explore patterns in participants’ use of a Web-based weight management intervention, termed "positive online weight reduction (POWeR)." We also demonstrate how the visualization tool can be used to perform subsequent statistical analyses of the association between usage patterns, participant characteristics, and intervention outcome.

Methods

The visualization tool was used to analyze data from 132 participants who had accessed at least one session of the POWeR intervention.

Results

There was a drop in usage of optional sessions after participants had accessed the initial, core POWeR sessions, but many users nevertheless continued to complete goal and weight reviews. The POWeR tools relating to the food diary and steps diary were reused most often. Differences in participant characteristics and usage of other intervention components were identified between participants who did and did not choose to access optional POWeR sessions (in addition to the initial core sessions) or reuse the food and steps diaries. Reuse of the steps diary and the getting support tools was associated with greater weight loss.

Conclusions

The visualization tool provided a quick and efficient method for exploring patterns of Web usage, which enabled further analyses of whether different usage patterns were associated with participant characteristics or differences in intervention outcome. Further usage of visualization techniques is recommended to (1) make sense of large datasets more quickly and efficiently; (2) determine the likely active ingredients in Web-based interventions, and thereby enhance the benefit they may provide; and (3) guide in designing (or redesigning) of future interventions to promote greater use and engagement by enabling users to easily access valued intervention content/tools.

Trial Registration

International Standard Randomized Controlled Trial Number (ISRCTN): 31685626; http://www.isrctn.com/ISRCTN31685626 (Archived by WebCite at http://www.webcitation.org/6YXYIw9vc).