Usability, learnability and performance evaluation of Intelligent Research and Intervention Software: A delivery platform for eHealth interventions

A randomized controlled trial evaluating the effect of an internet-based cognitive-behavioral program on anxiety symptoms in a community-based sample of adolescents

Adolescents' use of online resources to self-manage anxiety is growing. The objective of the current trial was to assess the effectiveness of an online, primarily self-led cognitive behavioral therapy (CBT) program in reducing anxiety symptoms compared to an active comparator, access to anxiety resources on a static website. A total of 563 adolescents (13-19 years) with self-identified anxiety concerns were enrolled. Self-reported anxiety symptoms were assessed pre- and post-intervention (6 weeks). Adolescents were further assessed 3 months post-intervention. Other outcomes assessed at the three time-points were quality of life (QOL) and healthcare utilization. Both interventions reduced anxiety symptoms after use. Group differences in symptom change were not significant post-intervention (p = 0.16), but were at 3 months (favouring online CBT; p = 0.04) with male participants reporting more symptom change (p = 0.03). Across time-points, as anxiety symptoms decreased, QOL increased (p < 0.001). Among participants that provided healthcare utilization before and after intervention use, the greatest changes in use were among online CBT users particularly for mental health provider visits (psychiatrist, -41.0 % vs. +18.5 %; social worker, -42.5 % vs. -22.1 %), hospital-based care (emergency department visits, -80.0 % vs. +79.4 %; hospital admissions, -76.1 % vs. +42.9 %), and use of self-help or alternative treatments (-60.0 % vs. +6.6 %). Results suggest that, over time, use of online CBT by adolescents can result in improved anxiety symptoms and fewer use of other healthcare resources compared to traditional online information seeking.

Improving COVID-19 Research of University Hospitals in Germany: Formative Usability Evaluation of the CODEX Feasibility Portal

BACKGROUND

 Within the German "Network University Medicine," a portal is to be developed to enable researchers to query on novel coronavirus disease 2019 (COVID-19) data from university hospitals for assessing the feasibility of a clinical study.

OBJECTIVES

 The usability of a prototype for federated feasibility queries was evaluated to identify design strengths and weaknesses and derive improvement recommendations for further development.

METHODS

 In the course of a remote usability test with the thinking-aloud method and posttask interviews, 15 clinical researchers evaluated the usability of a prototype of the Feasibility Portal. The identified usability problems were rated according to severity, and improvement recommendations were derived.

RESULTS

 The design of the prototype was rated as simple, intuitive, and as usable with little effort. The usability test reported a total of 26 problems, 8 of these were rated as "critical." Usability problems and revision recommendations focus primarily on improving the visual distinguishability of selected inclusion and exclusion criteria, enabling a flexible approach to criteria linking, and enhancing the free-text search.

CONCLUSION

 Improvement proposals were developed for these user problems which will guide further development and the adaptation of the portal to user needs. This is an important prerequisite for correct and efficient use in everyday clinical work in the future. Results can provide developers of similar systems with a good starting point for interface conceptualizations. The methodological approach/the developed test guideline can serve as a template for similar evaluations.

An Internet-Based Cognitive Behavioral Program for Adolescents With Anxiety: Pilot Randomized Controlled Trial

BACKGROUND

Internet-based cognitive behavioral therapy (ICBT) is a treatment approach recently developed and studied to provide frontline treatment to adolescents with anxiety disorders.

OBJECTIVE

This study aimed to pilot procedures and obtain data on methodological processes and intervention satisfaction to determine the feasibility of a definitive randomized controlled trial (RCT) to test the effectiveness of a self-managed ICBT program, Breathe (Being Real, Easing Anxiety: Tools Helping Electronically), for adolescents with anxiety concerns.

METHODS

This study employed a two-arm, multisite, pilot RCT. Adolescents aged 13 to 17 years with a self-identified anxiety concern were recruited online from health care settings and school-based mental health care services across Canada between April 2014 and May 2016. We compared 8 weeks of ICBT with ad hoc telephone and email support (Breathe experimental group) to access to a static webpage listing anxiety resources (control group). The primary outcome was the change in self-reported anxiety from baseline to 8 weeks (posttreatment), which was used to determine the sample size for a definitive RCT. Secondary outcomes were recruitment and retention rates, a minimal clinically important difference (MCID) for the primary outcome, intervention acceptability and satisfaction, use of cointerventions, and health care resource use, including a cost-consequence analysis.

RESULTS

Of the 588 adolescents screened, 94 were eligible and enrolled in the study (49 adolescents were allocated to Breathe and 45 were allocated to the control group). Analysis was based on 74% (70/94) of adolescents who completed baseline measures and progressed through the study. Enrolled adolescents were, on average, 15.3 years old (SD 1.2) and female (63/70, 90%). Retention rates at 8 weeks were 28% (13/46; Breathe group) and 58% (24/43; control group). Overall, 39% (14/36) of adolescents provided feedback on completion of the Breathe program. Adolescents' scores on a satisfaction survey indicated a moderate level of satisfaction. All but one adolescent indicated that Breathe was easy to use and they understood all the material presented. The most frequent barrier identified for program completion was difficulty in completing exposure activities. The power analysis indicated that 177 adolescents per group would be needed to detect a medium effect size (d=0.3) between groups in a definitive trial. Data for calculating an MCID or conducting a cost-consequence analysis were insufficient due to a low response rate at 8 weeks.

CONCLUSIONS

Adolescents were moderately satisfied with Breathe. However, program adjustments will be needed to address attrition and reduce perceived barriers to completing key aspects of the program. A definitive RCT to evaluate the effectiveness of the program is feasible if protocol adjustments are made to improve recruitment and retention to ensure timely study completion and increase the completeness of the data at each outcome measurement time point.

TRIAL REGISTRATION

ClinicalTrials.gov NCT02059226; http://clinicaltrials.gov/ct2/show/NCT02059226.

Implementation of e-mental health for depression and anxiety: A critical scoping review

The aim of this review was to scope the growth and development of implementation research of e-mental healthcare programs for anxiety and depression, the research and evaluation tools used, and the specific implementation processes and outcomes examined. A search of four electronic databases (MEDLINE, EMBASE, PsycINFO, and CINAHL) was conducted from January 2000 to January 2019. Of 33 studies identified, most (n = 28) were published in the last five years. Only 10 used an implementation framework to guide implementation or evaluation. Most studies reported on acceptability (n = 28), appropriateness (n = 23), and feasibility (n = 17). Less commonly reported implementation outcomes were fidelity (n = 10) and adoption (n = 7); with penetration (n = 4), sustainability (n = 3), and implementation cost (n = 2) being studied rarely. Of the 21 studies that used surveys to study implementation outcomes, less than half used a previously published survey (n = 9). More rigorous implementation studies, underpinned by strong theory and real-world understanding, are urgently needed.

Effects and Experiences of Families Following a Web-Based Psychosocial Intervention for Children with Functional Abdominal Pain and Their Parents: A Mixed-Methods Pilot Randomized Controlled Trial

Purpose

To evaluate post-treatment efficacy of DARWeb (online psychosocial intervention for children with functional abdominal pain) using a randomized clinical trial design and combining quantitative and qualitative data.

Patients and methods

Twenty-five families with children with FAP in the experimental group (EG: accessed to DARWeb) and 36 in the control group (CG: wait-list) were compared. Children and parents completed measures of abdominal pain severity (primary outcome), quality of life, and satisfaction. Moreover, children completed measures of depression, functional disability, catastrophizing and coping strategies; parents completed measures about parental responses to their children's pain. Families also answered open questions and were interviewed.

Results

A higher percentage of children in the EG achieved a significant clinical change in abdominal pain severity from the parents' perspective (28% in the EG vs 8.33% in the CG). There was a significantly greater reduction in pain frequency in the EG compared to the CG (both from the children's and parents' perspectives) from mixed repeated-measures analyses of variance (there was not a significant interaction in total scores of pain severity). A higher percentage of children in the EG improved in quality of life and depression compared to the CG (results from mixed methods repeated-measures analyses of variances were not significant). However, there were no differences for disability, pain catastrophizing or the coping strategies assessed from the children's perspective; neither from the parents' assessment of quality of life. There were significant interactions for parents' solicitousness responses and promotion of well behaviors in the expected directions. Families were quite satisfied with the intervention, and the qualitative results confirmed an improvement in pain and having learned important coping strategies.

Conclusion

Our results support the efficacy of our intervention, but future studies are needed with different profiles of initial severity of the pain problem, longer follow-ups, and other conditions.

A systematic review of electronic mental health interventions for Indigenous youth: Results and recommendations

Electronic health interventions involve health services delivered using the Internet and related communication technologies. These services can be particularly relevant for Indigenous populations who often have differential access to health-care services compared to general populations, especially within rural and remote areas. As the popularity of electronic health interventions grows, there is an increased need for evidence-based recommendations for the effective use of these technologies. The current study is a systematic review of peer-reviewed and available grey literature with the aim of understanding outcomes of electronic health interventions for mental health concerns among Indigenous people. Studies used electronic health technologies for substance use treatment or prevention, suicide prevention, parenting supports, goal setting and behaviour change and consultation services. Various technological platforms were used across interventions, with both novel and adapted intervention development. Most studies provided qualitative results, with fewer studies focusing on quantitative outcomes. Some preliminary results from the engagement of Indigenous individuals with electronic health services has been demonstrated, but further research is needed to confirm these results. Identified barriers and facilitators are identified from the reviewed literature. Recommendations for future development of electronic health interventions for Indigenous youth are provided.

Distance-Delivered Parent Training for Childhood Disruptive Behavior (Strongest Families™): a Randomized Controlled Trial and Economic Analysis

Disruptive behavior disorders are prevalent in youth, yet most children with disruptive behavior do not have access to timely, effective treatment. Distance-delivered service (e.g., via telephone, Internet) can overcome several barriers to care. This study tested the effectiveness of a 12-week parent training program, Strongest Families™ Parenting the Active Child, delivered via written material, skill-based videos, and telephone coaching sessions, as compared to usual care in reducing child externalizing behavior. Participants were 172 primary caregivers of a 6- to 12-year-old (29% girls; M age = 8.5 years) recruited from community children's mental health clinics. Participants were randomized to either Strongest Families™ or usual care and completed measures of child externalizing behavior, parenting practices, parent distress, and intervention services consumed at baseline and 5-, 10-, 16-, and 22-months post-baseline. Growth curve analysis showed significant reductions in externalizing behavior in both conditions over time. Improvements were significantly greater at 10 months in the Strongest Families™ condition (d = 0.43). At 22 months, however, the differences were not significant and small in magnitude (d = -0.05). The intervention decreased inconsistent discipline significantly more than usual care. Parents in both conditions showed significant reductions in distress. We also conducted a cost-effectiveness analysis to assess the value for money of the Strongest Families™ program versus usual care. Distance parent training is a promising way to increase access to, and reduce costs associated with, mental health care for families with a child with disruptive behavior.

Assessing usability of eHealth technology: A comparison of usability benchmarking instruments

BACKGROUND

It is generally assumed that usability benchmarking instruments are technology agnostic. The same methods for usability evaluations are used for digital commercial, educational, governmental and healthcare systems. However, eHealth technologies have unique characteristics. They need to support patients' health, provide treatment or monitor progress. Little research is done on the effectiveness of different benchmarks (qualitative and quantitative) within the eHealth context.

OBJECTIVES

In this study, we compared three usability benchmarking instruments (logging task performance, think aloud and the SUS, the System Usability Scale) to assess which metric is most indicative of usability in an eHealth technology. Also, we analyzed how these outcome variables (task completion, system usability score, serious and critical usability issues) interacted with the acceptance factors Perceived benefits, Usefulness and Intention to use.

METHODS

A usability evaluation protocol was set up that incorporated all three benchmarking methods. This protocol was deployed among 36 Dutch participants and across three different eHealth technologies: a gamified application for older adults (N = 19), an online tele-rehabilitation portal for healthcare professionals (N = 9), and a mobile health app for adolescents (N = 8).

RESULTS

The main finding was that task completion, compared to the SUS, had stronger correlations with usability benchmarks. Also, serious and critical issues were stronger correlated to task metrics than the SUS. With regard to acceptance factors, there were no significant differences between the three usability benchmarking instruments.

CONCLUSIONS

With this study, we took a first step in examining how to improve usability evaluations for eHealth. The results show that listing usability issues from think aloud protocols remains one of the most effective tools to explain the usability for eHealth. Using the SUS as a stand-alone usability metric for eHealth is not recommended. Preferably, the SUS should be combined with task metrics, especially task completion. We recommend to develop a usability benchmarking instrument specifically for eHealth.

Facebook ads to the rescue? Recruiting a hard to reach population into an Internet-based behavioral health intervention trial

OBJECTIVE

Facebook (FB) ads are touted as a way to facilitate recruitment of hard to reach participants into digital health research but the evidence has been mixed. This study aimed to empirically evaluate the impact and cost-effectiveness of paid ads for recruitment into a national trial testing an Internet-based, coached intervention for parents of children with Fetal Alcohol Spectrum Disorders.

METHODS

Post hoc analysis of FB ad data and Google analytics on the online trial consent site (myStudies) were conducted on 11 campaigns employing static image/text ads. Standard metrics (e.g., click through rate, cost per 1000 impressions, cost per consent) were calculated and descriptive statistics comparing FB ad engagement and enrolled participants over time were conducted.

RESULTS

Ad campaigns were active for a combined 115 days over 58 weeks resulting in 1533 links to the online recruitment site. During the ad campaigns, the mean rate of enrolment was 1 participant every 2 days. The first 3 ad campaigns were the most cost-effective. Mean cost per enrolment was $19.27 (Canadian dollars).

CONCLUSIONS

FB ads were efficient and cost-effective in broad dissemination of trial information, but more research is needed to explore the impact of saturation (how often ads are posted), design (what is in the ad), and individual determinants (who is likely to respond to an ad) on converting FB ad engagement into enrolment. Avoiding a reductionist approach to analytics will help ensure appropriate and targeted strategies remain the priority for digital health research recruitment through social media.

Pioneer Paper: An Accidental Scientist: Chance, Failure, Risk-Taking, and Mentoring

I never intended to become a scientist. My career developed on the basis of chance happenings, repeated failure, the willingness to take risks and the acceptance and provision of mentoring. My career has included periods of difficulty and shifted back and forth between academic health centers and universities in Canada. Although I have been amply recognized for my successes, my greatest learning has come from my failures. My greatest satisfaction has been in the development, evaluation and dissemination of interventions. The combination of intellectual stimulation and emotional gratification has meant a rewarding career.

Quantitative and qualitative testing of DARWeb: An online self-guided intervention for children with functional abdominal pain and their parents

The main objective of this study was to preliminary explore the effects of DARWeb on different outcomes. A Quasi-experimental, one-group, pretest-posttest design was used. Parents and children were asked to complete questionnaires and questions (separately) about quality of life, abdominal pain severity, and satisfaction. Semi-structured interviews with families were also performed. This study focuses on 17 families. Results showed that parent's ratings of children's abdominal pain severity were significantly lower after finishing the intervention and at the 3-month follow-up, and quality of life scores had increased significantly after 3 months. From children's ratings, mean abdominal pain severity scores were significantly lower after the intervention compared to the preintervention assessment. Both parents and children were quite satisfied with the intervention. In qualitative interviews, families suggested that DARWeb helped them to give less importance to pain and to learn coping strategies. In conclusion, this study showed the potential usefulness of DARWeb for children with functional abdominal pain and for their parents.

Usability and Suitability of the Omics-Integrating Analysis Platform tranSMART for Translational Research and Education

BACKGROUND

Platforms like tranSMART assist researchers in analyzing clinical and corresponding omics data. Usability is an important, yet often overlooked, factor affecting the adoption and meaningful use. Analyses on the specific needs of translational researchers and considerations about the application of such platforms for education are rare.

OBJECTIVES

The aim of this study was to test whether tranSMART can be used in education and how well medical students and professional researchers can handle it; to identify which kind of translational researchers-in terms of skills, experienced limitations, and available data-can take advantage of tranSMART; and to evaluate the usability and to generate recommendations for improvements.

METHODS

An online-based test has been done by medical students (N = 109) and researchers (N = 26). The test comprised 13 tasks in the context of four typical research scenarios based on experimental and clinical data. A web questionnaire was provided to identify both the needs and the conditions of research as well as to evaluate the system's usability based on the "System Usability Scale" (SUS).

RESULTS

Students and researchers were able to handle tranSMART well and coped with most scenarios: cohort identification, data exploration, hypothesis generation, and hypothesis validation were answered with a rate of correctness between 82 and 100%. Of the total, 72.2% of the teaching researchers considered tranSMART suitable for their lessons and 84.6% of the researchers considered the platform useful for their daily work; 65.4% of the researchers named the nonavailability of a platform like tranSMART as a restriction on their research. The usability was rated "acceptable" with a SUS of 70.8.

CONCLUSION

tranSMART is potentially suitable for education purposes and fits most of the needs of translational researchers. Improvements are needed on the presentation of analysis results and on the guidance of users through the analysis, especially to ensure the compliance of the analysis with the requirements of statistical testing.

Evaluating the parent-adolescent communication toolkit: Usability and preliminary content effectiveness of an online intervention

Aim

This study aimed to assess the Parent-Adolescent Communication Toolkit, an online intervention designed to help improve parent communication with their adolescents. Participant preferences for two module delivery systems (sequential and unrestricted module access) were identified.

Design

Usability assessment of the PACT intervention was completed using pre-test and posttest comparisons. Usability data, including participant completion and satisfaction ratings were examined.

Methods

Parents (N = 18) of adolescents were randomized to a sequential or unrestricted chapter access group. Parent participants completed pre-test measures, the PACT intervention and posttest measures. Participants provided feedback for the intervention to improve modules and provided usability ratings. Adolescent pre- and posttest ratings were evaluated.

Results

Usability ratings were high and parent feedback was positive. The sequential module access groups rated the intervention content higher and completed more content than the unrestricted chapter access group, indicating support for the sequential access design. Parent mean posttest communication scores were significantly higher (p < .05) than pre-test scores. No significant differences were detected for adolescent participants. Findings suggest that the Parent-Adolescent Communication Toolkit has potential to improve parent-adolescent communication but further effectiveness assessment is required.

Sociotechnical Human Factors Involved in Remote Online Usability Testing of Two eHealth Interventions

Background

Research in the fields of human performance technology and human computer interaction are challenging the traditional macro focus of usability testing arguing for methods that help test moderators assess “use in context” (ie, cognitive skills, usability understood over time) and in authentic “real world” settings. Human factors in these complex test scenarios may impact on the quality of usability results being derived yet there is a lack of research detailing moderator experiences in these test environments. Most comparative research has focused on the impact of the physical environment on results, and rarely on how the sociotechnical elements of the test environment affect moderator and test user performance. Improving our understanding of moderator roles and experiences with conducting “real world” usability testing can lead to improved techniques and strategies

Objective

To understand moderator experiences of using Web-conferencing software to conduct remote usability testing of 2 eHealth interventions.

Methods

An exploratory case study approach was used to study 4 moderators’ experiences using Blackboard Collaborate for remote testing sessions of 2 different eHealth interventions. Data collection involved audio-recording iterative cycles of test sessions, collecting summary notes taken by moderators, and conducting 2 90-minute focus groups via teleconference. A direct content analysis with an inductive coding approach was used to explore personal accounts, assess the credibility of data interpretation, and generate consensus on the thematic structure of the results.

Results

Following the convergence of data from the various sources, 3 major themes were identified: (1) moderators experienced and adapted to unpredictable changes in cognitive load during testing; (2) moderators experienced challenges in creating and sustaining social presence and untangling dialogue; and (3) moderators experienced diverse technical demands, but were able to collaboratively troubleshoot with test users.

Conclusions

Results highlight important human-computer interactions and human factor qualities that impact usability testing processes. Moderators need an advanced skill and knowledge set to address the social interaction aspects of Web-based usability testing and technical aspects of conferencing software during test sessions. Findings from moderator-focused studies can inform the design of remote testing platforms and real-time usability evaluation processes that place less cognitive burden on moderators and test users.

Guided Internet-Based Parent Training for Challenging Behavior in Children With Fetal Alcohol Spectrum Disorder (Strongest Families FASD): Study Protocol for a Randomized Controlled Trial

Background

Fetal alcohol spectrum disorder (FASD) is a term used to encompass the full range of neurobehavioral and cognitive dysfunction that may occur as a consequence of prenatal alcohol exposure. There is relatively little research on intervention strategies that specifically target the behavioral problems of children with FASD. Availability and access to services are barriers to timely and effective care for families. The Strongest Families FASD intervention was recently adapted from the Strongest Families “Parenting the Active Child” program to include FASD-specific content delivered via an Internet-based application in conjunction with 11 telephone coaching sessions.

Objective

Our objectives are to (1) evaluate the effectiveness of Strongest Families FASD in reducing externalizing problems (primary outcome), internalizing problems, and parent distress (secondary outcomes) in children aged between 4 and 12 years diagnosed with FASD when compared to a control group with access to a static resource Web page; (2) evaluate the effectiveness of Strongest Families FASD in improving social competence (secondary outcome) in school-aged children aged between 6 and 12 diagnosed with FASD when compared with an online psychoeducation control; and (3) explore parental satisfaction with the Strongest Families FASD online parenting program.

Methods

Parents and caregivers (N=200) of children diagnosed with FASD who have significant behavioral challenges, ages 4-12, are being recruited into a 2-arm randomized trial. The trial is designed to evaluate the effectiveness of the Web-based Strongest Families FASD parenting intervention on child behavior and caregiver distress, compared to a control group receiving access to a static resource Web page (ie, a list of FASD-specific websites, readings, videos, and organizations).

Results

The primary outcome will be externalizing problems measured by the Child Behavior Checklist (CBCL). Secondary outcomes include (1) internalizing problems and (2) social competence, both measured by the CBCL; and (3) parental distress measured by the Depression Anxiety Stress Scale-21. The Client Satisfaction Questionnaire-8 (CSQ-8) and the Satisfaction Survey are completed by the intervention group at the end of session 11. Results will be reported using the standards set out in the Consolidated Standards of Reporting Trials (CONSORT) Statement.

Conclusions

It is hypothesized that the Strongest Families FASD intervention group will improve child behavior and parental distress. Caregiver satisfaction is anticipated to be positive. Advancing evidence on the effectiveness and acceptance of distance services can inform policy and adoption of eHealth programs.

ClinicalTrial

ClinicalTrials.gov NCT02210455; https://clinicaltrials.gov/ct2/show/NCT02210455 (Archived by WebCite at http://www.webcitation.org/6bbW5BSsT)