The Service of Research Analytics to Optimize Digital Health Evidence Generation: Multilevel Case Study

Evaluating clinical decision support software (CDSS): challenges for robust evidence generation

OBJECTIVES

Computerized clinical decision support software (CDSS) are digital health technologies that have been traditionally categorized as medical devices. However, the evaluation frameworks for traditional medical devices are not well adapted to assess the value and safety of CDSS. In this study, we identified a range of challenges associated with CDSS evaluation as a medical device and investigated whether and how CDSS are evaluated in Australia.

METHODS

Using a qualitative approach, we interviewed 11 professionals involved in the implementation and evaluation of digital health technologies at national and regional levels. Data were thematically analyzed using both data-driven (inductive) and theory-based (deductive) approaches.

RESULTS

Our results suggest that current CDSS evaluations have an overly narrow perspective on the risks and benefits of CDSS due to an inability to capture the impact of the technology on the sociotechnical environment. By adopting a static view of the CDSS, these evaluation frameworks are unable to discern how rapidly evolving technologies and a dynamic clinical environment can impact CDSS performance. After software upgrades, CDSS can transition from providing information to specifying diagnoses and treatments. Therefore, it is not clear how CDSS can be monitored continuously when changes in the software can directly affect patient safety.

CONCLUSION

Our findings emphasize the importance of taking a living health technology assessment approach to the evaluation of digital health technologies that evolve rapidly. There is a role for observational (real-world) evidence to understand the impact of changes to the technology and the sociotechnical environment on CDSS performance.

Identifying essential factors that influence user engagement with digital mental health tools in clinical care settings: Protocol for a Delphi study

Introduction

Improving effective user engagement with digital mental health tools has become a priority in enabling the value of digital health. With increased interest from the mental health community in embedding digital health tools as part of care delivery, there is a need to examine and identify the essential factors in influencing user engagement with digital mental health tools in clinical care. The current study will use a Delphi approach to gain consensus from individuals with relevant experience and expertise (e.g. patients, clinicians and healthcare administrators) on factors that influence user engagement (i.e. an essential factor).

Methods

Participants will be invited to complete up to four rounds of online surveys. The first round of the Delphi study comprises of reviewing existing factors identified in literature and commenting on whether any factors they believe are important are missing from the list. Subsequent rounds will involve asking participants to rate the perceived impact of each factor in influencing user engagement with digital mental health tools in clinical care contexts. This work is expected to consolidate the perspectives from relevant stakeholders and the academic literature to identify a core set of factors considered essential in influencing user engagement with digital mental health tools in clinical care contexts.

Development of a Resource Guide to Support the Engagement of Mental Health Providers and Patients With Digital Health Tools: Multimethod Study

BACKGROUND

As mental illness continues to affect 1 in 5 individuals, and the need for support has increased during the COVID-19 pandemic, the promise of digital mental health tools remains largely unrealized due to a lack of uptake by patients and providers. Currently, most efforts on supporting the uptake of digital mental health tools remain fragmented across organizations and geography. There is a critical need to synthesize these efforts in order to provide a coordinated strategy of supporting the adoption of digital mental health tools.

OBJECTIVE

The specific aim of this project is to develop a web-based resource document to support the engagement of mental health providers and patients in the use of digital mental health tools.

METHODS

The web-based resource was developed using a multimethod approach. A grey literature review was conducted in 2019 to identify relevant toolkits that are available in the public domain. This was supplemented with an environmental scan where individuals with expertise in the development, acquisition, implementation, and evaluation of digital mental health tools were invited to contribute additional tools or documents not identified in the grey literature search. An engagement workshop was held with stakeholders to explore how the resource document should be developed and delivered. These findings were collectively used to develop the final iteration of the resource document.

RESULTS

Based on a gray literature review and environmental scan with 27 experts, 25 resources were identified and included in the resource guide. These resources were developed for patients and providers by organizations from 5 countries. An engagement workshop was held with 14 stakeholders, and barriers related to cultural sensitivity, sustainability, and accessibility of the toolkit were identified. The final iteration of the resource document was developed by the research team using findings from the gray literature review, environmental scan, and engagement workshop. The contents of the 45-page resource guide are directed at mental health care providers, administrators, and patients (inclusive of families and caregivers).

CONCLUSIONS

The use of a multimethod approach led to the development of a resource guide that builds on existing evidence on digital mental health tools and was co-designed with stakeholders and end-users. The resource guide is now publicly available online for free and is being promoted through digital health and mental health websites. Future work should explore how this document can be integrated into clinical care delivery and pathways.

The iCanCope pain self-management application for adolescents with juvenile idiopathic arthritis: a pilot randomized controlled trial

OBJECTIVES

To evaluate the feasibility and preliminary effectiveness of iCanCope with Pain (iCanCope), a smartphone-based pain self-management program, in adolescents with JIA. iCanCope featured symptom tracking, goal-setting, pain coping skills and social support.

METHODS

A two-arm pilot randomized controlled trial was used to evaluate the iCanCope app compared with a version with symptom tracking only. Primary (feasibility) outcomes were: participant accrual/attrition rates, success of app deployment, acceptability and adherence. Secondary (preliminary effectiveness) outcomes were: pain intensity, pain-related activity limitations and health-related quality of life. Outcomes were assessed at baseline and 8 weeks. Adherence was defined as the proportion of completed symptom reports: 'low' (≤24%); 'low-moderate' (25-49%); 'high-moderate' (50-75%); or 'high' (76-100%). Linear mixed models were applied for preliminary effectiveness analyses as per intention-to-treat.

RESULTS

Adolescents (N = 60) were recruited from three paediatric rheumatology centres. Rates of accrual and attrition were 82 and 13%, respectively. Both apps were deployed with high success (over 85%) and were rated as highly acceptable. Adherence was similar for both groups, with most participants demonstrating moderate-to-high adherence. Both groups exhibited a clinically meaningful reduction in pain intensity (≥1 point) that did not statistically differ between groups. There were no significant changes in activity limitations or health-related quality of life.

CONCLUSION

The iCanCope pilot randomized controlled trial was feasible to implement in a paediatric rheumatology setting. Both apps were deployed successfully, with high acceptability, and were associated with moderate-to-high adherence. Preliminary reductions in pain intensity warrant a future trial to evaluate effectiveness of iCanCope in improving health outcomes in adolescents with JIA.

TRIAL REGISTRATION

ClinicalTrials.gov identifier: NCT02764346.

Assessing the usability and user engagement of Thought Spot - A digital mental health help-seeking solution for transition-aged youth

OBJECTIVE

To evaluate the perceived usability of and user engagement with a digital platform (Thought Spot) designed to enhance mental health and wellness help-seeking among transition-aged youth (TAY; 17-29-years old).

MATERIALS AND METHODS

Survey responses and usage patterns were collected as part of a randomized controlled trial evaluating the efficacy of Thought Spot. Participants given Thought Spot completed an adapted Usefulness, Satisfaction, and Ease of Use (USE) Questionnaire to measure perceived usability of the platform. User engagement patterns on Thought Spot were examined using analytics data collected throughout the study (March 2018-June 2019).

RESULTS

A total of 131 transition-aged participants completed the USE questionnaire and logged on to Thought Spot at least once. Ease of learning scored higher than ease of use, usefulness and satisfaction. Participants identified numerous strengths and challenges related to usability, visual appeal, functionality and usefulness of the content. In terms of user engagement, most participants stopped using the platform after 3 weeks. Participants searched and were interested in a variety of resources, including mental health, counselling and social services.

DISCUSSION

Participants reported mixed experiences while using Thought Spot and exhibited low levels of long-term user engagement. User satisfaction, the willingness to recommend Thought Spot to others, and the willingness for future use appeared to be influenced by content relevance, ease of learning, available features, and other contextual factors. Analysis of the types of resources viewed and searches conducted by TAY end-users provided insight into their behaviour and needs.

CONCLUSION

Users had mixed perceptions about the usability of Thought Spot, which may have contributed to the high attrition rate. User satisfaction and engagement appears to be influenced by content relevance, ease of learning, and the types of features available. Further investigation to understand the contextual factors that affect TAYs' adoption and engagement with digital mental health tools is required.