Digital health benefits evaluation frameworks: building the evidence to support Australia's National Digital Health Strategy

Evaluating clinical decision support software (CDSS): challenges for robust evidence generation

OBJECTIVES

Computerized clinical decision support software (CDSS) are digital health technologies that have been traditionally categorized as medical devices. However, the evaluation frameworks for traditional medical devices are not well adapted to assess the value and safety of CDSS. In this study, we identified a range of challenges associated with CDSS evaluation as a medical device and investigated whether and how CDSS are evaluated in Australia.

METHODS

Using a qualitative approach, we interviewed 11 professionals involved in the implementation and evaluation of digital health technologies at national and regional levels. Data were thematically analyzed using both data-driven (inductive) and theory-based (deductive) approaches.

RESULTS

Our results suggest that current CDSS evaluations have an overly narrow perspective on the risks and benefits of CDSS due to an inability to capture the impact of the technology on the sociotechnical environment. By adopting a static view of the CDSS, these evaluation frameworks are unable to discern how rapidly evolving technologies and a dynamic clinical environment can impact CDSS performance. After software upgrades, CDSS can transition from providing information to specifying diagnoses and treatments. Therefore, it is not clear how CDSS can be monitored continuously when changes in the software can directly affect patient safety.

CONCLUSION

Our findings emphasize the importance of taking a living health technology assessment approach to the evaluation of digital health technologies that evolve rapidly. There is a role for observational (real-world) evidence to understand the impact of changes to the technology and the sociotechnical environment on CDSS performance.

Digital Biomarker-Based Interventions: Systematic Review of Systematic Reviews

BACKGROUND

The introduction of new medical technologies such as sensors has accelerated the process of collecting patient data for relevant clinical decisions, which has led to the introduction of a new technology known as digital biomarkers.

OBJECTIVE

This study aims to assess the methodological quality and quality of evidence from meta-analyses of digital biomarker-based interventions.

METHODS

This study follows the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for reporting systematic reviews, including original English publications of systematic reviews reporting meta-analyses of clinical outcomes (efficacy and safety endpoints) of digital biomarker-based interventions compared with alternative interventions without digital biomarkers. Imaging or other technologies that do not measure objective physiological or behavioral data were excluded from this study. A literature search of PubMed and the Cochrane Library was conducted, limited to 2019-2020. The quality of the methodology and evidence synthesis of the meta-analyses were assessed using AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews 2) and GRADE (Grading of Recommendations, Assessment, Development, and Evaluations), respectively. This study was funded by the National Research, Development and Innovation Fund of Hungary.

RESULTS

A total of 25 studies with 91 reported outcomes were included in the final analysis; 1 (4%), 1 (4%), and 23 (92%) studies had high, low, and critically low methodologic quality, respectively. As many as 6 clinical outcomes (7%) had high-quality evidence and 80 outcomes (88%) had moderate-quality evidence; 5 outcomes (5%) were rated with a low level of certainty, mainly due to risk of bias (85/91, 93%), inconsistency (27/91, 30%), and imprecision (27/91, 30%). There is high-quality evidence of improvements in mortality, transplant risk, cardiac arrhythmia detection, and stroke incidence with cardiac devices, albeit with low reporting quality. High-quality reviews of pedometers reported moderate-quality evidence, including effects on physical activity and BMI. No reports with high-quality evidence and high methodological quality were found.

CONCLUSIONS

Researchers in this field should consider the AMSTAR-2 criteria and GRADE to produce high-quality studies in the future. In addition, patients, clinicians, and policymakers are advised to consider the results of this study before making clinical decisions regarding digital biomarkers to be informed of the degree of certainty of the various interventions investigated in this study. The results of this study should be considered with its limitations, such as the narrow time frame.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/28204.

Diversity via datafication? Digital patient records and citizenship for sexuality and gender diverse people

In 2018, the Australian Government adopted an ‘opt-out’ strategy to increase participation in My Health Record (MHR), the national digital patient record system. Opt out was rationalised through discourse on the universal right to health. Media controversy ensued due to privacy fears, security and commercial exploitation of patient information. LGBT community organisations warned that people with complex health needs should consider their privacy and legal situation when deciding whether or not to opt out of MHR. With reference to the health needs of sexuality and gender diverse people, we examine MHR’s rights universalism, possessive individualism, and state-based rationalisation of health governance. MHR hails all but no-one in particular, erasing diversity and straightwashing data medicine. It is a technological solution to state-based imperatives for health governance, an emphasis that does not serve minority communities or address health needs that attract stigma and prejudice. We counterpose these effects with citizenship framings seated in critical approaches to data assemblages and sexuality and gender diversity. We suggest ways in which data medicine, of which MHR is but one example, can be made more relevant and effective for individuals and communities whose healthcare is poorly served by mainstream health systems.

Outcomes of Digital Biomarker-Based Interventions: Protocol for a Systematic Review of Systematic Reviews

BACKGROUND

Digital biomarkers are defined as objective, quantifiable, physiological, and behavioral data that are collected and measured using digital devices such as portables, wearables, implantables, or digestibles. For their widespread adoption in publicly financed health care systems, it is important to understand how their benefits translate into improved patient outcomes, which is essential for demonstrating their value.

OBJECTIVE

The paper presents the protocol for a systematic review that aims to assess the quality and strength of the evidence reported in systematic reviews regarding the impact of digital biomarkers on clinical outcomes compared to interventions without digital biomarkers.

METHODS

A comprehensive search for reviews from 2019 to 2020 will be conducted in PubMed and the Cochrane Library using keywords related to digital biomarkers and a filter for systematic reviews. Original full-text English publications of systematic reviews comparing clinical outcomes of interventions with and without digital biomarkers via meta-analysis will be included. The AMSTAR-2 tool will be used to assess the methodological quality of these reviews. To assess the quality of evidence, we will evaluate the systematic reviews using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) tool. To detect the possible presence of reporting bias, we will determine whether a protocol was published prior to the start of the studies. A qualitative summary of the results by digital biomarker technology and outcomes will be provided.

RESULTS

This protocol was submitted before data collection. Search, screening, and data extraction will commence in December 2021 in accordance with the published protocol.

CONCLUSIONS

Our study will provide a comprehensive summary of the highest level of evidence available on digital biomarker interventions, providing practical guidance for health care providers. Our results will help identify clinical areas in which the use of digital biomarkers has led to favorable clinical outcomes. In addition, our findings will highlight areas of evidence gaps where the clinical benefits of digital biomarkers have not yet been demonstrated.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/28204.

Understanding digital health ecosystem from Australian citizens' perspective: A scoping review

BACKGROUND

Digital health (DH) and the benefits of related services are fairly well understood. However, it still is critical to map the digital health care landscape including the key elements that define it as an ecosystem. Particularly, knowing the perspectives of citizens on this digital transformation is an important angle to capture. In this review we aim to analyze the relevant studies to identify how DH is understood and experienced by Australian citizens and what they may require from DH platforms.

MATERIALS AND METHODS

A scoping literature review was conducted across several electronic databases (ACM Digital Library, OVID, PubMed, Scopus, IEEE, Science Direct, SAGE), as well as grey literature. Additionally, citation mining was conducted to identify further relevant studies. Identified studies were subjected to eligibility criteria and the final set of articles was independently reviewed, analyzed, discussed and interpreted by three reviewers.

RESULTS

Of 3811 articles, 98 articles met the inclusion criteria with research-based articles-as opposed to review articles or white papers- comprising the largest proportion (72%) of the selected literature. The qualitative analysis of the literature revealed five key elements that capture the essence of the digital health ecosystem interventions from the viewpoint of the Australian citizens. The identified elements were "consumer/user", "health care", "technology", "use and usability", "data and information". These elements were further found to be associated with 127 subcategories.

CONCLUSIONS

This study is the first of its kind to analyze and synthesize the relevant literature on DH ecosystems from the citizens' perspective. Through the lens of two research questions, this study defines the key components that were found crucial to understanding citizens' experiences with DH. This understanding lays a strong foundation for designing and fostering DH ecosystem. The results provide a solid ground for empirical testing.

Approaches to Assess E-Health Programs: A Scoping Review

Context:

With the increasing trend of adopting e-Health technologies, the need for evidence for assessing e-Health technologies has become crucial. The appraisal of the e-Health program is important as this could provide guidance on further e-Health investment and adoption.

Aims:

The aim of the study was to provide an articulated body of literature on the current state of knowledge about the assessment of e-Health interventions.

Settings and Design:

Scoping review was conducted based on the framework provided by Arksey and O'Malley (2005) and considering enhancements proposed by Peters et al. (2015).

Subjects and Methods:

We searched the electronic databases and available gray literature from inception until the last week of October 2019. PRISMA flowchart for the study selection process was used to guide reporting. Data extraction included information on study design, authors, year of study, country, and key findings in terms of approaches used for assessment of e-Health programs. Data were compiled and summarized narratively.

Results:

Searches were performed between October and December 2019. Seventy-one relevant papers published between the years until the last week of October 2019 were reviewed and analyzed. After considering all the eligibility criteria, 15 papers were included. We identified 15 approaches for assessing e-Health programs, which were summarized and tabulated.

Conclusions:

This review showed that available literature on the assessment of e-Health programs is heterogeneous in terms of the methodology used. e-Health interventions are highly contextual; therefore; the phase of maturity and objective of the assessment should be considered while carrying out the assessment of e-Health programs.

Implementing models of care for musculoskeletal conditions in health systems to support value-based care

Models of Care (MoCs), and their local Models of Service Delivery, for people with musculoskeletal conditions are becoming an acceptable way of supporting effective implementation of value-based care. MoCs can support the quadruple aim of value-based care through providing people with musculoskeletal disease improved access to health services, better health outcomes and satisfactory experience of their healthcare; ensure the health professionals involved are experiencing satisfaction in delivering such care and health system resources are better utilised. Implementation of MoCs is relevant at the levels of clinical practice (micro), service delivery organisations (meso) and health system (macro) levels. The development, implementation and evaluation of MoCs has evolved over the last decade to more purposively engage people with lived experience of their condition, to operationalise the Chronic Care Model and to employ innovative solutions. This paper explores how MoCs have evolved and are supporting the delivery of value-based care in health systems.

Evaluation of Digital Technologies Tailored to Support Young People's Self-Management of Musculoskeletal Pain: Mixed Methods Study

BACKGROUND

Digital technologies connect young people with health services and resources that support their self-care. The lack of accessible, reliable digital resources tailored to young people with persistent musculoskeletal pain is a significant gap in the health services in Australia. Recognizing the intense resourcing required to develop and implement effective electronic health (eHealth) interventions, the adaptation of extant, proven digital technologies may improve access to pain care with cost and time efficiencies.

OBJECTIVE

This study aimed to test the acceptability and need for adaptation of extant digital technologies, the painHEALTH website and the iCanCope with Pain app, for use by young Australians with musculoskeletal pain.

METHODS

A 3-phased, mixed methods evaluation was undertaken from May 2019 to August 2019 in Australia. Young people aged 15 to 25 years with musculoskeletal pain for >3 months were recruited. Phases were sequential: (1) phase 1, participant testing (3 groups, each of n=5) of co-designed website prototypes compared with a control website (painHEALTH), with user tasks mapped to eHealth quality and engagement criteria; (2) phase 2, participants' week-long use of the iCanCope with Pain app with engagement data captured using a real-time analytic platform (daily check-ins for pain, interference, sleep, mood, physical activity, and energy levels; goal setting; and accessing resources); and (3) phase 3, semistructured interviews were conducted to gain insights into participants' experiences of using these digital technologies.

RESULTS

Fifteen young people (12/15, 80% female; mean age 20.5 [SD 3.3] years; range 15-25 years) participated in all 3 phases. The phase 1 aggregated group data informed the recommendations used to guide 3 rapid cycles of prototype iteration. Adaptations included optimizing navigation, improving usability (functionality), and enhancing content to promote user engagement and acceptability. In phase 2, all participants checked in, with the highest frequency of full check-ins attributed to pain intensity (183/183, 100.0%), pain interference (175/183, 95.6%), and mood (152/183, 83.1%), respectively. Individual variability was evident for monitoring progress with the highest frequency of history views for pain intensity (51/183, 32.3%), followed by pain interference (24/183, 15.2%). For the goals set feature, 87% (13/15) of participants set a total of 42 goals covering 5 areas, most frequently for activity (35/42, 83%). For phase 3, metasynthesis of qualitative data highlighted that these digital tools were perceived as youth-focused and acceptable. A total of 4 metathemes emerged: (1) importance of user-centered design to leverage user engagement; (2) website design (features) promoting user acceptability and engagement; (3) app functionality supporting self-management; and (4) the role of wider promotion, health professional digital prescriptions, and strategies to ensure longer-term engagement.

CONCLUSIONS

Leveraging extant digital tools, with appropriate user-informed adaptations, can help to build capacity tailored to support young people's self-management of musculoskeletal pain.

Digital health for COPD care: the current state of play

Chronic obstructive pulmonary disease (COPD) imposes a huge burden to our healthcare systems and societies. To alleviate the burden, digital health-"the use of digital technologies for health"-has been recognized as a potential solution for improving COPD care at scale. The aim of this review is to provide an overview of digital health interventions in COPD care. We accordingly reviewed recent and emerging evidence on digital transformation approaches for COPD care focusing on (I) self-management, (II) in-hospital care, (III) post-discharge care, (IV) hospital-at-home, (V) ambient environment, and (VI) public health surveillance. The emerging approaches included digital-technology-enabled homecare programs, electronic records, big data analytics, and environment-monitoring applications. The digital health approaches of telemonitoring, telehealth and mHealth support the self-management, post-discharge care, and hospital-at-home strategy, with prospective effects on reducing acute COPD exacerbations and hospitalizations. Electronic records and classification tools have been implemented; and their effectiveness needs to be further evaluated in future studies. Air pollution concentrations in the ambient environment are associated with declined lung functions and increased risks for hospitalization and mortality. In all the digital transformation approaches, clinical evidence on reducing mortality, the ultimate goal of digital health intervention, is often inconsistent or insufficient. Digital health transformation provides great opportunities for clinical innovations and discovery of new intervention strategies. Further research remains needed for achieving reliable improvements in clinical outcomes and cost-benefits in future studies.