BioMeT and Algorithm Challenges: A Proposed Digital Standardized Evaluation Framework

Unlocking the full potential of digital endpoints for decision making: a novel modular evidence concept enabling re-use and advancing collaboration

INTRODUCTION

Over the last decade increasing examples indicate opportunities to measure patient functioning and its relevance for clinical and regulatory decision making via endpoints collected through digital health technologies. More recently, we have seen such measures support primary study endpoints and enable smaller trials. The field is advancing fast: validation requirements have been proposed in the literature and regulators are releasing new guidances to review these endpoints. Pharmaceutical companies are embracing collaborations to develop them and working with academia and patient organizations in their development. However, the road to validation and regulatory acceptance is lengthy. The full value of digital endpoints cannot be unlocked until better collaboration and modular evidence frameworks are developed enabling re-use of evidence and repurposing of digital endpoints.

AREAS COVERED

This paper proposes a solution by presenting a novel modular evidence framework -the Digital Evidence Ecosystem and Protocols (DEEP)- enabling repurposing of measurement solutions, re-use of evidence, application of standards and also facilitates collaboration with health technology assessment bodies.

EXPERT OPINION

The integration of digital endpoints in healthcare, essential for personalized and remote care, requires harmonization and transparency. The proposed novel stack model offers a modular approach, fostering collaboration and expediting the adoption in patient care.

Considerations for integrating wearables into the everyday healthcare practice

Wearable technologies are becoming ever more popular as suggested tools for use in modern medicine. Studies evidence their growing pragmatism and provision of objective data for a more informative and personalised approach to patient care. Yet many wearables are one dimensional, despite the underlying technology being common across a large array of tools. That is primarily due to the accompanying software, unmodifiable or black box-based scripts that generally lack accessibility or modification, meaning wearables may often get discarded. Use of wearables for sustainable healthcare needs careful consideration.

Reverse Engineering of Digital Measures: Inviting Patients to the Conversation

Background

Digital measures offer an unparalleled opportunity to create a more holistic picture of how people who are patients behave in their real-world environments, thereby establishing a better connection between patients, caregivers, and the clinical evidence used to drive drug development and disease management. Reaching this vision will require achieving a new level of co-creation between the stakeholders who design, develop, use, and make decisions using evidence from digital measures.

Summary

In September 2022, the second in a series of meetings hosted by the Swiss Federal Institute of Technology in Zürich, the Foundation for the National Institutes of Health Biomarkers Consortium, and sponsored by Wellcome Trust, entitled "Reverse Engineering of Digital Measures," was held in Zurich, Switzerland, with a broad range of stakeholders sharing their experience across four case studies to examine how patient centricity is essential in shaping development and validation of digital evidence generation tools.

Key Messages

In this paper, we discuss progress and the remaining barriers to widespread use of digital measures for evidence generation in clinical development and care delivery. We also present key discussion points and takeaways in order to continue discourse and provide a basis for dissemination and outreach to the wider community and other stakeholders. The work presented here shows us a blueprint for how and why the patient voice can be thoughtfully integrated into digital measure development and that continued multistakeholder engagement is critical for further progress.

Assessment of neonatal respiratory rate variability

Accurate measurement of respiratory rate (RR) in neonates is challenging due to high neonatal RR variability (RRV). There is growing evidence that RRV measurement could inform and guide neonatal care. We sought to quantify neonatal RRV during a clinical study in which we compared multiparameter continuous physiological monitoring (MCPM) devices. Measurements of capnography-recorded exhaled carbon dioxide across 60-s epochs were collected from neonates admitted to the neonatal unit at Aga Khan University-Nairobi hospital. Breaths were manually counted from capnograms and using an automated signal detection algorithm which also calculated mean and median RR for each epoch. Outcome measures were between- and within-neonate RRV, between- and within-epoch RRV, and 95% limits of agreement, bias, and root-mean-square deviation. Twenty-seven neonates were included, with 130 epochs analysed. Mean manual breath count (MBC) was 48 breaths per minute. Median RRV ranged from 11.5% (interquartile range (IQR) 6.8-18.9%) to 28.1% (IQR 23.5-36.7%). Bias and limits of agreement for MBC vs algorithm-derived breath count, MBC vs algorithm-derived median breath rate, MBC vs algorithm-derived mean breath rate were - 0.5 (- 2.7, 1.66), - 3.16 (- 12.12, 5.8), and - 3.99 (- 11.3, 3.32), respectively. The marked RRV highlights the challenge of performing accurate RR measurements in neonates. More research is required to optimize the use of RRV to improve care. When evaluating MCPM devices, accuracy thresholds should be less stringent in newborns due to increased RRV. Lastly, median RR, which discounts the impact of extreme outliers, may be more reflective of the underlying physiological control of breathing.

Free-living gait does not differentiate chronic mTBI patients compared to healthy controls

Background

Physical function remains a crucial component of mild traumatic brain injury (mTBI) assessment and recovery. Traditional approaches to assess mTBI lack sensitivity to detect subtle deficits post-injury, which can impact a patient’s quality of life, daily function and can lead to chronic issues. Inertial measurement units (IMU) provide an opportunity for objective assessment of physical function and can be used in any environment. A single waist worn IMU has the potential to provide broad/macro quantity characteristics to estimate gait mobility, as well as more high-resolution micro spatial or temporal gait characteristics (herein, we refer to these as measures of quality). Our recent work showed that quantity measures of mobility were less sensitive than measures of turning quality when comparing the free-living physical function of chronic mTBI patients and healthy controls. However, no studies have examined whether measures of gait quality in free-living conditions can differentiate chronic mTBI patients and healthy controls. This study aimed to determine whether measures of free-living gait quality can differentiate chronic mTBI patients from controls.

Methods

Thirty-two patients with chronic self-reported balance symptoms after mTBI (age: 40.88 ± 11.78 years, median days post-injury: 440.68 days) and 23 healthy controls (age: 48.56 ± 22.56 years) were assessed for ~ 7 days using a single IMU at the waist on a belt. Free-living gait quality metrics were evaluated for chronic mTBI patients and controls using multi-variate analysis. Receiver operating characteristics (ROC) and Area Under the Curve (AUC) analysis were used to determine outcome sensitivity to chronic mTBI.

Results

Free-living gait quality metrics were not different between chronic mTBI patients and controls (all p > 0.05) whilst controlling for age and sex. ROC and AUC analysis showed stride length (0.63) was the most sensitive measure for differentiating chronic mTBI patients from controls.

Conclusions

Our results show that gait quality metrics determined through a free-living assessment were not significantly different between chronic mTBI patients and controls. These results suggest that measures of free-living gait quality were not impaired in our chronic mTBI patients, and/or, that the metrics chosen were not sensitive enough to detect subtle impairments in our sample.

Recommendations for Defining and Reporting Adherence Measured by Biometric Monitoring Technologies: Systematic Review

Background

Suboptimal adherence to data collection procedures or a study intervention is often the cause of a failed clinical trial. Data from connected sensors, including wearables, referred to here as biometric monitoring technologies (BioMeTs), are capable of capturing adherence to both digital therapeutics and digital data collection procedures, thereby providing the opportunity to identify the determinants of adherence and thereafter, methods to maximize adherence.

Objective

We aim to describe the methods and definitions by which adherence has been captured and reported using BioMeTs in recent years. Identifying key gaps allowed us to make recommendations regarding minimum reporting requirements and consistency of definitions for BioMeT-based adherence data.

Methods

We conducted a systematic review of studies published between 2014 and 2019, which deployed a BioMeT outside the clinical or laboratory setting for which a quantitative, nonsurrogate, sensor-based measurement of adherence was reported. After systematically screening the manuscripts for eligibility, we extracted details regarding study design, participants, the BioMeT or BioMeTs used, and the definition and units of adherence. The primary definitions of adherence were categorized as a continuous variable based on duration (highest resolution), a continuous variable based on the number of measurements completed, or a categorical variable (lowest resolution).

Results

Our PubMed search terms identified 940 manuscripts; 100 (10.6%) met our eligibility criteria and contained descriptions of 110 BioMeTs. During literature screening, we found that 30% (53/177) of the studies that used a BioMeT outside of the clinical or laboratory setting failed to report a sensor-based, nonsurrogate, quantitative measurement of adherence. We identified 37 unique definitions of adherence reported for the 110 BioMeTs and observed that uniformity of adherence definitions was associated with the resolution of the data reported. When adherence was reported as a continuous time-based variable, the same definition of adherence was adopted for 92% (46/50) of the tools. However, when adherence data were simplified to a categorical variable, we observed 25 unique definitions of adherence reported for 37 tools.

Conclusions

We recommend that quantitative, nonsurrogate, sensor-based adherence data be reported for all BioMeTs when feasible; a clear description of the sensor or sensors used to capture adherence data, the algorithm or algorithms that convert sample-level measurements to a metric of adherence, and the analytic validation data demonstrating that BioMeT-generated adherence is an accurate and reliable measurement of actual use be provided when available; and primary adherence data be reported as a continuous variable followed by categorical definitions if needed, and that the categories adopted are supported by clinical validation data and/or consistent with previous reports.

Exploring Inertial-Based Wearable Technologies for Objective Monitoring in Sports-Related Concussion: A Single-Participant Report

OBJECTIVE

Challenges remain in sports-related concussion (SRC) assessment to better inform return to play. Reliance on self-reported symptoms within the Sports Concussion Assessment Tool means that there are limited data on the effectiveness of novel methods to assess a player's readiness to return to play. Digital methods such as wearable technologies may augment traditional SRC assessment and improve objectivity in making decisions regarding return to play.

METHODS

The participant was a male university athlete who had a recent history of SRC. The single-participant design consisted of baseline laboratory testing immediately after SRC, free-living monitoring, and follow-up supervised testing after 2 months. The primary outcome measures were from traditional assessment (eg, Sports Concussion Assessment Tool and 2-minute instrumented walk/gait test; secondary outcome measures were from remote (free-living) assessment with a single wearable inertial measurement unit (eg, for gait and sleep).

RESULTS

The university athlete (age = 20 years, height = 175 cm, weight = 77 kg [176.37 lb]) recovered and returned to play 20 days after SRC. Primary measures returned to baseline levels after 12 days. However, supervised (laboratory-based) wearable device assessment showed that gait impairments (increased step time) remained even after the athlete was cleared for return to play (2 months). Similarly, a 24-hour remote gait assessment showed changes in step time, step time variability, and step time asymmetry immediately after SRC and at return to play (1 month after SRC). Remote sleep analysis showed differences in sleep quality and disturbance (increased movement between immediately after SRC and once the athlete had returned to play [1 month after SRC]).

CONCLUSION

The concern about missed or delayed SRC diagnosis is growing, but methods to objectively monitor return to play after concussion are still lacking. This report showed that wearable device assessment offers additional objective data for use in monitoring players who have SRC. This work could better inform SRC assessment and return-to-play protocols.

IMPACT

Digital technologies such as wearable technologies can yield additional data that traditional self-report approaches cannot. Combining data from nondigital (traditional) and digital (wearable) methods may augment SRC assessment for improved return-to-play decisions.

LAY SUMMARY

Inertia-based wearable technologies (eg, accelerometers) may be useful to help augment traditional, self-report approaches to sports-related concussion assessment and management by better informing return-to-play protocols.

Lack of Acceptance of Digital Healthcare in the Medical Market: Addressing Old Problems Raised by Various Clinical Professionals and Developing Possible Solutions

Various digital healthcare devices and apps, such as blood glucose meters, blood pressure monitors, and step-trackers are commonly used by patients; however, digital healthcare devices have not been widely accepted in the medical market as of yet. Despite the various legal and privacy issues involved in their use, the main reason for its poor acceptance is that users do not use such devices voluntarily and continuously. Digital healthcare devices generally do not provide valuable information to users except for tracking self-checked glucose or walking. To increase the use of these devices, users must first understand the health data produced in the context of their personal health, and the devices must be easy to use and integrated into everyday life. Thus, users need to know how to manage their own data. Medical staff must teach and encourage users to analyze and manage their patient-generated healthcare data, and users should be able to find medical values from these digital devices. Eventually, a single customized service that can comprehensively analyze various medical data to provide valuable customized services to users, and which can be linked to various heterogeneous digital healthcare devices based on the integration of various health data should be developed. Digital healthcare professionals should have detailed knowledge about a variety of digital healthcare devices and fully understand the advantages and disadvantages of digital healthcare to help patients understand and embrace the use of such devices.

Wearables in the SARS-CoV-2 Pandemic: What Are They Good for?

Recently, companies such as Apple Inc, Fitbit Inc, and Garmin Ltd have released new wearable blood oxygenation measurement technologies. Although the release of these technologies has great potential for generating health-related information, it is important to acknowledge the repercussions of consumer-targeted biometric monitoring technologies (BioMeTs), which in practice, are often used for medical decision making. BioMeTs are bodily connected digital medicine products that process data captured by mobile sensors that use algorithms to generate measures of behavioral and physiological function. These BioMeTs span both general wellness products and medical devices, and consumer-targeted BioMeTs intended for general wellness purposes are not required to undergo a standardized and transparent evaluation process for ensuring their quality and accuracy. The combination of product functionality, marketing, and the circumstances of the global SARS-CoV-2 pandemic have inevitably led to the use of consumer-targeted BioMeTs for reporting health-related measurements to drive medical decision making. In this viewpoint, we urge consumer-targeted BioMeT manufacturers to go beyond the bare minimum requirements described in US Food and Drug Administration guidance when releasing information on wellness BioMeTs. We also explore new methods and incentive systems that may result in a clearer public understanding of the performance and intended use of consumer-targeted BioMeTs.

Towards remote healthcare monitoring using accessible IoT technology: state-of-the-art, insights and experimental design

Healthcare studies are moving toward individualised measurement. There is need to move beyond supervised assessments in the laboratory/clinic. Longitudinal free-living assessment can provide a wealth of information on patient pathology and habitual behaviour, but cost and complexity of equipment have typically been a barrier. Lack of supervised conditions within free-living assessment means there is need to augment these studies with environmental analysis to provide context to individual measurements. This paper reviews low-cost and accessible Internet of Things (IoT) technologies with the aim of informing biomedical engineers of possibilities, workflows and limitations they present. In doing so, we evidence their use within healthcare research through literature and experimentation. As hardware becomes more affordable and feature rich, the cost of data magnifies. This can be limiting for biomedical engineers exploring low-cost solutions as data costs can make IoT approaches unscalable. IoT technologies can be exploited by biomedical engineers, but more research is needed before these technologies can become commonplace for clinicians and healthcare practitioners. It is hoped that the insights provided by this paper will better equip biomedical engineers to lead and monitor multi-disciplinary research investigations.