Evaluation, Acceptance, and Qualification of Digital Measures: From Proof of Concept to Endpoint

The utility and impact of digital endpoints for improving breast cancer outcomes

INTRODUCTION

In breast cancer clinical trials utilizing digital endpoints, wearable sensors record participants' health information during activities of daily living.  These sensors are worn on the wrist or finger, placed as a skin patch or headband, or embedded on clothing. Data collected from wearable sensors form the basis of a digital endpoint, useful for determining effects of novel treatments on health outcomes, particularly quality-of-life outcomes.

AREAS COVERED

References for this article were selected from a PubMed search spanning from 1 January 2017,to 1 July 2024, using the terms 'wearable sensors,' 'digital endpoints,' 'virtualtrials,' 'breast cancer.'  Additional articles from the authors' personal collection of papers and reviewers suggestions were also used.

EXPERT OPINION

Digital endpoints must be validated as proper surrogate measures for healthcare outcomes, prior to their use in breast cancer trials.  Wearable sensors may introduce biases, such as 'missing not-at-random bias,' and perhaps even exacerbate disparities in healthcare outcomes if patients not comfortable with their use are excluded from clinical trials, or if the accuracy of sensors varies between racial and ethnic groups. Therefore, before embarking on trials with digital endpoints, validation studies are required, and limitations and risks of such trials need to be addressed.

Digital endpoints in clinical trials: emerging themes from a multi-stakeholder Knowledge Exchange event

BACKGROUND

Digital technologies, such as wearable devices and smartphone applications (apps), can enable the decentralisation of clinical trials by measuring endpoints in people's chosen locations rather than in traditional clinical settings. Digital endpoints can allow high-frequency and sensitive measurements of health outcomes compared to visit-based endpoints which provide an episodic snapshot of a person's health. However, there are underexplored challenges in this emerging space that require interdisciplinary and cross-sector collaboration. A multi-stakeholder Knowledge Exchange event was organised to facilitate conversations across silos within this research ecosystem.

METHODS

A survey was sent to an initial list of stakeholders to identify potential discussion topics. Additional stakeholders were identified through iterative discussions on perspectives that needed representation. Co-design meetings with attendees were held to discuss the scope, format and ethos of the event. The event itself featured a cross-disciplinary selection of talks, a panel discussion, small-group discussions facilitated via a rolling seating plan and audience participation via Slido. A transcript was generated from the day, which, together with the output from Slido, provided a record of the day's discussions. Finally, meetings were held following the event to identify the key challenges for digital endpoints which emerged and reflections and recommendations for dissemination.

RESULTS

Several challenges for digital endpoints were identified in the following areas: patient adherence and acceptability; algorithms and software for devices; design, analysis and conduct of clinical trials with digital endpoints; the environmental impact of digital endpoints; and the need for ongoing ethical support. Learnings taken for next generation events include the need to include additional stakeholder perspectives, such as those of funders and regulators, and the need for additional resources and facilitation to allow patient and public contributors to engage meaningfully during the event.

CONCLUSIONS

The event emphasised the importance of consortium building and highlighted the critical role that collaborative, multi-disciplinary, and cross-sector efforts play in driving innovation in research design and strategic partnership building moving forward. This necessitates enhanced recognition by funders to support multi-stakeholder projects with patient involvement, standardised terminology, and the utilisation of open-source software.

Do Measures of Real-World Physical Behavior Provide Insights Into the Well-Being and Physical Function of Cancer Survivors? Cross-Sectional Analysis

BACKGROUND

As the number of cancer survivors increases, maintaining health-related quality of life in cancer survivorship is a priority. This necessitates accurate and reliable methods to assess how cancer survivors are feeling and functioning. Real-world digital measures derived from wearable sensors offer potential for monitoring well-being and physical function in cancer survivorship, but questions surrounding the clinical utility of these measures remain to be answered.

OBJECTIVE

In this secondary analysis, we used 2 existing data sets to examine how measures of real-world physical behavior, captured with a wearable accelerometer, were related to aerobic fitness and self-reported well-being and physical function in a sample of individuals who had completed cancer treatment.

METHODS

Overall, 86 disease-free cancer survivors aged 21-85 years completed self-report assessments of well-being and physical function, as well as a submaximal exercise test that was used to estimate their aerobic fitness, quantified as predicted submaximal oxygen uptake (VO2). A thigh-worn accelerometer was used to monitor participants' real-world physical behavior for 7 days. Accelerometry data were used to calculate average values of the following measures of physical behavior: sedentary time, step counts, time in light and moderate to vigorous physical activity, time and weighted median cadence in stepping bouts over 1 minute, and peak 30-second cadence.

RESULTS

Spearman correlation analyses indicated that 6 (86%) of the 7 accelerometry-derived measures of real-world physical behavior were not significantly correlated with Functional Assessment of Cancer Therapy-General total well-being or linked Patient-Reported Outcomes Measurement Information System-Physical Function scores (Ps≥.08). In contrast, all but one of the physical behavior measures were significantly correlated with submaximal VO2 (Ps≤.03). Comparing these associations using likelihood ratio tests, we found that step counts, time in stepping bouts over 1 minute, and time in moderate to vigorous activity were more strongly associated with submaximal VO2 than with self-reported well-being or physical function (Ps≤.03). In contrast, cadence in stepping bouts over 1 minute and peak 30-second cadence were not more associated with submaximal VO2 than with the self-reported measures (Ps≥.08).

CONCLUSIONS

In a sample of disease-free cancer survivors, we found that several measures of real-world physical behavior were more associated with aerobic fitness than with self-reported well-being and physical function. These results highlight the possibility that in individuals who have completed cancer treatment, measures of real-world physical behavior may provide additional information compared with self-reported and performance measures. To advance the appropriate use of digital measures in oncology clinical research, further research evaluating the clinical utility of real-world physical behavior over time in large, representative samples of cancer survivors is warranted.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03781154; https://clinicaltrials.gov/ct2/show/NCT03781154.

Development of data processing algorithm to calculate adherence for adults with cystic fibrosis using inhaled therapy - a multi-center observational study within the CFHealthHub learning health system

OBJECTIVES

To develop a robust algorithm to accurately calculate 'daily complete dose counts' for inhaled medicines, used in percent adherence calculations, from electronically-captured nebulizer data within the CFHealthHub Learning Health System.

METHODS

A multi-center, cross-sectional study involved participants and clinicians reviewing real-world inhaled medicine usage records and triangulating them with objective nebulizer data to establish a consensus on 'daily complete dose counts.' An algorithm, which used only objective nebulizer data, was then developed using a derivation dataset and evaluated using internal validation dataset. The agreement and accuracy between the algorithm-derived and consensus-derived 'daily complete dose counts' was examined, with the consensus-derived count as the reference standard.

RESULTS

Twelve people with CF participated. The algorithm derived a 'daily complete dose count' by screening out 'invalid' doses (those <60s in duration or run in cleaning mode), combining all doses starting within 120s of each other, and then screening out all doses with duration < 480s which were interrupted by power supply failure. The kappa co-efficient was 0.85 (0.71-0.91) in the derivation and 0.86 (0.77-0.94) in the validation dataset.

CONCLUSIONS

The algorithm demonstrated strong agreement with the participant-clinician consensus, enhancing confidence in CFHealthHub data. Publishingdata processing methods can encourage trust in digital endpoints and serve as an exemplar for other projects.

A coating machine for coating filaments with bioactive nanomaterials for extrusion 3D printing

Extrusion printing based on biocompatible filaments offers a wide variety of targeted medical and dental applications in the area of personalized medicine, if combined with bioactive nanomaterials. However, this requires filament to be coated with bioactive nanomaterial. This study introduces a concept of a machine to coat filament with bioactive nanomaterials and its application. A machine was constructed with modules manufactured using additive manufacturing. A filament spool of polylactide (PLA) or glycol-modified polyethylene terephthalate (PETG) was transported through a copper tube, with the outer surface of the filament heated to the appropriate glass transition temperature to incorporate added nanomaterials such as nano-hydroxyapatite (nHA) or nano-fluorapatite(nFA). Coatings with nHA led to an increase in diameter of around 3 μm, while coatings with nFA increased the diameter by 4 μm. Printing of cubes with a standard extrusion printer platform using PLA or PETG filaments with added nHA or nFA has been successfully carried out. Scanning electron microscope (SEM) images of coated filaments and printed cubes showed an irregular distribution of nHA or nFA, which could be verified by energy dispersive X-ray analysis (EDX). Adding and adjusting bioactive nanomaterials to filament with a coating machine for filament proved to generate printable filaments. With the wide range of possible applications by different nanomaterials it is anticipated that extrusion printing can cover needs for personalized medicine and dentistry.

From wearable sensor data to digital biomarker development: ten lessons learned and a framework proposal

Wearable sensor technologies are becoming increasingly relevant in health research, particularly in the context of chronic disease management. They generate real-time health data that can be translated into digital biomarkers, which can provide insights into our health and well-being. Scientific methods to collect, interpret, analyze, and translate health data from wearables to digital biomarkers vary, and systematic approaches to guide these processes are currently lacking. This paper is based on an observational, longitudinal cohort study, BarKA-MS, which collected wearable sensor data on the physical rehabilitation of people living with multiple sclerosis (MS). Based on our experience with BarKA-MS, we provide and discuss ten lessons we learned in relation to digital biomarker development across key study phases. We then summarize these lessons into a guiding framework (DACIA) that aims to informs the use of wearable sensor data for digital biomarker development and chronic disease management for future research and teaching.

Patient-centricity in digital measure development: co-evolution of best practice and regulatory guidance

Digital health technologies (DHTs) have the potential to modernize drug development and clinical trial operations by remotely, passively, and continuously collecting ecologically valid evidence that is meaningful to patients' lived experiences. Such evidence holds potential for all drug development stakeholders, including regulatory agencies, as it will help create a stronger evidentiary link between approval of new therapeutics and the ultimate aim of improving patient lives. However, only a very small number of novel digital measures have matured from exploratory usage into regulatory qualification or efficacy endpoints. This shows that despite the clear potential, actually gaining regulatory agreement that a new measure is both fit-for-purpose and delivers value remains a serious challenge. One of the key stumbling blocks for developers has been the requirement to demonstrate that a digital measure is meaningful to patients. This viewpoint aims to examine the co-evolution of regulatory guidance in the United States (U.S.) and best practice for integration of DHTs into the development of clinical outcome assessments. Contextualizing guidance on meaningfulness within the larger shift towards a patient-centric drug development approach, this paper reviews the U.S. Food and Drug Administration (FDA) guidance and existing literature surrounding the development of meaningful digital measures and patient engagement, including the recent examples of rejections by the FDA that further emphasize patient-centricity in digital measures. Finally, this paper highlights remaining hurdles and provides insights into the established frameworks for development and adoption of digital measures in clinical research.

Improved measurement of disease progression in people living with early Parkinson's disease using digital health technologies

BACKGROUND

Digital health technologies show promise for improving the measurement of Parkinson's disease in clinical research and trials. However, it is not clear whether digital measures demonstrate enhanced sensitivity to disease progression compared to traditional measurement approaches.

METHODS

To this end, we develop a wearable sensor-based digital algorithm for deriving features of upper and lower-body bradykinesia and evaluate the sensitivity of digital measures to 1-year longitudinal progression using data from the WATCH-PD study, a multicenter, observational digital assessment study in participants with early, untreated Parkinson's disease. In total, 82 early, untreated Parkinson's disease participants and 50 age-matched controls were recruited and took part in a variety of motor tasks over the course of a 12-month period while wearing body-worn inertial sensors. We establish clinical validity of sensor-based digital measures by investigating convergent validity with appropriate clinical constructs, known groups validity by distinguishing patients from healthy volunteers, and test-retest reliability by comparing measurements between visits.

RESULTS

We demonstrate clinical validity of the digital measures, and importantly, superior sensitivity of digital measures for distinguishing 1-year longitudinal change in early-stage PD relative to corresponding clinical constructs.

CONCLUSIONS

Our results demonstrate the potential of digital health technologies to enhance sensitivity to disease progression relative to existing measurement standards and may constitute the basis for use as drug development tools in clinical research.

Deep Learning-Based Psoriasis Assessment: Harnessing Clinical Trial Imaging for Accurate Psoriasis Area Severity Index Prediction

Introduction

Image-based machine learning holds great promise for facilitating clinical care; however, the datasets often used for model training differ from the interventional clinical trial-based findings frequently used to inform treatment guidelines. Here, we draw on longitudinal imaging of psoriasis patients undergoing treatment in the Ultima 2 clinical trial (NCT02684357), including 2,700 body images with psoriasis area severity index (PASI) annotations by uniformly trained dermatologists.

Methods

An image-processing workflow integrating clinical photos of multiple body regions into one model pipeline was developed, which we refer to as the "One-Step PASI" framework due to its simultaneous body detection, lesion detection, and lesion severity classification. Group-stratified cross-validation was performed with 145 deep convolutional neural network models combined in an ensemble learning architecture.

Results

The highest-performing model demonstrated a mean absolute error of 3.3, Lin's concordance correlation coefficient of 0.86, and Pearson correlation coefficient of 0.90 across a wide range of PASI scores comprising disease classifications of clear skin, mild, and moderate-to-severe disease. Within-person, time-series analysis of model performance demonstrated that PASI predictions closely tracked the trajectory of physician scores from severe to clear skin without systematically over- or underestimating PASI scores or percent changes from baseline.

Conclusion

This study demonstrates the potential of image processing and deep learning to translate otherwise inaccessible clinical trial data into accurate, extensible machine learning models to assess therapeutic efficacy.

Measuring What Is Meaningful in Cancer Cachexia Clinical Trials: A Path Forward With Digital Measures of Real-World Physical Behavior

PURPOSE

The burden of cancer cachexia on patients' health-related quality of life, specifically their physical functioning, is well documented, but clinical trials thus far have failed to show meaningful improvement in physical functioning. The purpose of this review is to summarize existing methods of assessing physical function in cancer cachexia, outline a path forward for measuring what is meaningful to patients using digital measures derived from digital health technologies (DHTs), and discuss the current landscape of digital measures from the clinical and regulatory standpoint.

DESIGN

For this narrative review, peer-reviewed articles were searched on PubMed, clinical trials records were searched on clinicaltrials.gov, and records of digital measures submitted for regulatory qualification were searched on the US Food and Drug Administration's Drug Development Tool Qualification Program database.

RESULTS

There are gaps in assessing aspects of physical function that matter to patients. Existing assessment methods such as patient-reported outcomes and objective performance outcomes have limitations, including their episodic nature and burden to patients. DHTs such as wearable sensors can capture real-world physical behavior continuously, passively, and remotely, and may provide a more comprehensive picture of patients' everyday functioning. Recent regulatory submissions showcase potential clinical implementation of digital measures in various therapeutic areas.

CONCLUSION

Digital measures of real-world physical behavior present an opportunity to detect and demonstrate improvements in physical functioning in cancer cachexia, but evidence-based development is critical. For their use in clinical and regulatory decision making, studies demonstrating meaningfulness to patients as well as feasibility and validation are necessary.

3D Video Tracking Technology in the Assessment of Orofacial Impairments in Neurological Disease: Clinical Validation

PURPOSE

This study sought to determine whether clinically interpretable kinematic features extracted automatically from three-dimensional (3D) videos were correlated with corresponding perceptual clinical orofacial ratings in individuals with orofacial impairments due to neurological disorders.

METHOD

45 participants (19 diagnosed with motor neuron diseases [MNDs] and 26 poststroke) performed two nonspeech tasks (mouth opening and lip spreading) and one speech task (repetition of a sentence "Buy Bobby a Puppy") while being video-recorded in a standardized lab setting. The color video recordings of participants were assessed by an expert clinician-a speech language pathologist-on the severity of three orofacial measures: symmetry, range of motion (ROM), and speed. Clinically interpretable 3D kinematic features, linked to symmetry, ROM, and speed, were automatically extracted from video recordings, using a deep facial landmark detection and tracking algorithm for each of the three tasks. Spearman correlations were used to identify features that were significantly correlated (p value < .05) with their corresponding clinical scores. Clinically significant kinematic features were then used in the subsequent multivariate regression models to predict the overall orofacial impairment severity score.

RESULTS

Several kinematic features extracted from 3D video recordings were associated with their corresponding perceptual clinical scores, indicating clinical validity of these automatically derived measures. Different patterns of significant features were observed between MND and poststroke groups; these differences were aligned with clinical expectations in both cases.

CONCLUSIONS

The results show that kinematic features extracted automatically from simple clinical tasks can capture characteristics used by clinicians during assessments. These findings support the clinical validity of video-based automatic extraction of kinematic features.

The use of digital outcome measures in clinical trials in rare neurological diseases: a systematic literature review

Developing drugs for rare diseases is challenging, and the precision and objectivity of outcome measures is critical to this process. In recent years, a number of technologies have increasingly been used for remote monitoring of patient health. We report a systematic literature review that aims to summarize the current state of progress with regard to the use of digital outcome measures for real-life motor function assessment of patients with rare neurological diseases. Our search of published literature identified 3826 records, of which 139 were included across 27 different diseases. This review shows that use of digital outcome measures for motor function outside a clinical setting is feasible and employed in a broad range of diseases, although we found few outcome measures that have been robustly validated and adopted as endpoints in clinical trials. Future research should focus on validation of devices, variables, and algorithms to allow for regulatory qualification and widespread adoption.

Capturing Measures That Matter: The Potential Value of Digital Measures of Physical Behavior for Alzheimer's Disease Drug Development

Alzheimer's disease (AD) is a devastating neurodegenerative disease and the primary cause of dementia worldwide. Despite the magnitude of AD's impact on patients, caregivers, and society, nearly all AD clinical trials fail. A potential contributor to this high rate of failure is that established clinical outcome assessments fail to capture subtle clinical changes, entail high burden for patients and their caregivers, and ineffectively address the aspects of health deemed important by patients and their caregivers. AD progression is associated with widespread changes in physical behavior that have impacts on the ability to function independently, which is a meaningful aspect of health for patients with AD and important for diagnosis. However, established assessments of functional independence remain underutilized in AD clinical trials and are limited by subjective biases and ceiling effects. Digital measures of real-world physical behavior assessed passively, continuously, and remotely using digital health technologies have the potential to address some of these limitations and to capture aspects of functional independence in patients with AD. In particular, measures of real-world gait, physical activity, and life-space mobility captured with wearable sensors may offer value. Additional research is needed to understand the validity, feasibility, and acceptability of these measures in AD clinical research.

Reverse Engineering of Digital Measures: Inviting Patients to the Conversation

Background

Digital measures offer an unparalleled opportunity to create a more holistic picture of how people who are patients behave in their real-world environments, thereby establishing a better connection between patients, caregivers, and the clinical evidence used to drive drug development and disease management. Reaching this vision will require achieving a new level of co-creation between the stakeholders who design, develop, use, and make decisions using evidence from digital measures.

Summary

In September 2022, the second in a series of meetings hosted by the Swiss Federal Institute of Technology in Zürich, the Foundation for the National Institutes of Health Biomarkers Consortium, and sponsored by Wellcome Trust, entitled "Reverse Engineering of Digital Measures," was held in Zurich, Switzerland, with a broad range of stakeholders sharing their experience across four case studies to examine how patient centricity is essential in shaping development and validation of digital evidence generation tools.

Key Messages

In this paper, we discuss progress and the remaining barriers to widespread use of digital measures for evidence generation in clinical development and care delivery. We also present key discussion points and takeaways in order to continue discourse and provide a basis for dissemination and outreach to the wider community and other stakeholders. The work presented here shows us a blueprint for how and why the patient voice can be thoughtfully integrated into digital measure development and that continued multistakeholder engagement is critical for further progress.

Regulatory Qualification of a Cross-Disease Digital Measure: Benefits and Challenges from the Perspective of IMI Consortium IDEA-FAST

Background

Innovative Medicines Initiative (IMI) consortium IDEA-FAST is developing novel digital measures of fatigue, sleep quality, and impact of sleep disturbances for neurodegenerative diseases and immune-mediated inflammatory diseases. In 2022, the consortium met with the European Medicines Agency (EMA) to receive advice on its plans for regulatory qualification of the measures. This viewpoint reviews the IDEA-FAST perspective on developing digital measures for multiple diseases and the advice provided by the EMA.

Summary

The EMA considered a cross-disease measure an interesting and arguably feasible concept. Developers should account for the need for a strong rationale that the clinical features to be measured are similar across diseases. In addition, they may expect increased complexity of study design, challenges when managing differences within and between disease populations, and the need for validation in both heterogeneous and homogeneous populations.

Key Messages

EMA highlighted the challenges teams may encounter when developing a cross-disease measure, though benefits potentially include reduced resources for the technology developer and health authority, faster access to innovation across different therapeutic fields, and feasibility of cross-disease comparisons. The insights included here can be used by project teams to guide them in the development of cross-disease digital measures intended for regulatory qualification.

The Case for the Patient-Centric Development of Novel Digital Sleep Assessment Tools in Major Depressive Disorder

Background

Depression imposes a major burden on public health as the leading cause of disability worldwide. Sleep disturbance is a core symptom of depression that affects the vast majority of patients. Nonetheless, it is frequently not resolved by depression treatment and may even be worsened through some pharmaceutical interventions. Disturbed sleep negatively impact patients' quality of life, and persistent sleep disturbance increases the risk of recurrence, relapse, and even suicide. However, the development of novel treatments that might improve sleep problems is hindered by the lack of reliable low-burden objective measures that can adequately assess disturbed sleep in this population.

Summary

Developing improved digital measurement tools that are fit for use in clinical trials for major depressive disorder could promote the inclusion of sleep as a focus for treatment, clinical drug development, and research. This perspective piece explores the path toward the development of novel digital measures, reviews the existing evidence on the meaningfulness of sleep in depression, and summarizes existing methods of sleep assessments, including the use of digital health technologies.

Key Messages

Our objective was to make a clear call to action and path forward for the qualification of new digital outcome measures which would enable assessment of sleep disturbance as an aspect of health that truly matters to patients, promoting sleep as an important outcome for clinical development, and ultimately ensure that disturbed sleep will not remain the forgotten symptom of depression.

Considerations for Analyzing and Interpreting Data from Biometric Monitoring Technologies in Clinical Trials

BACKGROUND

The proliferation and increasing maturity of biometric monitoring technologies allow clinical investigators to measure the health status of trial participants in a more holistic manner, especially outside of traditional clinical settings. This includes capturing meaningful aspects of health in daily living and a more granular and objective manner compared to traditional tools in clinical settings.

SUMMARY

Within multidisciplinary teams, statisticians and data scientists are increasingly involved in clinical trials that incorporate digital clinical measures. They are called upon to provide input into trial planning, generation of evidence on the clinical validity of novel clinical measures, and evaluation of the adequacy of existing evidence. Analysis objectives related to demonstrating clinical validity of novel clinical measures differ from typical objectives related to demonstrating safety and efficacy of therapeutic interventions using established measures which statisticians are most familiar with.

KEY MESSAGES

This paper discusses key considerations for generating evidence for clinical validity through the lens of the type and intended use of a clinical measure. This paper also briefly discusses the regulatory pathways through which clinical validity evidence may be reviewed and highlights challenges that investigators may encounter while dealing with data from biometric monitoring technologies.

Digital Biomarkers in Psychiatric Research: Data Protection Qualifications in a Complex Ecosystem

Psychiatric research traditionally relies on subjective observation, which is time-consuming and labor-intensive. The widespread use of digital devices, such as smartphones and wearables, enables the collection and use of vast amounts of user-generated data as "digital biomarkers." These tools may also support increased participation of psychiatric patients in research and, as a result, the production of research results that are meaningful to them. However, sharing mental health data and research results may expose patients to discrimination and stigma risks, thus discouraging participation. To earn and maintain participants' trust, the first essential requirement is to implement an appropriate data governance system with a clear and transparent allocation of data protection duties and responsibilities among the actors involved in the process. These include sponsors, investigators, operators of digital tools, as well as healthcare service providers and biobanks/databanks. While previous works have proposed practical solutions to this end, there is a lack of consideration of positive data protection law issues in the extant literature. To start filling this gap, this paper discusses the GDPR legal qualifications of controller, processor, and joint controllers in the complex ecosystem unfolded by the integration of digital biomarkers in psychiatric research, considering their implications and proposing some general practical recommendations.

Measuring Health-Related Quality of Life With Multimodal Data: Viewpoint

The ability to objectively measure aspects of performance and behavior is a fundamental pillar of digital health, enabling digital wellness products, decentralized trial concepts, evidence generation, digital therapeutics, and more. Emerging multimodal technologies capable of measuring several modalities simultaneously and efforts to integrate inputs across several sources are further expanding the limits of what digital measures can assess. Experts from the field of digital health were convened as part of a multi-stakeholder workshop to examine the progress of multimodal digital measures in two key areas: detection of disease and the measurement of meaningful aspects of health relevant to the quality of life. Here we present a meeting report, summarizing key discussion points, relevant literature, and finally a vision for the immediate future, including how multimodal measures can provide value to stakeholders across drug development and care delivery, as well as three key areas where headway will need to be made if we are to continue to build on the encouraging progress so far: collaboration and data sharing, removal of barriers to data integration, and alignment around robust modular evaluation of new measurement capabilities.

Recommendations for Defining and Reporting Adherence Measured by Biometric Monitoring Technologies: Systematic Review

Background

Suboptimal adherence to data collection procedures or a study intervention is often the cause of a failed clinical trial. Data from connected sensors, including wearables, referred to here as biometric monitoring technologies (BioMeTs), are capable of capturing adherence to both digital therapeutics and digital data collection procedures, thereby providing the opportunity to identify the determinants of adherence and thereafter, methods to maximize adherence.

Objective

We aim to describe the methods and definitions by which adherence has been captured and reported using BioMeTs in recent years. Identifying key gaps allowed us to make recommendations regarding minimum reporting requirements and consistency of definitions for BioMeT-based adherence data.

Methods

We conducted a systematic review of studies published between 2014 and 2019, which deployed a BioMeT outside the clinical or laboratory setting for which a quantitative, nonsurrogate, sensor-based measurement of adherence was reported. After systematically screening the manuscripts for eligibility, we extracted details regarding study design, participants, the BioMeT or BioMeTs used, and the definition and units of adherence. The primary definitions of adherence were categorized as a continuous variable based on duration (highest resolution), a continuous variable based on the number of measurements completed, or a categorical variable (lowest resolution).

Results

Our PubMed search terms identified 940 manuscripts; 100 (10.6%) met our eligibility criteria and contained descriptions of 110 BioMeTs. During literature screening, we found that 30% (53/177) of the studies that used a BioMeT outside of the clinical or laboratory setting failed to report a sensor-based, nonsurrogate, quantitative measurement of adherence. We identified 37 unique definitions of adherence reported for the 110 BioMeTs and observed that uniformity of adherence definitions was associated with the resolution of the data reported. When adherence was reported as a continuous time-based variable, the same definition of adherence was adopted for 92% (46/50) of the tools. However, when adherence data were simplified to a categorical variable, we observed 25 unique definitions of adherence reported for 37 tools.

Conclusions

We recommend that quantitative, nonsurrogate, sensor-based adherence data be reported for all BioMeTs when feasible; a clear description of the sensor or sensors used to capture adherence data, the algorithm or algorithms that convert sample-level measurements to a metric of adherence, and the analytic validation data demonstrating that BioMeT-generated adherence is an accurate and reliable measurement of actual use be provided when available; and primary adherence data be reported as a continuous variable followed by categorical definitions if needed, and that the categories adopted are supported by clinical validation data and/or consistent with previous reports.

Predicting Changes in Depression Severity Using the PSYCHE-D (Prediction of Severity Change-Depression) Model Involving Person-Generated Health Data: Longitudinal Case-Control Observational Study

Background

In 2017, an estimated 17.3 million adults in the United States experienced at least one major depressive episode, with 35% of them not receiving any treatment. Underdiagnosis of depression has been attributed to many reasons, including stigma surrounding mental health, limited access to medical care, and barriers due to cost.

Objective

This study aimed to determine if low-burden personal health solutions, leveraging person-generated health data (PGHD), could represent a possible way to increase engagement and improve outcomes.

Methods

Here, we present the development of PSYCHE-D (Prediction of Severity Change-Depression), a predictive model developed using PGHD from more than 4000 individuals, which forecasts the long-term increase in depression severity. PSYCHE-D uses a 2-phase approach. The first phase supplements self-reports with intermediate generated labels, and the second phase predicts changing status over a 3-month period, up to 2 months in advance. The 2 phases are implemented as a single pipeline in order to eliminate data leakage and ensure results are generalizable.

Results

PSYCHE-D is composed of 2 Light Gradient Boosting Machine (LightGBM) algorithm–based classifiers that use a range of PGHD input features, including objective activity and sleep, self-reported changes in lifestyle and medication, and generated intermediate observations of depression status. The approach generalizes to previously unseen participants to detect an increase in depression severity over a 3-month interval, with a sensitivity of 55.4% and a specificity of 65.3%, nearly tripling sensitivity while maintaining specificity when compared with a random model.

Conclusions

These results demonstrate that low-burden PGHD can be the basis of accurate and timely warnings that an individual’s mental health may be deteriorating. We hope this work will serve as a basis for improved engagement and treatment of individuals experiencing depression.

Standardized Integration of Person-Generated Data Into Routine Clinical Care

Person-generated data (PGD) are a valuable source of information on a person’s health state in daily life and in between clinic visits. To fully extract value from PGD, health care organizations must be able to smoothly integrate data from PGD devices into routine clinical workflows. Ideally, to enhance efficiency and flexibility, such integrations should follow reusable processes that can easily be replicated for multiple devices and data types. Instead, current PGD integrations tend to be one-off efforts entailing high costs to build and maintain custom connections with each device and their proprietary data formats. This viewpoint paper formulates the integration of PGD into clinical systems and workflow as a PGD integration pipeline and reviews the functional components of such a pipeline. A PGD integration pipeline includes PGD acquisition, aggregation, and consumption. Acquisition is the person-facing component that includes both technical (eg, sensors, smartphone apps) and policy components (eg, informed consent). Aggregation pools, standardizes, and structures data into formats that can be used in health care settings such as within electronic health record–based workflows. PGD consumption is wide-ranging, by different solutions in different care settings (inpatient, outpatient, consumer health) for different types of users (clinicians, patients). The adoption of data and metadata standards, such as those from IEEE and Open mHealth, would facilitate aggregation and enable broader consumption. We illustrate the benefits of a standards-based integration pipeline for the illustrative use case of home blood pressure monitoring. A standards-based PGD integration pipeline can flexibly streamline the clinical use of PGD while accommodating the complexity, scale, and rapid evolution of today’s health care systems.

The Patient Matters in the End(point)

Digital health technologies such as wearable sensors are increasingly being used in clinical trials. However, the endpoints created from these useful tools are wide and varied. Often, digital health technologies such as wearable sensors are used either to collect a raw metric like "step count" or with artificial intelligence algorithms to define a biomarker for improvement. In the case of the former, improvements in such a raw metric is difficult to attribute to the patient health in a meaningful way. In the case of the latter, despite the potential predictive accuracies of machine learning and artificial intelligence approaches, the resulting biomarkers are a black box, which has limited direct interpretability to the patient's specific health concerns. The paper represents a call to arms to really place the patient at the heart of the endpoint. By designing trial endpoints which are measured by digital health technologies using a patient centered approach from the outset, the patient benefits from understanding the implications of approved medication for their life.

Neuromechanical Biomarkers for Robotic Neurorehabilitation

One of the current challenges for translational rehabilitation research is to develop the strategies to deliver accurate evaluation, prediction, patient selection, and decision-making in the clinical practice. In this regard, the robot-assisted interventions have gained popularity as they can provide the objective and quantifiable assessment of the motor performance by taking the kinematics parameters into the account. Neurophysiological parameters have also been proposed for this purpose due to the novel advances in the non-invasive signal processing techniques. In addition, other parameters linked to the motor learning and brain plasticity occurring during the rehabilitation have been explored, looking for a more holistic rehabilitation approach. However, the majority of the research done in this area is still exploratory. These parameters have shown the capability to become the “biomarkers” that are defined as the quantifiable indicators of the physiological/pathological processes and the responses to the therapeutical interventions. In this view, they could be finally used for enhancing the robot-assisted treatments. While the research on the biomarkers has been growing in the last years, there is a current need for a better comprehension and quantification of the neuromechanical processes involved in the rehabilitation. In particular, there is a lack of operationalization of the potential neuromechanical biomarkers into the clinical algorithms. In this scenario, a new framework called the “Rehabilomics” has been proposed to account for the rehabilitation research that exploits the biomarkers in its design. This study provides an overview of the state-of-the-art of the biomarkers related to the robotic neurorehabilitation, focusing on the translational studies, and underlying the need to create the comprehensive approaches that have the potential to take the research on the biomarkers into the clinical practice. We then summarize some promising biomarkers that are being under investigation in the current literature and provide some examples of their current and/or potential applications in the neurorehabilitation. Finally, we outline the main challenges and future directions in the field, briefly discussing their potential evolution and prospective.

Sensor Data Integration: A New Cross-Industry Collaboration to Articulate Value, Define Needs, and Advance a Framework for Best Practices

Data integration, the processes by which data are aggregated, combined, and made available for use, has been key to the development and growth of many technological solutions. In health care, we are experiencing a revolution in the use of sensors to collect data on patient behaviors and experiences. Yet, the potential of this data to transform health outcomes is being held back. Deficits in standards, lexicons, data rights, permissioning, and security have been well documented, less so the cultural adoption of sensor data integration as a priority for large-scale deployment and impact on patient lives. The use and reuse of trustworthy data to make better and faster decisions across drug development and care delivery will require an understanding of all stakeholder needs and best practices to ensure these needs are met. The Digital Medicine Society is launching a new multistakeholder Sensor Data Integration Tour of Duty to address these challenges and more, providing a clear direction on how sensor data can fulfill its potential to enhance patient lives.

First Regulatory Qualification of a Novel Digital Endpoint in Duchenne Muscular Dystrophy: A Multi-Stakeholder Perspective on the Impact for Patients and for Drug Development in Neuromuscular Diseases

Background

Functional outcome measures used to assess efficacy in clinical trials of investigational treatments for rare neuromuscular diseases like Duchenne muscular dystrophy (DMD) are performance-based tasks completed by the patient during hospital visits. These are prone to bias and may not reflect motor abilities in real-world settings. Digital tools, such as wearable devices and other remote sensors, provide the opportunity for continuous, objective, and sensitive measurements of functional ability during daily life. Maintaining ambulation is of key importance to individuals with DMD. Stride velocity 95th centile (SV95C) is the first wearable acquired digital endpoint to receive qualification from the European Medicines Agency (EMA) to quantify the ambulation ability of ambulant DMD patients aged ≥5 years in drug therapeutic studies; it is also currently under review for the US Food and Drug Administration (FDA) qualification.

Summary

Focusing on SV95C as a key example, we describe perspectives of multiple stakeholders on the promise of novel digital endpoints in neuromuscular disease drug development.

Walking on common ground: a cross-disciplinary scoping review on the clinical utility of digital mobility outcomes

Physical mobility is essential to health, and patients often rate it as a high-priority clinical outcome. Digital mobility outcomes (DMOs), such as real-world gait speed or step count, show promise as clinical measures in many medical conditions. However, current research is nascent and fragmented by discipline. This scoping review maps existing evidence on the clinical utility of DMOs, identifying commonalities across traditional disciplinary divides. In November 2019, 11 databases were searched for records investigating the validity and responsiveness of 34 DMOs in four diverse medical conditions (Parkinson's disease, multiple sclerosis, chronic obstructive pulmonary disease, hip fracture). Searches yielded 19,672 unique records. After screening, 855 records representing 775 studies were included and charted in systematic maps. Studies frequently investigated gait speed (70.4% of studies), step length (30.7%), cadence (21.4%), and daily step count (20.7%). They studied differences between healthy and pathological gait (36.4%), associations between DMOs and clinical measures (48.8%) or outcomes (4.3%), and responsiveness to interventions (26.8%). Gait speed, step length, cadence, step time and step count exhibited consistent evidence of validity and responsiveness in multiple conditions, although the evidence was inconsistent or lacking for other DMOs. If DMOs are to be adopted as mainstream tools, further work is needed to establish their predictive validity, responsiveness, and ecological validity. Cross-disciplinary efforts to align methodology and validate DMOs may facilitate their adoption into clinical practice.

EVIDENCE Publication Checklist for Studies Evaluating Connected Sensor Technologies: Explanation and Elaboration

The EVIDENCE (EValuatIng connecteD sENsor teChnologiEs) checklist was developed by a multidisciplinary group of content experts convened by the Digital Medicine Society, representing the clinical sciences, data management, technology development, and biostatistics. The aim of EVIDENCE is to promote high quality reporting in studies where the primary objective is an evaluation of a digital measurement product or its constituent parts. Here we use the terms digital measurement product and connected sensor technology interchangeably to refer to tools that process data captured by mobile sensors using algorithms to generate measures of behavioral and/or physiological function. EVIDENCE is applicable to 5 types of evaluations: (1) proof of concept; (2) verification, (3) analytical validation, and (4) clinical validation as defined by the V3 framework; and (5) utility and usability assessments. Using EVIDENCE, those preparing, reading, or reviewing studies evaluating digital measurement products will be better equipped to distinguish necessary reporting requirements to drive high-quality research. With broad adoption, the EVIDENCE checklist will serve as a much-needed guide to raise the bar for quality reporting in published literature evaluating digital measurements products.