1
|
Tackney MS, Carpenter JR, Villar SS. Unleashing the full potential of digital outcome measures in clinical trials: eight questions that need attention. BMC Med 2024; 22:413. [PMID: 39334286 PMCID: PMC11438362 DOI: 10.1186/s12916-024-03590-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/11/2024] [Accepted: 08/27/2024] [Indexed: 09/30/2024] Open
Abstract
The use of digital health technologies to measure outcomes in clinical trials opens new opportunities as well as methodological challenges. Digital outcome measures may provide more sensitive and higher-frequency measurements but pose vital statistical challenges around how such outcomes should be defined and validated and how trials incorporating digital outcome measures should be designed and analysed. This article presents eight methodological questions, exploring issues such as the length of measurement period, choice of summary statistic and definition and handling of missing data as well as the potential for new estimands and new analyses to leverage the time series data from digital devices. The impact of key issues highlighted by the eight questions on a primary analysis of a trial are illustrated through a simulation study based on the 2019 Bellerophon INOPulse trial which had time spent in MVPA as a digital outcome measure. These eight questions present broad areas where methodological guidance is needed to enable wider uptake of digital outcome measures in trials.
Collapse
Affiliation(s)
- Mia S Tackney
- MRC-Biostatistics Unit, University of Cambridge, East Forvie Building, Forvie, Robinson Way, Cambridge, CB2 0SR, UK.
| | - James R Carpenter
- MRC Clinical Trials Unit at University College London, Institute of Clinical Trials and Methodology, 90 High Holborn, London, WC1V 6LJ, UK
- Department of Medical Statistics, London School of Hygiene and Tropical Medicine, Keppel St, London, WC1E 7HT, UK
| | - Sofía S Villar
- MRC-Biostatistics Unit, University of Cambridge, East Forvie Building, Forvie, Robinson Way, Cambridge, CB2 0SR, UK
| |
Collapse
|
2
|
Wipperman MF, Lin AZ, Gayvert KM, Lahner B, Somersan-Karakaya S, Wu X, Im J, Lee M, Koyani B, Setliff I, Thakur M, Duan D, Breazna A, Wang F, Lim WK, Halasz G, Urbanek J, Patel Y, Atwal GS, Hamilton JD, Stuart S, Levy O, Avbersek A, Alaj R, Hamon SC, Harari O. Digital wearable insole-based identification of knee arthropathies and gait signatures using machine learning. eLife 2024; 13:e86132. [PMID: 38686919 DOI: 10.7554/elife.86132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Accepted: 04/26/2024] [Indexed: 05/02/2024] Open
Abstract
Gait is impaired in musculoskeletal conditions, such as knee arthropathy. Gait analysis is used in clinical practice to inform diagnosis and monitor disease progression or intervention response. However, clinical gait analysis relies on subjective visual observation of walking as objective gait analysis has not been possible within clinical settings due to the expensive equipment, large-scale facilities, and highly trained staff required. Relatively low-cost wearable digital insoles may offer a solution to these challenges. In this work, we demonstrate how a digital insole measuring osteoarthritis-specific gait signatures yields similar results to the clinical gait-lab standard. To achieve this, we constructed a machine learning model, trained on force plate data collected in participants with knee arthropathy and controls. This model was highly predictive of force plate data from a validation set (area under the receiver operating characteristics curve [auROC] = 0.86; area under the precision-recall curve [auPR] = 0.90) and of a separate, independent digital insole dataset containing control and knee osteoarthritis subjects (auROC = 0.83; auPR = 0.86). After showing that digital insole-derived gait characteristics are comparable to traditional gait measurements, we next showed that a single stride of raw sensor time-series data could be accurately assigned to each subject, highlighting that individuals using digital insoles can be identified by their gait characteristics. This work provides a framework for a promising alternative to traditional clinical gait analysis methods, adds to the growing body of knowledge regarding wearable technology analytical pipelines, and supports clinical development of at-home gait assessments, with the potential to improve the ease, frequency, and depth of patient monitoring.
Collapse
Affiliation(s)
- Matthew F Wipperman
- Precision Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, United States
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Allen Z Lin
- Molecular Profiling & Data Science, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Kaitlyn M Gayvert
- Molecular Profiling & Data Science, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Benjamin Lahner
- Precision Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, United States
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
- Molecular Profiling & Data Science, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Selin Somersan-Karakaya
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Xuefang Wu
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Joseph Im
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Minji Lee
- Molecular Profiling & Data Science, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Bharatkumar Koyani
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Ian Setliff
- Molecular Profiling & Data Science, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Malika Thakur
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Daoyu Duan
- Precision Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, United States
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Aurora Breazna
- Biostatistics and Data Management, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Fang Wang
- Precision Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, United States
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Wei Keat Lim
- Molecular Profiling & Data Science, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Gabor Halasz
- Molecular Profiling & Data Science, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Jacek Urbanek
- Biostatistics and Data Management, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Yamini Patel
- General Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Gurinder S Atwal
- Molecular Profiling & Data Science, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Jennifer D Hamilton
- Precision Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, United States
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Samuel Stuart
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Oren Levy
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Andreja Avbersek
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Rinol Alaj
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Sara C Hamon
- Precision Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, United States
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| | - Olivier Harari
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, United States
| |
Collapse
|
3
|
Bertha A, Alaj R, Bousnina I, Doyle MK, Friend D, Kalamegham R, Oliva L, Knezevic I, Kramer F, Podhaisky HP, Reimann S. Incorporating digitally derived endpoints within clinical development programs by leveraging prior work. NPJ Digit Med 2023; 6:139. [PMID: 37563201 PMCID: PMC10415378 DOI: 10.1038/s41746-023-00886-9] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2022] [Accepted: 07/26/2023] [Indexed: 08/12/2023] Open
Affiliation(s)
- Amy Bertha
- Bayer, 801 Pennsylvania Ave NW, Washington, DC, 20004, USA.
| | | | | | | | | | | | | | | | | | | | | |
Collapse
|
4
|
Wipperman MF, Pogoncheff G, Mateo KF, Wu X, Chen Y, Levy O, Avbersek A, Deterding RR, Hamon SC, Vu T, Alaj R, Harari O. A pilot study of the Earable device to measure facial muscle and eye movement tasks among healthy volunteers. PLOS DIGITAL HEALTH 2022; 1:e0000061. [PMID: 36812552 PMCID: PMC9931353 DOI: 10.1371/journal.pdig.0000061] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/09/2022] [Accepted: 05/09/2022] [Indexed: 11/18/2022]
Abstract
The Earable device is a behind-the-ear wearable originally developed to measure cognitive function. Since Earable measures electroencephalography (EEG), electromyography (EMG), and electrooculography (EOG), it may also have the potential to objectively quantify facial muscle and eye movement activities relevant in the assessment of neuromuscular disorders. As an initial step to developing a digital assessment in neuromuscular disorders, a pilot study was conducted to determine whether the Earable device could be utilized to objectively measure facial muscle and eye movements intended to be representative of Performance Outcome Assessments, (PerfOs) with tasks designed to model clinical PerfOs, referred to as mock-PerfO activities. The specific aims of this study were: To determine whether the Earable raw EMG, EOG, and EEG signals could be processed to extract features describing these waveforms; To determine Earable feature data quality, test re-test reliability, and statistical properties; To determine whether features derived from Earable could be used to determine the difference between various facial muscle and eye movement activities; and, To determine what features and feature types are important for mock-PerfO activity level classification. A total of N = 10 healthy volunteers participated in the study. Each study participant performed 16 mock-PerfOs activities, including talking, chewing, swallowing, eye closure, gazing in different directions, puffing cheeks, chewing an apple, and making various facial expressions. Each activity was repeated four times in the morning and four times at night. A total of 161 summary features were extracted from the EEG, EMG, and EOG bio-sensor data. Feature vectors were used as input to machine learning models to classify the mock-PerfO activities, and model performance was evaluated on a held-out test set. Additionally, a convolutional neural network (CNN) was used to classify low-level representations of the raw bio-sensor data for each task, and model performance was correspondingly evaluated and compared directly to feature classification performance. The model's prediction accuracy on the Earable device's classification ability was quantitatively assessed. Study results indicate that Earable can potentially quantify different aspects of facial and eye movements and may be used to differentiate mock-PerfO activities. Specially, Earable was found to differentiate talking, chewing, and swallowing tasks from other tasks with observed F1 scores >0.9. While EMG features contribute to classification accuracy for all tasks, EOG features are important for classifying gaze tasks. Finally, we found that analysis with summary features outperformed a CNN for activity classification. We believe Earable may be used to measure cranial muscle activity relevant for neuromuscular disorder assessment. Classification performance of mock-PerfO activities with summary features enables a strategy for detecting disease-specific signals relative to controls, as well as the monitoring of intra-subject treatment responses. Further testing is needed to evaluate the Earable device in clinical populations and clinical development settings.
Collapse
Affiliation(s)
- Matthew F. Wipperman
- Precision Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
- * E-mail: (MFW); (RA); (OH)
| | | | - Katrina F. Mateo
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
| | - Xuefang Wu
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
| | - Yiziying Chen
- Biostatistics and Data Management, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
| | - Oren Levy
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
| | - Andreja Avbersek
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
| | | | - Sara C. Hamon
- Precision Medicine, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
| | - Tam Vu
- Earable Inc., Boulder, Colorado, United States of America
| | - Rinol Alaj
- Clinical Outcomes Assessment and Patient Innovation, Global Clinical Trial Services, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
- * E-mail: (MFW); (RA); (OH)
| | - Olivier Harari
- Early Clinical Development & Experimental Sciences, Regeneron Pharmaceuticals Inc, Tarrytown, New York, United States of America
- * E-mail: (MFW); (RA); (OH)
| |
Collapse
|