1
|
Ratitch B, Trigg A, Majumder M, Vlajnic V, Rethemeier N, Nkulikiyinka R. Clinical Validation of Novel Digital Measures: Statistical Methods for Reliability Evaluation. Digit Biomark 2023; 7:74-91. [PMID: 37588480 PMCID: PMC10425717 DOI: 10.1159/000531054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2023] [Accepted: 05/02/2023] [Indexed: 08/18/2023] Open
Abstract
Background Assessment of reliability is one of the key components of the validation process designed to demonstrate that a novel clinical measure assessed by a digital health technology tool is fit-for-purpose in clinical research, care, and decision-making. Reliability assessment contributes to characterization of the signal-to-noise ratio and measurement error and is the first indicator of potential usefulness of the proposed clinical measure. Summary Methodologies for reliability analyses are scattered across literature on validation of PROs, wet biomarkers, etc., yet are equally useful for digital clinical measures. We review a general modeling framework and statistical metrics typically used for reliability assessments as part of the clinical validation. We also present methods for the assessment of agreement and measurement error, alongside modified approaches for categorical measures. We illustrate the discussed techniques using physical activity data from a wearable device with an accelerometer sensor collected in clinical trial participants. Key Messages This paper provides statisticians and data scientists, involved in development and validation of novel digital clinical measures, an overview of the statistical methodologies and analytical tools for reliability assessment.
Collapse
Affiliation(s)
- Bohdana Ratitch
- Statistics and Data Insights, Bayer Inc., Mississauga, ON, Canada
| | - Andrew Trigg
- Medical Affairs Statistics, Bayer plc, Reading, UK
| | | | - Vanja Vlajnic
- Statistics and Data Insights, Bayer Corporation, Whippany, NJ, USA
| | | | | |
Collapse
|
2
|
Clay I, Peerenboom N, Connors DE, Bourke S, Keogh A, Wac K, Gur-Arie T, Baker J, Bull C, Cereatti A, Cormack F, Eggenspieler D, Foschini L, Ganea R, Groenen PM, Gusset N, Izmailova E, Kanzler CM, Leyens L, Lyden K, Mueller A, Nam J, Ng WF, Nobbs D, Orfaniotou F, Perumal TM, Piwko W, Ries A, Scotland A, Taptiklis N, Torous J, Vereijken B, Xu S, Baltzer L, Vetter T, Goldhahn J, Hoffmann SC. Reverse Engineering of Digital Measures: Inviting Patients to the Conversation. Digit Biomark 2023; 7:28-44. [PMID: 37206894 PMCID: PMC10189241 DOI: 10.1159/000530413] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 03/07/2023] [Indexed: 05/21/2023] Open
Abstract
Background Digital measures offer an unparalleled opportunity to create a more holistic picture of how people who are patients behave in their real-world environments, thereby establishing a better connection between patients, caregivers, and the clinical evidence used to drive drug development and disease management. Reaching this vision will require achieving a new level of co-creation between the stakeholders who design, develop, use, and make decisions using evidence from digital measures. Summary In September 2022, the second in a series of meetings hosted by the Swiss Federal Institute of Technology in Zürich, the Foundation for the National Institutes of Health Biomarkers Consortium, and sponsored by Wellcome Trust, entitled "Reverse Engineering of Digital Measures," was held in Zurich, Switzerland, with a broad range of stakeholders sharing their experience across four case studies to examine how patient centricity is essential in shaping development and validation of digital evidence generation tools. Key Messages In this paper, we discuss progress and the remaining barriers to widespread use of digital measures for evidence generation in clinical development and care delivery. We also present key discussion points and takeaways in order to continue discourse and provide a basis for dissemination and outreach to the wider community and other stakeholders. The work presented here shows us a blueprint for how and why the patient voice can be thoughtfully integrated into digital measure development and that continued multistakeholder engagement is critical for further progress.
Collapse
Affiliation(s)
| | | | | | | | - Alison Keogh
- Insight Centre for Data Analytics, UC Dublin, Dublin, Ireland
- Mobilise-D, Newcastle University, Newcastle upon Tyne, UK
| | - Katarzyna Wac
- Quality of Life Lab, University of Geneva, Geneva, Switzerland
| | - Tova Gur-Arie
- Mobilise-D, Newcastle University, Newcastle upon Tyne, UK
| | | | - Christopher Bull
- Newcastle University, Newcastle, UK
- IDEA-FAST, Newcastle University, Newcastle upon Tyne, UK
| | - Andrea Cereatti
- Mobilise-D, Newcastle University, Newcastle upon Tyne, UK
- Polytechnic University of Torino, Torino, Italy
| | - Francesca Cormack
- IDEA-FAST, Newcastle University, Newcastle upon Tyne, UK
- Cambridge Cognition Ltd, Cambridge, UK
| | | | | | | | | | | | | | | | | | | | - Arne Mueller
- Mobilise-D, Newcastle University, Newcastle upon Tyne, UK
- Novartis, Basel, Switzerland
| | - Julian Nam
- F. Hoffmann-La Roche, Basel, Switzerland
| | - Wan-Fai Ng
- Newcastle University, Newcastle, UK
- IDEA-FAST, Newcastle University, Newcastle upon Tyne, UK
| | - David Nobbs
- IDEA-FAST, Newcastle University, Newcastle upon Tyne, UK
- F. Hoffmann-La Roche, Basel, Switzerland
| | | | | | - Wojciech Piwko
- Takeda Pharmaceuticals International, Zurich, Switzerland
| | - Anja Ries
- F. Hoffmann-La Roche, Basel, Switzerland
| | - Alf Scotland
- Biogen Digital Health International GmbH, Baar, Switzerland
| | - Nick Taptiklis
- IDEA-FAST, Newcastle University, Newcastle upon Tyne, UK
- Cambridge Cognition Ltd, Cambridge, UK
| | | | - Beatrix Vereijken
- Mobilise-D, Newcastle University, Newcastle upon Tyne, UK
- Norwegian University of Science and Technology, Trondheim, Norway
| | | | | | | | - Jörg Goldhahn
- Swiss Federal Institute of Technology, Zurich, Switzerland
| | | |
Collapse
|