1
|
Jaiswal A, Kruiper R, Rasool A, Nandkeolyar A, Wall DP, Washington P. Digitally Diagnosing Multiple Developmental Delays Using Crowdsourcing Fused With Machine Learning: Protocol for a Human-in-the-Loop Machine Learning Study. JMIR Res Protoc 2024; 13:e52205. [PMID: 38329783 PMCID: PMC10884895 DOI: 10.2196/52205] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 12/17/2023] [Accepted: 12/26/2023] [Indexed: 02/09/2024] Open
Abstract
BACKGROUND A considerable number of minors in the United States are diagnosed with developmental or psychiatric conditions, potentially influenced by underdiagnosis factors such as cost, distance, and clinician availability. Despite the potential of digital phenotyping tools with machine learning (ML) approaches to expedite diagnoses and enhance diagnostic services for pediatric psychiatric conditions, existing methods face limitations because they use a limited set of social features for prediction tasks and focus on a single binary prediction, resulting in uncertain accuracies. OBJECTIVE This study aims to propose the development of a gamified web system for data collection, followed by a fusion of novel crowdsourcing algorithms with ML behavioral feature extraction approaches to simultaneously predict diagnoses of autism spectrum disorder and attention-deficit/hyperactivity disorder in a precise and specific manner. METHODS The proposed pipeline will consist of (1) gamified web applications to curate videos of social interactions adaptively based on the needs of the diagnostic system, (2) behavioral feature extraction techniques consisting of automated ML methods and novel crowdsourcing algorithms, and (3) the development of ML models that classify several conditions simultaneously and that adaptively request additional information based on uncertainties about the data. RESULTS A preliminary version of the web interface has been implemented, and a prior feature selection method has highlighted a core set of behavioral features that can be targeted through the proposed gamified approach. CONCLUSIONS The prospect for high reward stems from the possibility of creating the first artificial intelligence-powered tool that can identify complex social behaviors well enough to distinguish conditions with nuanced differentiators such as autism spectrum disorder and attention-deficit/hyperactivity disorder. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) PRR1-10.2196/52205.
Collapse
Affiliation(s)
- Aditi Jaiswal
- Department of Information and Computer Sciences, University of Hawaii at Manoa, Honolulu, HI, United States
| | - Ruben Kruiper
- Department of Information and Computer Sciences, University of Hawaii at Manoa, Honolulu, HI, United States
| | - Abdur Rasool
- Department of Information and Computer Sciences, University of Hawaii at Manoa, Honolulu, HI, United States
| | - Aayush Nandkeolyar
- Department of Information and Computer Sciences, University of Hawaii at Manoa, Honolulu, HI, United States
| | - Dennis P Wall
- Department of Pediatrics (Systems Medicine), Stanford University School of Medicine, Stanford, CA, United States
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, United States
- Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, United States
| | - Peter Washington
- Department of Information and Computer Sciences, University of Hawaii at Manoa, Honolulu, HI, United States
| |
Collapse
|
2
|
Ponzo S, May M, Tamayo-Elizalde M, Bailey K, Shand AJ, Bamford R, Multmeier J, Griessel I, Szulyovszky B, Blakey W, Valentine S, Plans D. App Characteristics and Accuracy Metrics of Available Digital Biomarkers for Autism: Scoping Review. JMIR Mhealth Uhealth 2023; 11:e52377. [PMID: 37976084 PMCID: PMC10692878 DOI: 10.2196/52377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Revised: 10/30/2023] [Accepted: 11/02/2023] [Indexed: 11/19/2023] Open
Abstract
BACKGROUND Diagnostic delays in autism are common, with the time to diagnosis being up to 3 years from the onset of symptoms. Such delays have a proven detrimental effect on individuals and families going through the process. Digital health products, such as mobile apps, can help close this gap due to their scalability and ease of access. Further, mobile apps offer the opportunity to make the diagnostic process faster and more accurate by providing additional and timely information to clinicians undergoing autism assessments. OBJECTIVE The aim of this scoping review was to synthesize the available evidence about digital biomarker tools to aid clinicians, researchers in the autism field, and end users in making decisions as to their adoption within clinical and research settings. METHODS We conducted a structured literature search on databases and search engines to identify peer-reviewed studies and regulatory submissions that describe app characteristics, validation study details, and accuracy and validity metrics of commercial and research digital biomarker apps aimed at aiding the diagnosis of autism. RESULTS We identified 4 studies evaluating 4 products: 1 commercial and 3 research apps. The accuracy of the identified apps varied between 28% and 80.6%. Sensitivity and specificity also varied, ranging from 51.6% to 81.6% and 18.5% to 80.5%, respectively. Positive predictive value ranged from 20.3% to 76.6%, and negative predictive value fluctuated between 48.7% and 97.4%. Further, we found a lack of details around participants' demographics and, where these were reported, important imbalances in sex and ethnicity in the studies evaluating such products. Finally, evaluation methods as well as accuracy and validity metrics of available tools were not clearly reported in some cases and varied greatly across studies. Different comparators were also used, with some studies validating their tools against the Diagnostic and Statistical Manual of Mental Disorders criteria and others through self-reported measures. Further, while in most cases, 2 classes were used for algorithm validation purposes, 1 of the studies reported a third category (indeterminate). These discrepancies substantially impact the comparability and generalizability of the results, thus highlighting the need for standardized validation processes and the reporting of findings. CONCLUSIONS Despite their popularity, systematic evaluations and syntheses of the current state of the art of digital health products are lacking. Standardized and transparent evaluations of digital health tools in diverse populations are needed to assess their real-world usability and validity, as well as help researchers, clinicians, and end users safely adopt novel tools within clinical and research practices.
Collapse
Affiliation(s)
- Sonia Ponzo
- Healios Limited, London, United Kingdom
- Institute of Health Informatics, University College London, London, United Kingdom
| | - Merle May
- Healios Limited, London, United Kingdom
| | | | | | - Alanna J Shand
- Healios Limited, London, United Kingdom
- Institute of Epidemiology and Healthcare, University College London, London, United Kingdom
| | | | | | | | | | - William Blakey
- Healios Limited, London, United Kingdom
- School of Computing, Engineering and Physical Sciences, University of the West of Scotland, Paisley, United Kingdom
| | | | - David Plans
- Healios Limited, London, United Kingdom
- Department of Psychology, Royal Holloway, University of London, London, United Kingdom
| |
Collapse
|
3
|
Perochon S, Di Martino JM, Carpenter KLH, Compton S, Davis N, Eichner B, Espinosa S, Franz L, Krishnappa Babu PR, Sapiro G, Dawson G. Early detection of autism using digital behavioral phenotyping. Nat Med 2023; 29:2489-2497. [PMID: 37783967 PMCID: PMC10579093 DOI: 10.1038/s41591-023-02574-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Accepted: 08/25/2023] [Indexed: 10/04/2023]
Abstract
Early detection of autism, a neurodevelopmental condition associated with challenges in social communication, ensures timely access to intervention. Autism screening questionnaires have been shown to have lower accuracy when used in real-world settings, such as primary care, as compared to research studies, particularly for children of color and girls. Here we report findings from a multiclinic, prospective study assessing the accuracy of an autism screening digital application (app) administered during a pediatric well-child visit to 475 (17-36 months old) children (269 boys and 206 girls), of which 49 were diagnosed with autism and 98 were diagnosed with developmental delay without autism. The app displayed stimuli that elicited behavioral signs of autism, quantified using computer vision and machine learning. An algorithm combining multiple digital phenotypes showed high diagnostic accuracy with the area under the receiver operating characteristic curve = 0.90, sensitivity = 87.8%, specificity = 80.8%, negative predictive value = 97.8% and positive predictive value = 40.6%. The algorithm had similar sensitivity performance across subgroups as defined by sex, race and ethnicity. These results demonstrate the potential for digital phenotyping to provide an objective, scalable approach to autism screening in real-world settings. Moreover, combining results from digital phenotyping and caregiver questionnaires may increase autism screening accuracy and help reduce disparities in access to diagnosis and intervention.
Collapse
Affiliation(s)
- Sam Perochon
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
- Ecole Normale Supérieure Paris-Saclay, Gif-sur-Yvette, France
| | - J Matias Di Martino
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
| | - Kimberly L H Carpenter
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA
- Duke Center for Autism and Brain Development, Duke University, Durham, NC, USA
| | - Scott Compton
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA
- Duke Center for Autism and Brain Development, Duke University, Durham, NC, USA
| | - Naomi Davis
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA
| | - Brian Eichner
- Department of Pediatrics, Duke University, Durham, NC, USA
| | - Steven Espinosa
- Office of Information Technology, Duke University, Durham, NC, USA
| | - Lauren Franz
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA
- Duke Center for Autism and Brain Development, Duke University, Durham, NC, USA
- Duke Global Health Institute, Duke University, Durham, NC, USA
| | | | - Guillermo Sapiro
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
- Departments of Biomedical Engineering, Mathematics, and Computer Science, Duke University, Durham, NC, USA
| | - Geraldine Dawson
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA.
- Duke Center for Autism and Brain Development, Duke University, Durham, NC, USA.
| |
Collapse
|
4
|
Phan P, Mitragotri S, Zhao Z. Digital therapeutics in the clinic. Bioeng Transl Med 2023; 8:e10536. [PMID: 37476062 PMCID: PMC10354777 DOI: 10.1002/btm2.10536] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Revised: 04/05/2023] [Accepted: 04/12/2023] [Indexed: 07/22/2023] Open
Abstract
Digital therapeutics are emerging as a new form of therapeutic interventions. Unlike conventional therapeutics, digital therapeutics deliver interventions directly to patients using an evidence-based, clinically evaluated software to treat, manage, or prevent diseases. Digital therapeutics manifest in diverse forms such as web-based applications, mobile applications on smart devices, virtual reality, and video games. As its own product category for FDA approval, digital therapeutics can function as stand-alone treatments or in combination with conventional therapeutics to improve adherence and/or efficacy. Here, we review the clinical landscape of digital therapeutics. We summarize FDA-approved products and their clinical use, overview >300 ongoing clinical trials, and discuss challenges for their clinical translation and strategies to overcome the same.
Collapse
Affiliation(s)
- Philana Phan
- Department of Pharmaceutical Sciences, College of PharmacyUniversity of Illinois ChicagoChicagoIllinoisUSA
| | - Samir Mitragotri
- John A. Paulson School of Engineering and Applied SciencesHarvard UniversityCambridgeMassachusettsUSA
- Wyss Institute for Biologically Inspired Engineering at Harvard UniversityBostonMassachusettsUSA
| | - Zongmin Zhao
- Department of Pharmaceutical Sciences, College of PharmacyUniversity of Illinois ChicagoChicagoIllinoisUSA
| |
Collapse
|
5
|
Washington P. Digitally Diagnosing Multiple Developmental Delays using Crowdsourcing fused with Machine Learning: A Research Protocol. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.03.05.23286817. [PMID: 36945467 PMCID: PMC10029023 DOI: 10.1101/2023.03.05.23286817] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/09/2023]
Abstract
Background Roughly 17% percent of minors in the United States aged 3 through 17 years have a diagnosis of one or more developmental or psychiatric conditions, with the true prevalence likely being higher due to underdiagnosis in rural areas and for minority populations. Unfortunately, timely diagnostic services are inaccessible to a large portion of the United States and global population due to cost, distance, and clinician availability. Digital phenotyping tools have the potential to shorten the time-to-diagnosis and to bring diagnostic services to more people by enabling accessible evaluations. While automated machine learning (ML) approaches for detection of pediatric psychiatry conditions have garnered increased research attention in recent years, existing approaches use a limited set of social features for the prediction task and focus on a single binary prediction. Objective I propose the development of a gamified web system for data collection followed by a fusion of novel crowdsourcing algorithms with machine learning behavioral feature extraction approaches to simultaneously predict diagnoses of Autism Spectrum Disorder (ASD) and Attention-Deficit/Hyperactivity Disorder (ADHD) in a precise and specific manner. Methods The proposed pipeline will consist of: (1) a gamified web applications to curate videos of social interactions adaptively based on needs of the diagnostic system, (2) behavioral feature extraction techniques consisting of automated ML methods and novel crowdsourcing algorithms, and (3) development of ML models which classify several conditions simultaneously and which adaptively request additional information based on uncertainties about the data. Conclusions The prospective for high reward stems from the possibility of creating the first AI-powered tool which can identify complex social behaviors well enough to distinguish conditions with nuanced differentiators such as ASD and ADHD.
Collapse
|