Toward Continuous Social Phenotyping: Analyzing Gaze Patterns in an Emotion Recognition Task for Children With Autism Through Wearable Smart Glasses

A Review of and Roadmap for Data Science and Machine Learning for the Neuropsychiatric Phenotype of Autism

Autism spectrum disorder (autism) is a neurodevelopmental delay that affects at least 1 in 44 children. Like many neurological disorder phenotypes, the diagnostic features are observable, can be tracked over time, and can be managed or even eliminated through proper therapy and treatments. However, there are major bottlenecks in the diagnostic, therapeutic, and longitudinal tracking pipelines for autism and related neurodevelopmental delays, creating an opportunity for novel data science solutions to augment and transform existing workflows and provide increased access to services for affected families. Several efforts previously conducted by a multitude of research labs have spawned great progress toward improved digital diagnostics and digital therapies for children with autism. We review the literature on digital health methods for autism behavior quantification and beneficial therapies using data science. We describe both case-control studies and classification systems for digital phenotyping. We then discuss digital diagnostics and therapeutics that integrate machine learning models of autism-related behaviors, including the factors that must be addressed for translational use. Finally, we describe ongoing challenges and potential opportunities for the field of autism data science. Given the heterogeneous nature of autism and the complexities of the relevant behaviors, this review contains insights that are relevant to neurological behavior analysis and digital psychiatry more broadly.

Global trends and hotspots in the digital therapeutics of autism spectrum disorders: a bibliometric analysis from 2002 to 2022

Introduction

Autism spectrum disorder (ASD) is a severe neurodevelopmental disorder that has become a major cause of disability in children. Digital therapeutics (DTx) delivers evidence-based therapeutic interventions to patients that are driven by software to prevent, manage, or treat a medical disorder or disease. This study objectively analyzed the current research status of global DTx in ASD from 2002 to 2022, aiming to explore the current global research status and trends in the field.

Methods

The Web of Science database was searched for articles about DTx in ASD from January 2002 to October 2022. CiteSpace was used to analyze the co-occurrence of keywords in literature, partnerships between authors, institutions, and countries, the sudden occurrence of keywords, clustering of keywords over time, and analysis of references, cited authors, and cited journals.

Results

A total of 509 articles were included. The most productive country and institution were the United States and Vanderbilt University. The largest contributing authors were Warren, Zachary, and Sarkar, Nilanjan. The most-cited journal was the Journal of Autism and Developmental Disorders. The most-cited and co-cited articles were Brian Scarselati (Robots for Use in Autism Research, 2012) and Ralph Adolphs (Abnormal processing of social information from faces in autism, 2001). "Artificial Intelligence," "machine learning," "Virtual Reality," and "eye tracking" were common new and cutting-edge trends in research on DTx in ASD.

Discussion

The use of DTx in ASD is developing rapidly and gaining the attention of researchers worldwide. The publications in this field have increased year by year, mainly concentrated in the developed countries, especially in the United States. Both Vanderbilt University and Yale University are very important institutions in the field. The researcher from Vanderbilt University, Warren and Zachary, his dynamics or achievements in the field is also more worth our attention. The application of new technologies such as virtual reality, machine learning, and eye-tracking in this field has driven the development of DTx on ASD and is currently a popular research topic. More cross-regional and cross-disciplinary collaborations are recommended to advance the development and availability of DTx.

Technologies to support the diagnosis and/or treatment of neurodevelopmental disorders: A systematic review

In recent years, there has been a great interest in utilizing technology in mental health research. The rapid technological development has encouraged researchers to apply technology as a part of a diagnostic process or treatment of Neurodevelopmental Disorders (NDDs). With the large number of studies being published comes an urgent need to inform clinicians and researchers about the latest advances in this field. Here, we methodically explore and summarize findings from studies published between August 2019 and February 2022. A search strategy led to the identification of 4108 records from PubMed and APA PsycInfo databases. 221 quantitative studies were included, covering a wide range of technologies used for diagnosis and/or treatment of NDDs, with the biggest focus on Autism Spectrum Disorder (ASD). The most popular technologies included machine learning, functional magnetic resonance imaging, electroencephalogram, magnetic resonance imaging, and neurofeedback. The results of the review indicate that technology-based diagnosis and intervention for NDD population is promising. However, given a high risk of bias of many studies, more high-quality research is needed.

Machine learning models using mobile game play accurately classify children with autism

Digitally-delivered healthcare is well suited to address current inequities in the delivery of care due to barriers of access to healthcare facilities. As the COVID-19 pandemic phases out, we have a unique opportunity to capitalize on the current familiarity with telemedicine approaches and continue to advocate for mainstream adoption of remote care delivery. In this paper, we specifically focus on the ability of GuessWhat? a smartphone-based charades-style gamified therapeutic intervention for autism spectrum disorder (ASD) to generate a signal that distinguishes children with ASD from neurotypical (NT) children. We demonstrate the feasibility of using "in-the-wild", naturalistic gameplay data to distinguish between ASD and NT by children by training a random forest classifier to discern the two classes (AU-ROC = 0.745, recall = 0.769). This performance demonstrates the potential for GuessWhat? to facilitate screening for ASD in historically difficult-to-reach communities. To further examine this potential, future work should expand the size of the training sample and interrogate differences in predictive ability by demographic.

Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study

BACKGROUND

Automated emotion classification could aid those who struggle to recognize emotions, including children with developmental behavioral conditions such as autism. However, most computer vision emotion recognition models are trained on adult emotion and therefore underperform when applied to child faces.

OBJECTIVE

We designed a strategy to gamify the collection and labeling of child emotion-enriched images to boost the performance of automatic child emotion recognition models to a level closer to what will be needed for digital health care approaches.

METHODS

We leveraged our prototype therapeutic smartphone game, GuessWhat, which was designed in large part for children with developmental and behavioral conditions, to gamify the secure collection of video data of children expressing a variety of emotions prompted by the game. Independently, we created a secure web interface to gamify the human labeling effort, called HollywoodSquares, tailored for use by any qualified labeler. We gathered and labeled 2155 videos, 39,968 emotion frames, and 106,001 labels on all images. With this drastically expanded pediatric emotion-centric database (>30 times larger than existing public pediatric emotion data sets), we trained a convolutional neural network (CNN) computer vision classifier of happy, sad, surprised, fearful, angry, disgust, and neutral expressions evoked by children.

RESULTS

The classifier achieved a 66.9% balanced accuracy and 67.4% F1-score on the entirety of the Child Affective Facial Expression (CAFE) as well as a 79.1% balanced accuracy and 78% F1-score on CAFE Subset A, a subset containing at least 60% human agreement on emotions labels. This performance is at least 10% higher than all previously developed classifiers evaluated against CAFE, the best of which reached a 56% balanced accuracy even when combining "anger" and "disgust" into a single class.

CONCLUSIONS

This work validates that mobile games designed for pediatric therapies can generate high volumes of domain-relevant data sets to train state-of-the-art classifiers to perform tasks helpful to precision health efforts.

Technology Behavior Model—Beyond Your Sight with Extended Reality in Surgery

Extended Reality Smart Glasses is a new pattern that uses extended reality technology to present a visual environment that combines the physical and virtual worlds. However, the surgical technique using Smart Glasses implementation is still unknown, to the infancy in clinical surgery, derived to the limits of existing technology. This study researched the acceptability and possibility of XRSG for medical experts. It combines human seen behavioral control with information technology research to construct a new “Extended Reality Technology Behavior Model” using method Technology Acceptance Model and Theory of Planned Behavior. To improve the accuracy of the study, statistical analysis, exploratory analysis, and cross-sectional research triangulation were used to collect data in five hospitals in Malaysia using a convenience sampling method and a questionnaire on behavioral influences. From the collected data, PLS-SEM analysis was used to reflect the relationship between variables. The strong positive results suggest that using XRSG by medical experts helps to improve the composition, interactivity, standardization, and clarity of medical images, resulting in increased efficiency and reduced procedure time and felt the usefulness and ease of use of XRSG through their behavior, providing a basis for technology acceptance in surgery.

Crowd annotations can approximate clinical autism impressions from short home videos with privacy protections

Artificial Intelligence (A.I.) solutions are increasingly considered for telemedicine. For these methods to serve children and their families in home settings, it is crucial to ensure the privacy of the child and parent or caregiver. To address this challenge, we explore the potential for global image transformations to provide privacy while preserving the quality of behavioral annotations. Crowd workers have previously been shown to reliably annotate behavioral features in unstructured home videos, allowing machine learning classifiers to detect autism using the annotations as input. We evaluate this method with videos altered via pixelation, dense optical flow, and Gaussian blurring. On a balanced test set of 30 videos of children with autism and 30 neurotypical controls, we find that the visual privacy alterations do not drastically alter any individual behavioral annotation at the item level. The AUROC on the evaluation set was 90.0% ±7.5% for unaltered videos, 85.0% ±9.0% for pixelation, 85.0% ±9.0% for optical flow, and 83.3% ±9.3% for blurring, demonstrating that an aggregation of small changes across behavioral questions can collectively result in increased misdiagnosis rates. We also compare crowd answers against clinicians who provided the same annotations for the same videos as crowd workers, and we find that clinicians have higher sensitivity in their recognition of autism-related symptoms. We also find that there is a linear correlation (r = 0.75, p < 0.0001) between the mean Clinical Global Impression (CGI) score provided by professional clinicians and the corresponding score emitted by a previously validated autism classifier with crowd inputs, indicating that the classifier’s output probability is a reliable estimate of the clinical impression of autism. A significant correlation is maintained with privacy alterations, indicating that crowd annotations can approximate clinician-provided autism impression from home videos in a privacy-preserved manner.

Applications of Smart Glasses in Applied Sciences: A Systematic Review

The aim of this study is to review academic papers on the applications of smart glasses. Among 82 surveyed papers, 57 were selected through filtering. The papers were published from January 2014 to October 2020. Four research questions were set up using the systematic review method, and conclusions were drawn focusing on the research trends by year and application fields; product and operating system; sensors depending on the application purpose; and data visualization, processing, and transfer methods. It was found that the most popular commercial smart glass products are Android-based Google products. In addition, smart glasses are most often used in the healthcare field, particularly for clinical and surgical assistance or for assisting mentally or physically disabled persons. For visual data transfer, 90% of the studies conducted used a camera sensor. Smart glasses have mainly been used to visualize data based on augmented reality, in contrast with the use of mixed reality. The results of this review indicate that research related to smart glasses is steadily increasing, and technological research into the development of smart glasses is being actively conducted.

Machine Learning Applications in Pediatric Ophthalmology

Purpose: To describe emerging applications of machine learning (ML) in pediatric ophthalmology with an emphasis on the diagnosis and treatment of disorders affecting visual development. Methods: Literature review of studies applying ML algorithms to problems in pediatric ophthalmology. Results: At present, the ML literature emphasizes applications in retinopathy of prematurity. However, there are increasing efforts to apply ML techniques in the diagnosis of amblyogenic conditions such as pediatric cataracts, strabismus, and high refractive error. Conclusions: A greater understanding of the principles governing ML will enable pediatric eye care providers to apply the methodology to unexplored challenges within the subspecialty.

Selection of trustworthy crowd workers for telemedical diagnosis of pediatric autism spectrum disorder

Crowd-powered telemedicine has the potential to revolutionize healthcare, especially during times that require remote access to care. However, sharing private health data with strangers from around the world is not compatible with data privacy standards, requiring a stringent filtration process to recruit reliable and trustworthy workers who can go through the proper training and security steps. The key challenge, then, is to identify capable, trustworthy, and reliable workers through high-fidelity evaluation tasks without exposing any sensitive patient data during the evaluation process. We contribute a set of experimentally validated metrics for assessing the trustworthiness and reliability of crowd workers tasked with providing behavioral feature tags to unstructured videos of children with autism and matched neurotypical controls. The workers are blinded to diagnosis and blinded to the goal of using the features to diagnose autism. These behavioral labels are fed as input to a previously validated binary logistic regression classifier for detecting autism cases using categorical feature vectors. While the metrics do not incorporate any ground truth labels of child diagnosis, linear regression using the 3 correlative metrics as input can predict the mean probability of the correct class of each worker with a mean average error of 7.51% for performance on the same set of videos and 10.93% for performance on a distinct balanced video set with different children. These results indicate that crowd workers can be recruited for performance based largely on behavioral metrics on a crowdsourced task, enabling an affordable way to filter crowd workforces into a trustworthy and reliable diagnostic workforce.

Feature replacement methods enable reliable home video analysis for machine learning detection of autism

Autism Spectrum Disorder is a neuropsychiatric condition affecting 53 million children worldwide and for which early diagnosis is critical to the outcome of behavior therapies. Machine learning applied to features manually extracted from readily accessible videos (e.g., from smartphones) has the potential to scale this diagnostic process. However, nearly unavoidable variability in video quality can lead to missing features that degrade algorithm performance. To manage this uncertainty, we evaluated the impact of missing values and feature imputation methods on two previously published autism detection classifiers, trained on standard-of-care instrument scoresheets and tested on ratings of 140 children videos from YouTube. We compare the baseline method of listwise deletion to classic univariate and multivariate techniques. We also introduce a feature replacement method that, based on a score, selects a feature from an expanded dataset to fill-in the missing value. The replacement feature selected can be identical for all records (general) or automatically adjusted to the record considered (dynamic). Our results show that general and dynamic feature replacement methods achieve a higher performance than classic univariate and multivariate methods, supporting the hypothesis that algorithmic management can maintain the fidelity of video-based diagnostics in the face of missing values and variable video quality.

Exploring the Impact of Augmented Reality in Children and Adolescents with Autism Spectrum Disorder: A Systematic Review

Autistic Spectrum Disorder (ASD) is a neurodevelopmental condition characterized by persistent difficulties in communication and social interaction along with a restriction in interests and the presence of repetitive behaviors. The development and use of augmented reality technology for autism has increased in recent years. However, little is known about the impact of these virtual reality technologies on clinical health symptoms. The aim of this systematic review was to investigate the impact of augmented reality through social, cognitive, and behavioral domains in children and adolescents with autism. This study is the first contribution that has carried out an evidence-based systematic review including relevant science databases about the effectiveness of augmented reality-based intervention in ASD. The initial search identified a total of 387 records. After the exclusion of papers that are not research studies and are duplicated articles and after screening the abstract and full text, 20 articles were selected for analysis. The studies examined suggest promising findings about the effectiveness of augmented reality-based treatments for the promotion, support, and protection of health and wellbeing in children and adolescents with autism. Finally, possible directions for future work are discussed.