1
|
Ahn YA, Moffitt JM, Tao Y, Custode S, Parlade M, Beaumont A, Cardona S, Hale M, Durocher J, Alessandri M, Shyu ML, Perry LK, Messinger DS. Objective Measurement of Social Gaze and Smile Behaviors in Children with Suspected Autism Spectrum Disorder During Administration of the Autism Diagnostic Observation Schedule, 2nd Edition. J Autism Dev Disord 2024; 54:2124-2137. [PMID: 37103660 DOI: 10.1007/s10803-023-05990-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/09/2023] [Indexed: 04/28/2023]
Abstract
Best practice for the assessment of autism spectrum disorder (ASD) symptom severity relies on clinician ratings of the Autism Diagnostic Observation Schedule, 2nd Edition (ADOS-2), but the association of these ratings with objective measures of children's social gaze and smiling is unknown. Sixty-six preschool-age children (49 boys, M = 39.97 months, SD = 10.58) with suspected ASD (61 confirmed ASD) were administered the ADOS-2 and provided social affect calibrated severity scores (SA CSS). Children's social gaze and smiling during the ADOS-2, captured with a camera contained in eyeglasses worn by the examiner and parent, were obtained via a computer vision processing pipeline. Children who gazed more at their parents (p = .04) and whose gaze at their parents involved more smiling (p = .02) received lower social affect severity scores, indicating fewer social affect symptoms, adjusted R2 = .15, p = .003.
Collapse
Affiliation(s)
- Yeojin A Ahn
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | | | - Yudong Tao
- Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL, USA
| | - Stephanie Custode
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | - Meaghan Parlade
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | - Amy Beaumont
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | - Sandra Cardona
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | - Melissa Hale
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | - Jennifer Durocher
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | | | - Mei-Ling Shyu
- Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL, USA
| | - Lynn K Perry
- Department of Psychology, University of Miami, Coral Gables, FL, USA
| | - Daniel S Messinger
- Department of Psychology, University of Miami, Coral Gables, FL, USA.
- Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL, USA.
- Departments of Pediatrics and Music Engineering, University of Miami, Coral Gables, FL, USA.
- Department of Psychology, University of Miami, 5665 Ponce de Leon Blvd., P.O. Box 248185, Coral Gables, FL, 33124, USA.
| |
Collapse
|
2
|
So WC, Wong E, Ng W, Fuego J, Lay S, So MT, Lee YY, Chan WY, Chua LY, Lam HL, Lam WT, Li HM, Leung WT, Ng YH, Wong WT. Seeing through a robot's eyes: A cross-sectional exploratory study in developing a robotic screening technology for autism. Autism Res 2024; 17:366-380. [PMID: 38183409 DOI: 10.1002/aur.3087] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Accepted: 12/09/2023] [Indexed: 01/08/2024]
Abstract
The present exploratory cross-sectional case-control study sought to develop a reliable and scalable screening tool for autism using a social robot. The robot HUMANE, installed with computer vision and linked with recognition technology, detected the direction of eye gaze of children. Children aged 3-8 (M = 5.52; N = 199) participated, 87 of whom had been confirmed with autism, 55 of whom were suspected to have autism, and 57 of whom were not considered to cause any concern for having autism. Before a session, a human experimenter instructed HUMANE to narrate a story to a child. HUMANE prompted the child to return his/her eye gaze to the robot if the child looked away, and praised the child when it re-established its eye gaze quickly after a prompt. The reliability of eye gaze detection was checked across all pairs of human raters and HUMANE and reached 0.90, indicating excellent interrater agreement. Using the pre-specified reference standard (Autism Spectrum Quotient), the sensitivity and specificity of the index tests (i.e., the number of robot prompts and duration of inattentiveness) reached 0.88 or above and the Diagnostic Odds Ratios were beyond 190. These results show that social robots may detect atypical eye patterns, suggesting a potential future for screening autism using social robots.
Collapse
Affiliation(s)
- Wing-Chee So
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Elsa Wong
- NEC Hong Kong Limited, Hung Hom, Hong Kong
| | - Wingo Ng
- NEC Hong Kong Limited, Hung Hom, Hong Kong
| | - John Fuego
- NEC Hong Kong Limited, Hung Hom, Hong Kong
| | - Sally Lay
- NEC Hong Kong Limited, Hung Hom, Hong Kong
| | - Ming-Ting So
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Yuen-Yung Lee
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Wai-Yan Chan
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Lok-Ying Chua
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Hiu-Lok Lam
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Wing-Tung Lam
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Hin-Miu Li
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Wing-To Leung
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Yu-Hei Ng
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Wing-Ting Wong
- Department of Educational Psychology, The Chinese University of Hong Kong, Shatin, Hong Kong
| |
Collapse
|
3
|
Alshammari RFN, Abd Rahman AH, Arshad H, Albahri OS. Real-Time Robotic Presentation Skill Scoring Using Multi-Model Analysis and Fuzzy Delphi-Analytic Hierarchy Process. SENSORS (BASEL, SWITZERLAND) 2023; 23:9619. [PMID: 38139465 PMCID: PMC10747450 DOI: 10.3390/s23249619] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/27/2023] [Revised: 10/30/2023] [Accepted: 11/17/2023] [Indexed: 12/24/2023]
Abstract
Existing methods for scoring student presentations predominantly rely on computer-based implementations and do not incorporate a robotic multi-classification model. This limitation can result in potential misclassification issues as these approaches lack active feature learning capabilities due to fixed camera positions. Moreover, these scoring methods often solely focus on facial expressions and neglect other crucial factors, such as eye contact, hand gestures and body movements, thereby leading to potential biases or inaccuracies in scoring. To address these limitations, this study introduces Robotics-based Presentation Skill Scoring (RPSS), which employs a multi-model analysis. RPSS captures and analyses four key presentation parameters in real time, namely facial expressions, eye contact, hand gestures and body movements, and applies the fuzzy Delphi method for criteria selection and the analytic hierarchy process for weighting, thereby enabling decision makers or managers to assign varying weights to each criterion based on its relative importance. RPSS identifies five academic facial expressions and evaluates eye contact to achieve a comprehensive assessment and enhance its scoring accuracy. Specific sub-models are employed for each presentation parameter, namely EfficientNet for facial emotions, DeepEC for eye contact and an integrated Kalman and heuristic approach for hand and body movements. The scores are determined based on predefined rules. RPSS is implemented on a robot, and the results highlight its practical applicability. Each sub-model is rigorously evaluated offline and compared against benchmarks for selection. Real-world evaluations are also conducted by incorporating a novel active learning approach to improve performance by leveraging the robot's mobility. In a comparative evaluation with human tutors, RPSS achieves a remarkable average agreement of 99%, showcasing its effectiveness in assessing students' presentation skills.
Collapse
Affiliation(s)
- Rafeef Fauzi Najim Alshammari
- Center for Artificial Intelligence Technology, Faculty of Information Science & Technology, Universiti Kebangsaan Malaysia (UKM), Bangi 43600, Selangor, Malaysia; (R.F.N.A.); (H.A.)
- College of Science, University of Kerbala, Karbala 56001, Iraq
| | - Abdul Hadi Abd Rahman
- Center for Artificial Intelligence Technology, Faculty of Information Science & Technology, Universiti Kebangsaan Malaysia (UKM), Bangi 43600, Selangor, Malaysia; (R.F.N.A.); (H.A.)
| | - Haslina Arshad
- Center for Artificial Intelligence Technology, Faculty of Information Science & Technology, Universiti Kebangsaan Malaysia (UKM), Bangi 43600, Selangor, Malaysia; (R.F.N.A.); (H.A.)
| | - Osamah Shihab Albahri
- Victorian Institute of Technology (VIT), Melbourne, VIC 3000, Australia;
- Computer Techniques Engineering Department, Mazaya University College, Nasiriyah 64001, Iraq
| |
Collapse
|
4
|
Wu Z, Zhang C, Gu X, Duporge I, Hughey LF, Stabach JA, Skidmore AK, Hopcraft JGC, Lee SJ, Atkinson PM, McCauley DJ, Lamprey R, Ngene S, Wang T. Deep learning enables satellite-based monitoring of large populations of terrestrial mammals across heterogeneous landscape. Nat Commun 2023; 14:3072. [PMID: 37244940 DOI: 10.1038/s41467-023-38901-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2022] [Accepted: 05/19/2023] [Indexed: 05/29/2023] Open
Abstract
New satellite remote sensing and machine learning techniques offer untapped possibilities to monitor global biodiversity with unprecedented speed and precision. These efficiencies promise to reveal novel ecological insights at spatial scales which are germane to the management of populations and entire ecosystems. Here, we present a robust transferable deep learning pipeline to automatically locate and count large herds of migratory ungulates (wildebeest and zebra) in the Serengeti-Mara ecosystem using fine-resolution (38-50 cm) satellite imagery. The results achieve accurate detection of nearly 500,000 individuals across thousands of square kilometers and multiple habitat types, with an overall F1-score of 84.75% (Precision: 87.85%, Recall: 81.86%). This research demonstrates the capability of satellite remote sensing and machine learning techniques to automatically and accurately count very large populations of terrestrial mammals across a highly heterogeneous landscape. We also discuss the potential for satellite-derived species detections to advance basic understanding of animal behavior and ecology.
Collapse
Affiliation(s)
- Zijing Wu
- Department of Natural Resources, Faculty of Geo-Information Science and Earth Observation, University of Twente, Enschede, The Netherlands
| | - Ce Zhang
- Lancaster Environment Center, Lancaster University, Lancaster, UK
- UK Centre for Ecology & Hydrology, Lancaster, UK
| | - Xiaowei Gu
- School of Computing, University of Kent, Canterbury, UK
| | - Isla Duporge
- Department of Ecology and Evolutionary Biology, Princeton University, Princeton, NJ, USA
- U.S. Army Research Laboratory, Army Research Office, Durham, NC, USA
- The National Academies of Sciences, Washington, D.C., USA
| | - Lacey F Hughey
- Conservation Ecology Center, Smithsonian National Zoo and Conservation Biology Institute, Front Royal, VA, USA
| | - Jared A Stabach
- Conservation Ecology Center, Smithsonian National Zoo and Conservation Biology Institute, Front Royal, VA, USA
| | - Andrew K Skidmore
- Department of Natural Resources, Faculty of Geo-Information Science and Earth Observation, University of Twente, Enschede, The Netherlands
- School of Natural Sciences, Macquarie University, Sydney, NSW, Australia
| | - J Grant C Hopcraft
- Institute of Biodiversity, Animal Health, and Comparative Medicine, University of Glasgow, Glasgow, UK
| | - Stephen J Lee
- U.S. Army Research Laboratory, Army Research Office, Durham, NC, USA
| | - Peter M Atkinson
- Lancaster Environment Center, Lancaster University, Lancaster, UK
- Geography and Environmental Science, University of Southampton, Southampton, UK
| | - Douglas J McCauley
- Department of Ecology, Evolution and Marine Biology, University of California, Santa Barbara, CA, USA
| | - Richard Lamprey
- Department of Natural Resources, Faculty of Geo-Information Science and Earth Observation, University of Twente, Enschede, The Netherlands
| | - Shadrack Ngene
- Wildlife Research and Training Institute, Naivasha, Kenya
| | - Tiejun Wang
- Department of Natural Resources, Faculty of Geo-Information Science and Earth Observation, University of Twente, Enschede, The Netherlands.
| |
Collapse
|
5
|
Li Y, Reed A, Kavoussi N, Wu JY. Eye gaze metrics for skill assessment and feedback in kidney stone surgery. Int J Comput Assist Radiol Surg 2023:10.1007/s11548-023-02901-6. [PMID: 37202714 DOI: 10.1007/s11548-023-02901-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Accepted: 03/31/2023] [Indexed: 05/20/2023]
Abstract
PURPOSE Surgical skill assessment is essential for safe operations. In endoscopic kidney stone surgery, surgeons must perform a highly skill-dependent mental mapping from the pre-operative scan to the intraoperative endoscope image. Poor mental mapping can lead to incomplete exploration of the kidney and high reoperation rates. Yet there are few objective ways to evaluate competency. We propose to use unobtrusive eye-gaze measurements in the task space to evaluate skill and provide feedback. METHODS We capture the surgeons' eye gaze on the surgical monitor with the Microsoft Hololens 2. To enable stable and accurate gaze detection, we develop a calibration algorithm to refine the eye tracking of the Hololens. In addition, we use a QR code to locate the eye gaze on the surgical monitor. We then run a user study with three expert and three novice surgeons. Each surgeon is tasked to locate three needles representing kidney stones in three different kidney phantoms. RESULTS We find that experts have more focused gaze patterns. They complete the task faster, have smaller total gaze area, and the gaze fewer times outside the area of interest. While fixation to non-fixation ratio did not show significant difference in our findings, tracking the ratio over time shows different patterns between novices and experts. CONCLUSION We show that a non-negligible difference holds between novice and expert surgeons' gaze metrics in kidney stone identification in phantoms. Expert surgeons demonstrate more targeted gaze throughout a trial, indicating their higher level of proficiency. To improve the skill acquisition process for novice surgeons, we suggest providing sub-task specific feedback. This approach presents an objective and non-invasive method to assess surgical competence.
Collapse
Affiliation(s)
- Yizhou Li
- Department of Computer Science, Vanderbilt University, 2301 Vanderbilt Pl, Nashville, TN, 37240, USA.
| | - Amy Reed
- Department of Urology, Vanderbilt University Medical Center, 1211 Medical Center Dr, Nashville, TN, 37232, USA
| | - Nicholas Kavoussi
- Department of Urology, Vanderbilt University Medical Center, 1211 Medical Center Dr, Nashville, TN, 37232, USA
| | - Jie Ying Wu
- Department of Computer Science, Vanderbilt University, 2301 Vanderbilt Pl, Nashville, TN, 37240, USA.
| |
Collapse
|
6
|
Lakkapragada A, Kline A, Mutlu OC, Paskov K, Chrisman B, Stockham N, Washington P, Wall DP. The Classification of Abnormal Hand Movement to Aid in Autism Detection: Machine Learning Study. JMIR BIOMEDICAL ENGINEERING 2022. [DOI: 10.2196/33771] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/13/2023] Open
Abstract
Background
A formal autism diagnosis can be an inefficient and lengthy process. Families may wait several months or longer before receiving a diagnosis for their child despite evidence that earlier intervention leads to better treatment outcomes. Digital technologies that detect the presence of behaviors related to autism can scale access to pediatric diagnoses. A strong indicator of the presence of autism is self-stimulatory behaviors such as hand flapping.
Objective
This study aims to demonstrate the feasibility of deep learning technologies for the detection of hand flapping from unstructured home videos as a first step toward validation of whether statistical models coupled with digital technologies can be leveraged to aid in the automatic behavioral analysis of autism. To support the widespread sharing of such home videos, we explored privacy-preserving modifications to the input space via conversion of each video to hand landmark coordinates and measured the performance of corresponding time series classifiers.
Methods
We used the Self-Stimulatory Behavior Dataset (SSBD) that contains 75 videos of hand flapping, head banging, and spinning exhibited by children. From this data set, we extracted 100 hand flapping videos and 100 control videos, each between 2 to 5 seconds in duration. We evaluated five separate feature representations: four privacy-preserved subsets of hand landmarks detected by MediaPipe and one feature representation obtained from the output of the penultimate layer of a MobileNetV2 model fine-tuned on the SSBD. We fed these feature vectors into a long short-term memory network that predicted the presence of hand flapping in each video clip.
Results
The highest-performing model used MobileNetV2 to extract features and achieved a test F1 score of 84 (SD 3.7; precision 89.6, SD 4.3 and recall 80.4, SD 6) using 5-fold cross-validation for 100 random seeds on the SSBD data (500 total distinct folds). Of the models we trained on privacy-preserved data, the model trained with all hand landmarks reached an F1 score of 66.6 (SD 3.35). Another such model trained with a select 6 landmarks reached an F1 score of 68.3 (SD 3.6). A privacy-preserved model trained using a single landmark at the base of the hands and a model trained with the average of the locations of all the hand landmarks reached an F1 score of 64.9 (SD 6.5) and 64.2 (SD 6.8), respectively.
Conclusions
We created five lightweight neural networks that can detect hand flapping from unstructured videos. Training a long short-term memory network with convolutional feature vectors outperformed training with feature vectors of hand coordinates and used almost 900,000 fewer model parameters. This study provides the first step toward developing precise deep learning methods for activity detection of autism-related behaviors.
Collapse
|
7
|
Chi NA, Washington P, Kline A, Husic A, Hou C, He C, Dunlap K, Wall DP. Classifying Autism From Crowdsourced Semistructured Speech Recordings: Machine Learning Model Comparison Study. JMIR Pediatr Parent 2022; 5:e35406. [PMID: 35436234 PMCID: PMC9052034 DOI: 10.2196/35406] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/09/2021] [Revised: 01/18/2022] [Accepted: 01/25/2022] [Indexed: 01/27/2023] Open
Abstract
BACKGROUND Autism spectrum disorder (ASD) is a neurodevelopmental disorder that results in altered behavior, social development, and communication patterns. In recent years, autism prevalence has tripled, with 1 in 44 children now affected. Given that traditional diagnosis is a lengthy, labor-intensive process that requires the work of trained physicians, significant attention has been given to developing systems that automatically detect autism. We work toward this goal by analyzing audio data, as prosody abnormalities are a signal of autism, with affected children displaying speech idiosyncrasies such as echolalia, monotonous intonation, atypical pitch, and irregular linguistic stress patterns. OBJECTIVE We aimed to test the ability for machine learning approaches to aid in detection of autism in self-recorded speech audio captured from children with ASD and neurotypical (NT) children in their home environments. METHODS We considered three methods to detect autism in child speech: (1) random forests trained on extracted audio features (including Mel-frequency cepstral coefficients); (2) convolutional neural networks trained on spectrograms; and (3) fine-tuned wav2vec 2.0-a state-of-the-art transformer-based speech recognition model. We trained our classifiers on our novel data set of cellphone-recorded child speech audio curated from the Guess What? mobile game, an app designed to crowdsource videos of children with ASD and NT children in a natural home environment. RESULTS The random forest classifier achieved 70% accuracy, the fine-tuned wav2vec 2.0 model achieved 77% accuracy, and the convolutional neural network achieved 79% accuracy when classifying children's audio as either ASD or NT. We used 5-fold cross-validation to evaluate model performance. CONCLUSIONS Our models were able to predict autism status when trained on a varied selection of home audio clips with inconsistent recording qualities, which may be more representative of real-world conditions. The results demonstrate that machine learning methods offer promise in detecting autism automatically from speech without specialized equipment.
Collapse
Affiliation(s)
- Nathan A Chi
- Division of Systems Medicine, Department of Pediatrics, Stanford University, Palo Alto, CA, United States
| | - Peter Washington
- Department of Bioengineering, Stanford University, Stanford, CA, United States
| | - Aaron Kline
- Division of Systems Medicine, Department of Pediatrics, Stanford University, Palo Alto, CA, United States
| | - Arman Husic
- Division of Systems Medicine, Department of Pediatrics, Stanford University, Palo Alto, CA, United States
| | - Cathy Hou
- Department of Computer Science, Stanford University, Stanford, CA, United States
| | - Chloe He
- Department of Biomedical Data Science, Stanford University, Stanford, CA, United States
| | - Kaitlyn Dunlap
- Division of Systems Medicine, Department of Pediatrics, Stanford University, Palo Alto, CA, United States
| | - Dennis P Wall
- Division of Systems Medicine, Department of Pediatrics, Stanford University, Palo Alto, CA, United States
- Department of Biomedical Data Science, Stanford University, Stanford, CA, United States
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, United States
| |
Collapse
|
8
|
Lombardi M, Maiettini E, De Tommaso D, Wykowska A, Natale L. Toward an Attentive Robotic Architecture: Learning-Based Mutual Gaze Estimation in Human–Robot Interaction. Front Robot AI 2022; 9:770165. [PMID: 35321344 PMCID: PMC8935014 DOI: 10.3389/frobt.2022.770165] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2021] [Accepted: 02/10/2022] [Indexed: 11/23/2022] Open
Abstract
Social robotics is an emerging field that is expected to grow rapidly in the near future. In fact, it is increasingly more frequent to have robots that operate in close proximity with humans or even collaborate with them in joint tasks. In this context, the investigation of how to endow a humanoid robot with social behavioral skills typical of human–human interactions is still an open problem. Among the countless social cues needed to establish a natural social attunement, this article reports our research toward the implementation of a mechanism for estimating the gaze direction, focusing in particular on mutual gaze as a fundamental social cue in face-to-face interactions. We propose a learning-based framework to automatically detect eye contact events in online interactions with human partners. The proposed solution achieved high performance both in silico and in experimental scenarios. Our work is expected to be the first step toward an attentive architecture able to endorse scenarios in which the robots are perceived as social partners.
Collapse
Affiliation(s)
- Maria Lombardi
- Humanoid Sensing and Perception, Istituto Italiano di Tecnologia, Genova, Italy
- *Correspondence: Maria Lombardi,
| | - Elisa Maiettini
- Humanoid Sensing and Perception, Istituto Italiano di Tecnologia, Genova, Italy
| | - Davide De Tommaso
- Social Cognition in Human-Robot Interaction, Istituto Italiano di Tecnologia, Genova, Italy
| | - Agnieszka Wykowska
- Social Cognition in Human-Robot Interaction, Istituto Italiano di Tecnologia, Genova, Italy
| | - Lorenzo Natale
- Humanoid Sensing and Perception, Istituto Italiano di Tecnologia, Genova, Italy
| |
Collapse
|
9
|
Messinger DS, Perry LK, Mitsven SG, Tao Y, Moffitt J, Fasano RM, Custode SA, Jerry CM. Computational approaches to understanding interaction and development. ADVANCES IN CHILD DEVELOPMENT AND BEHAVIOR 2022; 62:191-230. [PMID: 35249682 PMCID: PMC9840818 DOI: 10.1016/bs.acdb.2021.12.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/17/2023]
Abstract
Audio-visual recording and location tracking produce enormous quantities of digital data with which researchers can document children's everyday interactions in naturalistic settings and assessment contexts. Machine learning and other computational approaches can produce replicable, automated measurements of these big behavioral data. The economies of scale afforded by repeated automated measurements offer a potent approach to investigating linkages between real-time behavior and developmental change. In our work, automated measurement of audio from child-worn recorders-which quantify the frequency of child and adult speech and index its phonemic complexity-are paired with ultrawide radio tracking of children's location and interpersonal orientation. Applications of objective measurement indicate the influence of adult behavior in both expert ratings of attachment behavior and ratings of autism severity, suggesting the role of dyadic factors in these "child" assessments. In the preschool classroom, location/orientation measures provide data-driven measures of children's social contact, fertile ground for vocal interactions. Both the velocity of children's movement toward one another and their social contact with one another evidence homophily: children with autism spectrum disorder, other developmental disabilities, and typically developing children were more likely to interact with children in the same group even in inclusive preschool classrooms designed to promote interchange between all children. In the vocal domain, the frequency of peer speech and the phonemic complexity of teacher speech predict the frequency and phonemic complexity of children's own speech over multiple timescales. Moreover, children's own speech predicts their assessed language abilities across disability groups, suggesting how everyday interactions facilitate development.
Collapse
Affiliation(s)
- D S Messinger
- Department of Psychology, University of Miami, Coral Gables, FL, United States; Department of Electrical and Computer Engineering, University of Miami, Coral Gables, FL, United States; Departments of Pediatrics and Music Engineering, University of Miami, Coral Gables, FL, United States; Department of Music Engineering, University of Miami, Coral Gables, FL, United States.
| | - L K Perry
- Department of Psychology, University of Miami, Coral Gables, FL, United States.
| | - S G Mitsven
- Department of Psychology, University of Miami, Coral Gables, FL, United States
| | - Y Tao
- Departments of Pediatrics and Music Engineering, University of Miami, Coral Gables, FL, United States
| | - J Moffitt
- Department of Psychology, University of Miami, Coral Gables, FL, United States
| | - R M Fasano
- Department of Psychology, University of Miami, Coral Gables, FL, United States
| | - S A Custode
- Department of Psychology, University of Miami, Coral Gables, FL, United States
| | - C M Jerry
- Department of Psychology, University of Miami, Coral Gables, FL, United States; Department of Psychology, Indiana University, Bloomington, IN, United States
| |
Collapse
|
10
|
Sumioka H, Shiomi M, Honda M, Nakazawa A. Technical Challenges for Smooth Interaction With Seniors With Dementia: Lessons From Humanitude™. Front Robot AI 2021; 8:650906. [PMID: 34150858 PMCID: PMC8207295 DOI: 10.3389/frobt.2021.650906] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2021] [Accepted: 05/20/2021] [Indexed: 11/13/2022] Open
Abstract
Due to cognitive and socio-emotional decline and mental diseases, senior citizens, especially people with dementia (PwD), struggle to interact smoothly with their caregivers. Therefore, various care techniques have been proposed to develop good relationships with seniors. Among them, Humanitude is one promising technique that provides caregivers with useful interaction skills to improve their relationships with PwD, from four perspectives: face-to-face interaction, verbal communication, touch interaction, and helping care receivers stand up (physical interaction). Regardless of advances in elderly care techniques, since current social robots interact with seniors in the same manner as they do with younger adults, they lack several important functions. For example, Humanitude emphasizes the importance of interaction at a relatively intimate distance to facilitate communication with seniors. Unfortunately, few studies have developed an interaction model for clinical care communication. In this paper, we discuss the current challenges to develop a social robot that can smoothly interact with PwDs and overview the interaction skills used in Humanitude as well as the existing technologies.
Collapse
Affiliation(s)
- Hidenobu Sumioka
- Advanced Telecommunications Research Institute International, Kyoto, Japan
| | - Masahiro Shiomi
- Advanced Telecommunications Research Institute International, Kyoto, Japan
| | - Miwako Honda
- National Hospital Organization Tokyo Medical Center, Tokyo, Japan
| | | |
Collapse
|