Motion Tracker: Camera-Based Monitoring of Bodily Movements Using Motion Silhouettes

Machine learning-based classification of the movements of children with profound or severe intellectual or multiple disabilities using environment data features

Communication interventions have broadened from dialogical meaning-making, assessment approaches, to remote-controlled interactive objects. Yet, interpretation of the mostly pre-or protosymbolic, distinctive, and idiosyncratic movements of children with intellectual disabilities (IDs) or profound intellectual and multiple disabilities (PIMD) using computer-based assistive technology (AT), machine learning (ML), and environment data (ED: location, weather indices and time) remain insufficiently unexplored. We introduce a novel behavior inference computer-based communication-aid AT system structured on machine learning (ML) framework to interpret the movements of children with PIMD/IDs using ED. To establish a stable system, our study aimed to train, cross-validate (10-fold), test and compare the classification accuracy performance of ML classifiers (eXtreme gradient boosting [XGB], support vector machine [SVM], random forest [RF], and neural network [NN]) on classifying the 676 movements to 2, 3, or 7 behavior outcome classes using our proposed dataset recalibration (adding ED to movement datasets) with or without Boruta feature selection (53 child characteristics and movements, and ED-related features). Natural-child-caregiver-dyadic interactions observed in 105 single-dyad video-recorded (30-hour) sessions targeted caregiver-interpreted facial, body, and limb movements of 20 8-to 16-year-old children with PIMD/IDs and simultaneously app-and-sensor-collected ED. Classification accuracy variances and the influences of and the interaction among recalibrated dataset, feature selection, classifiers, and classes on the pooled classification accuracy rates were evaluated using three-way ANOVA. Results revealed that Boruta and NN-trained dataset in class 2 and the non-Boruta SVM-trained dataset in class 3 had >76% accuracy rates. Statistically significant effects indicating high classification rates (>60%) were found among movement datasets: with ED, non-Boruta, class 3, SVM, RF, and NN. Similar trends (>69%) were found in class 2, NN, Boruta-trained movement dataset with ED, and SVM and RF, and non-Boruta-trained movement dataset with ED in class 3. These results support our hypotheses that adding environment data to movement datasets, selecting important features using Boruta, using NN, SVM and RF classifiers, and classifying movements to 2 and 3 behavior outcomes can provide >73.3% accuracy rates, a promising performance for a stable ML-based behavior inference communication-aid AT system for children with PIMD/IDs.

A Self-Powered Portable Flexible Sensor of Monitoring Speed Skating Techniques

With the development of 5G technology, contemporary technologies such as Internet of Things (IoT) and Big Data analyses have been widely applied to the sport industry. This paper focuses on the design of a portable, self-powered, flexible sensor, which does not require an external power supply. The sensor is capable of monitoring speed skating techniques, thereby helping professional athletes to enhance their performance. This sensor mainly consists of Polyvinylidene Fluoride (PVDF) with polarization after a silvering electrode and a flexible polyester substrate. Flexible sensors are attached to the push-off joint part of speed skaters and the ice skate blade. During motion, it produces different piezoelectricity signals depending on the states of motion. The monitoring and analyzing of the real-time sensor signals will adjust the athlete's skating angle, frequency, and push-off techniques, thus improving user training and enhancing performance. Moreover, the production of piezoelectric signals can charge the capacitor, provide power for small electronic equipment (e.g., wireless device), and extend the applications of wearable flexible sensors to the Big Data and IoT technologies in the sport industry.

Beyond Dyadic Coordination: Multimodal Behavioral Irregularity in Triads Predicts Facets of Collaborative Problem Solving

We hypothesize that effective collaboration is facilitated when individuals and environmental components form a synergy where they work together and regulate one another to produce stable patterns of behavior, or regularity, as well as adaptively reorganize to form new behaviors, or irregularity. We tested this hypothesis in a study with 32 triads who collaboratively solved a challenging visual computer programming task for 20 min following an introductory warm-up phase. Multidimensional recurrence quantification analysis was used to examine fine-grained (i.e., every 10 s) collective patterns of regularity across team members' speech rate, body movement, and team interaction with the shared user interface. We found that teams exhibited significant patterns of regularity as compared to shuffled baselines, but there were no systematic trends in regularity across time. We also found that periods of regularity were associated with a reduction in overall behavior. Notably, the production of irregular behavior predicted expert-coded metrics of collaborative activity, such as teams' ability to construct shared knowledge and effectively negotiate and coordinate execution of solutions, net of overall behavioral production and behavioral self-similarity. Our findings support the theory that groups can interact to form interpersonal synergies and indicate that information about system-level dynamics is a viable way to understand and predict effective collaborative processes.

The Human Touch: Using a Webcam to Autonomously Monitor Compliance During Visual Field Assessments

Purpose

To explore the feasibility of using various easy-to-obtain biomarkers to monitor non-compliance (measurement error) during visual field assessments.

Methods

Forty-two healthy adults (42 eyes) and seven glaucoma patients (14 eyes) underwent two same-day visual field assessments. An ordinary webcam was used to compute seven potential biomarkers of task compliance, based primarily on eye gaze, head pose, and facial expression. We quantified the association between each biomarker and measurement error, as defined by (1) test–retest differences in overall test scores (mean sensitivity), and (2) failures to respond to visible stimuli on individual trials (stimuli −3 dB or more brighter than threshold).

Results

In healthy eyes, three of the seven biomarkers were significantly associated with overall (test–retest) measurement error (P = 0.003–0.007), and at least two others exhibited possible trends (P = 0.052–0.060). The weighted linear sum of all seven biomarkers was associated with overall measurement error, in both healthy eyes (r = 0.51, P < 0.001) and patients (r = 0.65, P < 0.001). Five biomarkers were each associated with failures to respond to visible stimuli on individual trials (all P < 0.001).

Conclusions

Inexpensive, autonomous measures of task compliance are associated with measurement error in visual field assessments, in terms of both the overall reliability of a test and failures to respond on particular trials (“lapses”). This could be helpful for identifying low-quality assessments and for improving assessment techniques (e.g., by discounting suspect responses or by automatically triggering comfort breaks or encouragement).

Translational Relevance

This study explores a potential way of improving the reliability of visual field assessments, a crucial but notoriously unreliable clinical measure.

Elderly Body Movement Alteration at 2nd Experience of Digital Art Installation with Cognitive and Motivation Scores

The prevalence of advanced medical treatment has led to global population aging, resulting in increased numbers of dementia patients. One of the most intractable symptoms of dementia is apathy, or lack of interest and enthusiasm, which can accompany memory and cognitive deterioration. Development of a novel method to ameliorate apathy is desirable. In this feasibility trial, we propose a series of digital art installations as a candidate dementia intervention approach. Seven, three-minute scenes of digital images and sounds were presented to visitors either passively or in response to their reactions (motion and sound). We evaluated the potential of this application as an intervention against apathy in an elderly living home. We collected the dementia global standard Mini-Mental State Examination (MMSE) and questionnaire scores of sensory pleasure and motivation along with behavioral motion data in twenty participants. We further compared responses between the first and second experiences in the thirteen participants that were present for both days. Overall, we found a significant increase in participants’ motivation. In these subjects, head and right hand motion increased over different scenes and MMSE degrees, but was most significant during passive scenes. Despite a small number of subjects and limited evaluations, this new digital art technology holds promise as an apathy intervention in the elderly and can be improved with use of wearable motion sensors.

Sit still and pay attention: Using the Wii Balance-Board to detect lapses in concentration in children during psychophysical testing

During psychophysical testing, a loss of concentration can cause observers to answer incorrectly, even when the stimulus is clearly perceptible. Such lapses limit the accuracy and speed of many psychophysical measurements. This study evaluates an automated technique for detecting lapses based on body movement (postural instability). Thirty-five children (8-11 years of age) and 34 adults performed a typical psychophysical task (orientation discrimination) while seated on a Wii Fit Balance Board: a gaming device that measures center of pressure (CoP). Incorrect responses on suprathreshold catch trials provided the "reference standard" measure of when lapses in concentration occurred. Children exhibited significantly greater variability in CoP on lapse trials, indicating that postural instability provides a feasible, real-time index of concentration. Limitations and potential applications of this method are discussed.

Quantitative Analysis of Multimodal Speech Data

This study presents techniques for quantitatively analyzing coordination and kinematics in multimodal speech using video, audio and electromagnetic articulography (EMA) data. Multimodal speech research has flourished due to recent improvements in technology, yet gesture detection/annotation strategies vary widely, leading to difficulty in generalizing across studies and in advancing this field of research. We describe how FlowAnalyzer software can be used to extract kinematic signals from basic video recordings; and we apply a technique, derived from speech kinematic research, to detect bodily gestures in these kinematic signals. We investigate whether kinematic characteristics of multimodal speech differ dependent on communicative context, and we find that these contexts can be distinguished quantitatively, suggesting a way to improve and standardize existing gesture identification/annotation strategy. We also discuss a method, Correlation Map Analysis (CMA), for quantifying the relationship between speech and bodily gesture kinematics over time. We describe potential applications of CMA to multimodal speech research, such as describing characteristics of speech-gesture coordination in different communicative contexts. The use of the techniques presented here can improve and advance multimodal speech and gesture research by applying quantitative methods in the detection and description of multimodal speech.

The Affective Computing Approach to Affect Measurement

Affective computing (AC) adopts a computational approach to study affect. We highlight the AC approach towards automated affect measures that jointly model machine-readable physiological/behavioral signals with affect estimates as reported by humans or experimentally elicited. We describe the conceptual and computational foundations of the approach followed by two case studies: one on discrimination between genuine and faked expressions of pain in the lab, and the second on measuring nonbasic affect in the wild. We discuss applications of the measures, analyze measurement accuracy and generalizability, and highlight advances afforded by computational tipping points, such as big data, wearable sensing, crowdsourcing, and deep learning. We conclude by advocating for increasing synergies between AC and affective science and offer suggestions toward that direction.

Behavioral Priming 2.0: Enter a Dynamical Systems Perspective

On a daily basis, people are exposed to numerous stimuli, ranging from colors and smells to sounds and words, that could potentially activate different cognitive constructs and influence their actions. This type of influence on human behavior is referred to as priming. Roughly two decades ago, behavioral priming was hailed as one of the core forces that shape automatic behavior. However, failures to replicate some of the representative findings in this domain soon followed, which posed the following question: "How robust are behavioral priming effects, and to what extent are they actually important in shaping people's actions?" To shed a new light on this question, I revisit behavioral priming through the prism of a dynamical systems perspective (DSP). The DSP is a scientific paradigm that has been developed through a combined effort of many different academic disciplines, ranging from mathematics and physics to biology, economics, psychology, etc., and it deals with behavior of simple and complex systems over time. In the present paper, I use conceptual and methodological tools stemming from the DSP to propose circumstances under which behavioral priming effects are likely to occur. More precisely, I outline three possible types of the influence of priming on human behavior, to which I refer as emergence, readjustment, and attractor switch, and propose experimental designs to examine them. Finally, I discuss relevant implications for behavioral priming effects and their replications.

Advanced, Analytic, Automated (AAA) Measurement of Engagement During Learning

It is generally acknowledged that engagement plays a critical role in learning. Unfortunately, the study of engagement has been stymied by a lack of valid and efficient measures. We introduce the advanced, analytic, and automated (AAA) approach to measure engagement at fine-grained temporal resolutions. The AAA measurement approach is grounded in embodied theories of cognition and affect, which advocate a close coupling between thought and action. It uses machine-learned computational models to automatically infer mental states associated with engagement (e.g., interest, flow) from machine-readable behavioral and physiological signals (e.g., facial expressions, eye tracking, click-stream data) and from aspects of the environmental context. We present15 case studies that illustrate the potential of the AAA approach for measuring engagement in digital learning environments. We discuss strengths and weaknesses of the AAA approach, concluding that it has significant promise to catalyze engagement research.