Unsupervised Synchrony Discovery in Human Interaction

Assessing Sensorimotor Synchronisation in Toddlers Using the Lookit Online Experiment Platform and Automated Movement Extraction

Adapting gross motor movement to match the tempo of auditory rhythmic stimulation (sensorimotor synchronisation; SMS) is a complex skill with a long developmental trajectory. Drumming tasks have previously been employed with infants and young children to measure the emergence of rhythmic entrainment, and may provide a tool for identification of those with atypical rhythm perception and production. Here we describe a new protocol for measuring infant rhythmic movement that can be employed at scale. In the current study, 50 two-year-olds drummed along with the audiovisual presentation of four steady rhythms, using videos of isochronous drumming at 400, 500, 600, and 700 ms IOI, and provided their spontaneous motor tempo (SMT) by drumming in silence. Toddlers' drumming is observed from video recordings made in participants' own homes, obtained via the Lookit platform for online infant studies. We use OpenPose deep-learning model to generate wireframe estimates of hand and body location for each video. The vertical displacement of the hand was extracted, and the power and frequency of infants' rhythmic entrainment quantified using Fast Fourier Transforms. We find evidence for age-appropriate tempo-flexibility in our sample. Our results demonstrate the feasibility of a fully digital approach to measuring rhythmic entrainment from within the participant's home, from early in development.

Discovering Synchronized Subsets of Sequences: A Large Scale Solution

Finding the largest subset of sequences (i.e., time series) that are correlated above a certain threshold, within large datasets, is of significant interest for computer vision and pattern recognition problems across domains, including behavior analysis, computational biology, neuroscience, and finance. Maximal clique algorithms can be used to solve this problem, but they are not scalable. We present an approximate, but highly efficient and scalable, method that represents the search space as a union of sets called ϵ-expanded clusters, one of which is theoretically guaranteed to contain the largest subset of synchronized sequences. The method finds synchronized sets by fitting a Euclidean ball on ϵ-expanded clusters, using Jung's theorem. We validate the method on data from the three distinct domains of facial behavior analysis, finance, and neuroscience, where we respectively discover the synchrony among pixels of face videos, stock market item prices, and dynamic brain connectivity data. Experiments show that our method produces results comparable to, but up to 300 times faster than, maximal clique algorithms, with speed gains increasing exponentially with the number of input sequences.

A Branch-and-Bound Framework for Unsupervised Common Event Discovery

Event discovery aims to discover a temporal segment of interest, such as human behavior, actions or activities. Most approaches to event discovery within or between time series use supervised learning. This becomes problematic when some relevant event labels are unknown, are difficult to detect, or not all possible combinations of events have been anticipated. To overcome these problems, this paper explores Common Event Discovery (CED), a new problem that aims to discover common events of variable-length segments in an unsupervised manner. A potential solution to CED is searching over all possible pairs of segments, which would incur a prohibitive quartic cost. In this paper, we propose an efficient branch-and-bound (B&B) framework that avoids exhaustive search while guaranteeing a globally optimal solution. To this end, we derive novel bounding functions for various commonality measures and provide extensions to multiple commonality discovery and accelerated search. The B&B framework takes as input any multidimensional signal that can be quantified into histograms. A generalization of the framework can be readily applied to discover events at the same or different times (synchrony and event commonality, respectively). We consider extensions to video search and supervised event detection. The effectiveness of the B&B framework is evaluated in motion capture of deliberate behavior and in video of spontaneous facial behavior in diverse interpersonal contexts: interviews, small groups of young adults, and parent-infant face-to-face interaction.