Audience facial expressions detected by automated face analysis software reflect emotions in music

Age and familiarity effects on musical memory

BACKGROUND

A common complaint in older adults is trouble with their memory, especially for new information. Current knowledge about normal aging and changes in memory identify a divide between memory tasks that are unaffected by aging and those that are. Among the unaffected are recognition tasks. These memory tasks rely on accessing well-known information, often include environmental support, and tend to be automatic. Negative age effects on memory are often observed at both encoding and during recall. Older adults often have difficulty with recall tasks, particularly those that require effortful self-initiated processing, episodic memory, and retention of information about contextual cues. Research in memory for music in healthy aging suggests a skill-invariance hypothesis: that age effects dominate when general-purpose cognitive mechanisms are needed to perform the musical task at hand, while experience effects dominate when music-specific knowledge is needed to perform the task [1].

AIMS

The goals of this pair of studies were to investigate the effects of age and familiarity on musical memory in the context of real pieces of music, and to compare a live concert experimental setting with a lab-based experimental setting.

METHOD

Participants' task was to click a button (or press the spacebar) when they heard the target theme in three pieces of music. One was Mozart's Eine Kleine Nachtmusik and the others were original pieces commissioned for this study, one tonal and one atonal. Participants heard the relevant theme three times before listening to a piece of music. The music was performed by the Newfoundland Symphony Orchestra; participants either attended the concert, or watched a recording of the concert in the lab. Participants also completed two short cognitive tests and filled out a questionnaire collecting demographic information and a hearing abilities self-assessment.

RESULTS

We find a significant effect of familiarity and setting but not of age or musical training on recognition performance as measured by d'. More specifically, performance is best for the familiar, tonal piece, moderate for the unfamiliar tonal piece and worst for the unfamiliar atonal piece. Performance was better in the live setting than the lab setting.

CONCLUSIONS

The absence of an age effect provides encouraging evidence that music's diverse cues may encourage cognitive scaffolding, in turn improving encoding and subsequent recognition. Better performance in an ecological versus lab setting supports the expansion of ecological studies in the field.

Inter-subject correlation of audience facial expressions predicts audience engagement during theatrical performances

During performances, audiences experience various emotional states, and these are reflected in their ongoing facial expressions. We investigated if audience engagement could be determined by measuring the inter-subject correlation (ISC) of non-invasively recorded audience facial expressions. We filmed the faces of multiple audience members at theatrical performances and determined the intensity of their different facial expressions throughout the performances. Neutral, happy, anger, and disgust expression ISCs accounted for up to 24% of the performance dramaturge's predictions of audience engagement. Expression synchrony was greater between individuals in close proximity, suggesting effects of emotional contagion or cognitive similarities between neighboring individuals, whereas expression synchrony was greatest between individuals who were younger, female, and with greater levels of empathy, showing that individual characteristics impact shared audience experiences. Together, our results show that facial expression synchronization could be used as a real-time non-invasive indicator of engagement in audiences larger than achieved using previous approaches.

Closed-Loop Tracking and Regulation of Emotional Valence State From Facial Electromyogram Measurements

Affective studies provide essential insights to address emotion recognition and tracking. In traditional open-loop structures, a lack of knowledge about the internal emotional state makes the system incapable of adjusting stimuli parameters and automatically responding to changes in the brain. To address this issue, we propose to use facial electromyogram measurements as biomarkers to infer the internal hidden brain state as feedback to close the loop. In this research, we develop a systematic way to track and control emotional valence, which codes emotions as being pleasant or obstructive. Hence, we conduct a simulation study by modeling and tracking the subject's emotional valence dynamics using state-space approaches. We employ Bayesian filtering to estimate the person-specific model parameters along with the hidden valence state, using continuous and binary features extracted from experimental electromyogram measurements. Moreover, we utilize a mixed-filter estimator to infer the secluded brain state in a real-time simulation environment. We close the loop with a fuzzy logic controller in two categories of regulation: inhibition and excitation. By designing a control action, we aim to automatically reflect any required adjustments within the simulation and reach the desired emotional state levels. Final results demonstrate that, by making use of physiological data, the proposed controller could effectively regulate the estimated valence state. Ultimately, we envision future outcomes of this research to support alternative forms of self-therapy by using wearable machine interface architectures capable of mitigating periods of pervasive emotions and maintaining daily well-being and welfare.