EEG-based neurocinematics: challenges and prospects

Consistent movement of viewers' facial keypoints while watching emotionally evocative videos

Neuropsychological research aims to unravel how diverse individuals' brains exhibit similar functionality when exposed to the same stimuli. The evocation of consistent responses when different subjects watch the same emotionally evocative stimulus has been observed through modalities like fMRI, EEG, physiological signals and facial expressions. We refer to the quantification of these shared consistent signals across subjects at each time instant across the temporal dimension as Consistent Response Measurement (CRM). CRM is widely explored through fMRI, occasionally with EEG, physiological signals and facial expressions using metrics like Inter-Subject Correlation (ISC). However, fMRI tools are expensive and constrained, while EEG and physiological signals are prone to facial artifacts and environmental conditions (such as temperature, humidity, and health condition of subjects). In this research, facial expression videos are used as a cost-effective and flexible alternative for CRM, minimally affected by external conditions. By employing computer vision-based automated facial keypoint tracking, a new metric similar to ISC, called the Average t-statistic, is introduced. Unlike existing facial expression-based methodologies that measure CRM of secondary indicators like inferred emotions, keypoint, and ICA-based features, the Average t-statistic is closely associated with the direct measurement of consistent facial muscle movement using the Facial Action Coding System (FACS). This is evidenced in DISFA dataset where the time-series of Average t-statistic has a high correlation (R2 = 0.78) with a metric called AU consistency, which directly measures facial muscle movement through FACS coding of video frames. The simplicity of recording facial expressions with the automated Average t-statistic expands the applications of CRM such as measuring engagement in online learning, customer interactions, etc., and diagnosing outliers in healthcare conditions like stroke, autism, depression, etc. To promote further research, we have made the code repository publicly available.

Estimation of Emotional Arousal Changes of a Group of Individuals During Movie Screening Using Steady-State Visual-Evoked Potential

Neurocinematics is an emerging discipline in neuroscience, which aims to provide new filmmaking techniques by analyzing the brain activities of a group of audiences. Several neurocinematics studies attempted to track temporal changes in mental states during movie screening; however, it is still needed to develop efficient and robust electroencephalography (EEG) features for tracking brain states precisely over a long period. This study proposes a novel method for estimating emotional arousal changes in a group of individuals during movie screening by employing steady-state visual evoked potential (SSVEP), which is a widely used EEG response elicited by the presentation of periodic visual stimuli. Previous studies have reported that the emotional arousal of each individual modulates the strength of SSVEP responses. Based on this phenomenon, movie clips were superimposed on a background, eliciting an SSVEP response with a specific frequency. Two emotionally arousing movie clips were presented to six healthy male participants, while EEG signals were recorded from the occipital channels. We then investigated whether the movie scenes that elicited higher SSVEP responses coincided well with those rated as the most impressive scenes by 37 viewers in a separate experimental session. Our results showed that the SSVEP response averaged across six participants could accurately predict the overall impressiveness of each movie, evaluated with a much larger group of individuals.

Chaotic and Fast Audiovisuals Increase Attentional Scope but Decrease Conscious Processing

Audiovisual cuts involve spatial, temporal, and action narrative leaps. They can even change the meaning of the narrative through film editing. Many cuts are not consciously perceived, others are, just as we perceive or not the changes in real events. In this paper, we analyze the effects of cuts and different editing styles on 36 subjects, using electroencephalographic (EEG) techniques and the projection of stimuli with different audiovisual style of edition but the same narrative. Eyeblinks, event-related potentials (ERPs), EEG spectral power and disturbances, and the functional and effective connectivity before and after the cuts were analyzed. Cuts decreased blink frequency in the first second following them. Cuts also caused an increase of the alpha rhythm, with a cortical evolution from visual toward rostral areas. There were marked differences between a video-clip editing style, with greater activities evoked in visual areas, and the classic continuous style of editing, which presented greater activities in the frontal zones. This was reflected by differences in the theta rhythm between 200 and 400 ms, in visual and frontal zones, and can be connected to the different demands that each style of edition makes on working memory and conscious processing after cutting. Also, at the time of cuts, the causality between visual, somatosensory, and frontal networks is altered in any editing style. Our findings suggest that cuts affect media perception and chaotic and fast audiovisuals increase attentional scope but decrease conscious processing.