The Auditory Kuleshov Effect: Multisensory Integration in Movie Editing

Reexamining the Kuleshov effect: Behavioral and neural evidence from authentic film experiments

Film cognition explores the influence of cinematic elements, such as editing and film color, on viewers' perception. The Kuleshov effect, a famous example of how editing influences viewers' emotional perception, was initially proposed to support montage theory through the Kuleshov experiment. This effect, which has since been recognized as a manifestation of point-of-view (POV) editing practices, posits that the emotional interpretation of neutral facial expressions is influenced by the accompanying emotional scene in a face-scene-face sequence. However, concerns persist regarding the validity of previous studies, often employing inauthentic film materials like static images, leaving the question of its existence in authentic films unanswered. This study addresses these concerns by utilizing authentic films in two experiments. In Experiment 1, multiple film clips were captured under the guidance of a professional film director and seamlessly integrated into authentic film sequences. 59 participants viewed these face-scene-face film sequences and were tasked with rating the valence and emotional intensity of neutral faces. The findings revealed that the accompanying fearful or happy scenes significantly influence the interpretation of emotion on neutral faces, eliciting perceptions of negative or positive emotions from the neutral face. These results affirm the existence of the Kuleshov effect within authentic films. In Experiment 2, 31 participants rated the valence and arousal of neutral faces while undergoing functional magnetic resonance imaging (fMRI). The behavioral results confirm the Kuleshov effect in the MRI scanner, while the neural data identify neural correlates that support its existence at the neural level. These correlates include the cuneus, precuneus, hippocampus, parahippocampal gyrus, post cingulate gyrus, orbitofrontal cortex, fusiform gyrus, and insula. These findings also underscore the contextual framing inherent in the Kuleshov effect. Overall, the study integrates film theory and cognitive neuroscience experiments, providing robust evidence supporting the existence of the Kuleshov effect through both subjective ratings and objective neuroimaging measurements. This research also contributes to a deeper understanding of the impact of film editing on viewers' emotional perception from the contemporary POV editing practices and neurocinematic perspective, advancing the knowledge of film cognition.

Analysis and Classification of Music-Induced States of Sadness

The enjoyment and pleasure derived from sad music has sparked fascination among researchers due to its seemingly paradoxical nature in producing positive affect. Research is yet to develop a comprehensive understanding of this “paradox.” Contradictory findings have resulted in a great variability within the literature, meaning results and interpretations can be difficult to derive. Consequently, this review collated the current literature, seeking to utilize the variability in the findings to propose a model of differential sad states, providing a means for past and future findings to be interpreted. The proposed model is based on theoretical understanding, as such it requires full empirical support. Comparisons to alternative models, theoretical, clinical, and cognitive implications, as well as future directions are discussed.

Multimodal Recognition of Emotions in Music and Facial Expressions

The aim of the study was to investigate the neural processing of congruent vs. incongruent affective audiovisual information (facial expressions and music) by means of ERPs (Event Related Potentials) recordings. Stimuli were 200 infant faces displaying Happiness, Relaxation, Sadness, Distress and 32 piano musical pieces conveying the same emotional states (as specifically assessed). Music and faces were presented simultaneously, and paired so that in half cases they were emotionally congruent or incongruent. Twenty subjects were told to pay attention and respond to infrequent targets (adult neutral faces) while their EEG was recorded from 128 channels. The face-related N170 (160-180 ms) component was the earliest response affected by the emotional content of faces (particularly by distress), while visual P300 (250-450 ms) and auditory N400 (350-550 ms) responses were specifically modulated by the emotional content of both facial expressions and musical pieces. Face/music emotional incongruence elicited a wide N400 negativity indicating the detection of a mismatch in the expressed emotion. A swLORETA inverse solution applied to N400 (difference wave Incong. - Cong.), showed the crucial role of Inferior and Superior Temporal Gyri in the multimodal representation of emotional information extracted from faces and music. Furthermore, the prefrontal cortex (superior and medial, BA 10) was also strongly active, possibly supporting working memory. The data hints at a common system for representing emotional information derived by social cognition and music processing, including uncus and cuneus.

How Context Influences Our Perception of Emotional Faces: A Behavioral Study on the Kuleshov Effect

Facial expressions are of major importance in understanding the mental and emotional states of others. So far, most studies on the perception and comprehension of emotions have used isolated facial expressions as stimuli; for example, photographs of actors displaying facial expressions corresponding to one of the so called ‘basic emotions.’ However, our real experience during social interactions is different: facial expressions of emotion are mostly perceived in a wider context, constituted by body language, the surrounding environment, and our beliefs and expectations. Already in the early twentieth century, the Russian filmmaker Lev Kuleshov argued that such context, established by intermediate shots of strong emotional content, could significantly change our interpretation of facial expressions in film. Prior experiments have shown behavioral effects pointing in this direction, but have only used static images as stimuli. Our study used a more ecological design with participants watching film sequences of neutral faces, crosscut with scenes of strong emotional content (evoking happiness or fear, plus neutral stimuli as a baseline condition). The task was to rate the emotion displayed by a target person’s face in terms of valence, arousal, and category. Results clearly demonstrated the presence of a significant effect in terms of both valence and arousal in the fear condition only. Moreover, participants tended to categorize the target person’s neutral facial expression choosing the emotion category congruent with the preceding context. Our results highlight the context-sensitivity of emotions and the importance of studying them under ecologically valid conditions.