Audience preferences are predicted by temporal reliability of neural processing

Brain-to-Brain Synchrony Tracks Real-World Dynamic Group Interactions in the Classroom

The human brain has evolved for group living [1]. Yet we know so little about how it supports dynamic group interactions that the study of real-world social exchanges has been dubbed the "dark matter of social neuroscience" [2]. Recently, various studies have begun to approach this question by comparing brain responses of multiple individuals during a variety of (semi-naturalistic) tasks [3-15]. These experiments reveal how stimulus properties [13], individual differences [14], and contextual factors [15] may underpin similarities and differences in neural activity across people. However, most studies to date suffer from various limitations: they often lack direct face-to-face interaction between participants, are typically limited to dyads, do not investigate social dynamics across time, and, crucially, they rarely study social behavior under naturalistic circumstances. Here we extend such experimentation drastically, beyond dyads and beyond laboratory walls, to identify neural markers of group engagement during dynamic real-world group interactions. We used portable electroencephalogram (EEG) to simultaneously record brain activity from a class of 12 high school students over the course of a semester (11 classes) during regular classroom activities (Figures 1A-1C; Supplemental Experimental Procedures, section S1). A novel analysis technique to assess group-based neural coherence demonstrates that the extent to which brain activity is synchronized across students predicts both student class engagement and social dynamics. This suggests that brain-to-brain synchrony is a possible neural marker for dynamic social interactions, likely driven by shared attention mechanisms. This study validates a promising new method to investigate the neuroscience of group interactions in ecologically natural settings.

A resource for assessing information processing in the developing brain using EEG and eye tracking

We present a dataset combining electrophysiology and eye tracking intended as a resource for the investigation of information processing in the developing brain. The dataset includes high-density task-based and task-free EEG, eye tracking, and cognitive and behavioral data collected from 126 individuals (ages: 6–44). The task battery spans both the simple/complex and passive/active dimensions to cover a range of approaches prevalent in modern cognitive neuroscience. The active task paradigms facilitate principled deconstruction of core components of task performance in the developing brain, whereas the passive paradigms permit the examination of intrinsic functional network activity during varying amounts of external stimulation. Alongside these neurophysiological data, we include an abbreviated cognitive test battery and questionnaire-based measures of psychiatric functioning. We hope that this dataset will lead to the development of novel assays of neural processes fundamental to information processing, which can be used to index healthy brain development as well as detect pathologic processes.

EEG in the classroom: Synchronised neural recordings during video presentation

We performed simultaneous recordings of electroencephalography (EEG) from multiple students in a classroom, and measured the inter-subject correlation (ISC) of activity evoked by a common video stimulus. The neural reliability, as quantified by ISC, has been linked to engagement and attentional modulation in earlier studies that used high-grade equipment in laboratory settings. Here we reproduce many of the results from these studies using portable low-cost equipment, focusing on the robustness of using ISC for subjects experiencing naturalistic stimuli. The present data shows that stimulus-evoked neural responses, known to be modulated by attention, can be tracked for groups of students with synchronized EEG acquisition. This is a step towards real-time inference of engagement in the classroom.

Collective Behaviour in Video Viewing: A Thermodynamic Analysis of Gaze Position

Videos and commercials produced for large audiences can elicit mixed opinions. We wondered whether this diversity is also reflected in the way individuals watch the videos. To answer this question, we presented 65 commercials with high production value to 25 individuals while recording their eye movements, and asked them to provide preference ratings for each video. We find that gaze positions for the most popular videos are highly correlated. To explain the correlations of eye movements, we model them as "interactions" between individuals. A thermodynamic analysis of these interactions shows that they approach a "critical" point such that any stronger interaction would put all viewers into lock-step and any weaker interaction would fully randomise patterns. At this critical point, groups with similar collective behaviour in viewing patterns emerge while maintaining diversity between groups. Our results suggest that popularity of videos is already evident in the way we look at them, and that we maintain diversity in viewing behaviour even as distinct patterns of groups emerge. Our results can be used to predict popularity of videos and commercials at the population level from the collective behaviour of the eye movements of a few viewers.

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.

Time-domain analysis of neural tracking of hierarchical linguistic structures

When listening to continuous speech, cortical activity measured by MEG concurrently follows the rhythms of multiple linguistic structures, e.g., syllables, phrases, and sentences. This phenomenon was previously characterized in the frequency domain. Here, we investigate the waveform of neural activity tracking linguistic structures in the time domain and quantify the coherence of neural response phases over subjects listening to the same stimulus. These analyses are achieved by decomposing the multi-channel MEG recordings into components that maximize the correlation between neural response waveforms across listeners. Each MEG component can be viewed as the recording from a virtual sensor that is spatially tuned to a cortical network showing coherent neural activity over subjects. This analysis reveals information not available from previous frequency-domain analysis of MEG global field power: First, concurrent neural tracking of hierarchical linguistic structures emerges at the beginning of the stimulus, rather than slowly building up after repetitions of the same sentential structure. Second, neural tracking of the sentential structure is reflected by slow neural fluctuations, rather than, e.g., a series of short-lasting transient responses at sentential boundaries. Lastly and most importantly, it shows that the MEG responses tracking the syllabic rhythm are spatially separable from the MEG responses tracking the sentential and phrasal rhythms.

Adolescents growing up amidst intractable conflict attenuate brain response to pain of outgroup

Adolescents' participation in intergroup conflicts comprises an imminent global risk, and understanding its neural underpinnings may open new perspectives. We assessed Jewish-Israeli and Arab-Palestinian adolescents for brain response to the pain of ingroup/outgroup protagonists using magnetoencephalography (MEG), one-on-one positive and conflictual interactions with an outgroup member, attitudes toward the regional conflict, and oxytocin levels. A neural marker of ingroup bias emerged, expressed via alpha modulations in the somatosensory cortex (S1) that characterized an automatic response to the pain of all protagonists followed by rebound/enhancement to ingroup pain only. Adolescents' hostile social interactions with outgroup members and uncompromising attitudes toward the conflict influenced this neural marker. Furthermore, higher oxytocin levels in the Jewish-Israeli majority and tighter brain-to-brain synchrony among group members in the Arab-Palestinian minority enhanced the neural ingroup bias. Findings suggest that in cases of intractable intergroup conflict, top-down control mechanisms may block the brain's evolutionary-ancient resonance to outgroup pain, pinpointing adolescents' interpersonal and sociocognitive processes as potential targets for intervention.

Memorable Audiovisual Narratives Synchronize Sensory and Supramodal Neural Responses

Our brains integrate information across sensory modalities to generate perceptual experiences and form memories. However, it is difficult to determine the conditions under which multisensory stimulation will benefit or hinder the retrieval of everyday experiences. We hypothesized that the determining factor is the reliability of information processing during stimulus presentation, which can be measured through intersubject correlation of stimulus-evoked activity. We therefore presented biographical auditory narratives and visual animations to 72 human subjects visually, auditorily, or combined, while neural activity was recorded using electroencephalography. Memory for the narrated information, contained in the auditory stream, was tested 3 weeks later. While the visual stimulus alone led to no meaningful retrieval, this related stimulus improved memory when it was combined with the story, even when it was temporally incongruent with the audio. Further, individuals with better subsequent memory elicited neural responses during encoding that were more correlated with their peers. Surprisingly, portions of this predictive synchronized activity were present regardless of the sensory modality of the stimulus. These data suggest that the strength of sensory and supramodal activity is predictive of memory performance after 3 weeks, and that neural synchrony may explain the mnemonic benefit of the functionally uninformative visual context observed for these real-world stimuli.

Variability in functional brain networks predicts expertise during action observation

Observing an action performed by another individual activates, in the observer, similar circuits as those involved in the actual execution of that action. This activation is modulated by prior experience; indeed, sustained training in a particular motor domain leads to structural and functional changes in critical brain areas. Here, we capitalized on a novel graph-theory approach to electroencephalographic data (Fraiman et al., 2016) to test whether variability in functional brain networks implicated in Tango observation can discriminate between groups differing in their level of expertise. We found that experts and beginners significantly differed in the functional organization of task-relevant networks. Specifically, networks in expert Tango dancers exhibited less variability and a more robust functional architecture. Notably, these expertise-dependent effects were captured within networks derived from electrophysiological brain activity recorded in a very short time window (2s). In brief, variability in the organization of task-related networks seems to be a highly sensitive indicator of long-lasting training effects. This finding opens new methodological and theoretical windows to explore the impact of domain-specific expertise on brain plasticity, while highlighting variability as a fruitful measure in neuroimaging research.

Attention Strongly Modulates Reliability of Neural Responses to Naturalistic Narrative Stimuli

Attentional engagement is a major determinant of how effectively we gather information through our senses. Alongside the sheer growth in the amount and variety of information content that we are presented with through modern media, there is increased variability in the degree to which we "absorb" that information. Traditional research on attention has illuminated the basic principles of sensory selection to isolated features or locations, but it provides little insight into the neural underpinnings of our attentional engagement with modern naturalistic content. Here, we show in human subjects that the reliability of an individual's neural responses with respect to a larger group provides a highly robust index of the level of attentional engagement with a naturalistic narrative stimulus. Specifically, fast electroencephalographic evoked responses were more strongly correlated across subjects when naturally attending to auditory or audiovisual narratives than when attention was directed inward to a mental arithmetic task during stimulus presentation. This effect was strongest for audiovisual stimuli with a cohesive narrative and greatly reduced for speech stimuli lacking meaning. For compelling audiovisual narratives, the effect is remarkably strong, allowing perfect discrimination between attentional state across individuals. Control experiments rule out possible confounds related to altered eye movement trajectories or order of presentation. We conclude that reliability of evoked activity reproduced across subjects viewing the same movie is highly sensitive to the attentional state of the viewer and listener, which is aided by a cohesive narrative.

Functional brain imaging predicts public health campaign success

Mass media can powerfully affect health decision-making. Pre-testing through focus groups or surveys is a standard, though inconsistent, predictor of effectiveness. Converging evidence demonstrates that activity within brain systems associated with self-related processing can predict individual behavior in response to health messages. Preliminary evidence also suggests that neural activity in small groups can forecast population-level campaign outcomes. Less is known about the psychological processes that link neural activity and population-level outcomes, or how these predictions are affected by message content. We exposed 50 smokers to antismoking messages and used their aggregated neural activity within a 'self-localizer' defined region of medial prefrontal cortex to predict the success of the same campaign messages at the population level (n = 400,000 emails). Results demonstrate that: (i) independently localized neural activity during health message exposure complements existing self-report data in predicting population-level campaign responses (model combined R(2) up to 0.65) and (ii) this relationship depends on message content-self-related neural processing predicts outcomes in response to strong negative arguments against smoking and not in response to compositionally similar neutral images. These data advance understanding of the psychological link between brain and large-scale behavior and may aid the construction of more effective media health campaigns.

Detecting and interpreting conscious experiences in behaviorally non-responsive patients

Decoding the contents of consciousness from brain activity is one of the most challenging frontiers of cognitive neuroscience. The ability to interpret mental content without recourse to behavior is most relevant for understanding patients who may be demonstrably conscious, but entirely unable to speak or move willfully in any way, precluding any systematic investigation of their conscious experience. The lack of consistent behavioral responsivity engenders unique challenges to decoding any conscious experiences these patients may have solely based on their brain activity. For this reason, paradigms that have been successful in healthy individuals cannot serve to interpret conscious mental states in this patient group. Until recently, patient studies have used structured instructions to elicit willful modulation of brain activity according to command, in order to decode the presence of willful brain-based responses in this patient group. In recent work, we have used naturalistic paradigms, such as watching a movie or listening to an audio-story, to demonstrate that a common neural code supports conscious experiences in different individuals. Moreover, we have demonstrated that this code can be used to interpret the conscious experiences of a patient who had remained non-responsive for several years. This approach is easy to administer, brief, and does not require compliance with task instructions. Rather, it engages attention naturally through meaningful stimuli that are similar to the real-world sensory information in a patient's environment. Therefore, it may be particularly suited to probing consciousness and revealing residual brain function in highly impaired, acute, patients in a comatose state, thus helping to improve diagnostication and prognostication for this vulnerable patient group from the critical early stages of severe brain-injury.

Cortical Components of Reaction-Time during Perceptual Decisions in Humans

The mechanisms of perceptual decision-making are frequently studied through measurements of reaction time (RT). Classical sequential-sampling models (SSMs) of decision-making posit RT as the sum of non-overlapping sensory, evidence accumulation, and motor delays. In contrast, recent empirical evidence hints at a continuous-flow paradigm in which multiple motor plans evolve concurrently with the accumulation of sensory evidence. Here we employ a trial-to-trial reliability-based component analysis of encephalographic data acquired during a random-dot motion task to directly image continuous flow in the human brain. We identify three topographically distinct neural sources whose dynamics exhibit contemporaneous ramping to time-of-response, with the rate and duration of ramping discriminating fast and slow responses. Only one of these sources, a parietal component, exhibits dependence on strength-of-evidence. The remaining two components possess topographies consistent with origins in the motor system, and their covariation with RT overlaps in time with the evidence accumulation process. After fitting the behavioral data to a popular SSM, we find that the model decision variable is more closely matched to the combined activity of the three components than to their individual activity. Our results emphasize the role of motor variability in shaping RT distributions on perceptual decision tasks, suggesting that physiologically plausible computational accounts of perceptual decision-making must model the concurrent nature of evidence accumulation and motor planning.

Multiview Bayesian Correlated Component Analysis

Correlated component analysis as proposed by Dmochowski, Sajda, Dias, and Parra (2012) is a tool for investigating brain process similarity in the responses to multiple views of a given stimulus. Correlated components are identified under the assumption that the involved spatial networks are identical. Here we propose a hierarchical probabilistic model that can infer the level of universality in such multiview data, from completely unrelated representations, corresponding to canonical correlation analysis, to identical representations as in correlated component analysis. This new model, which we denote Bayesian correlated component analysis, evaluates favorably against three relevant algorithms in simulated data. A well-established benchmark EEG data set is used to further validate the new model and infer the variability of spatial representations across multiple subjects.

Idiosyncratic responding during movie-watching predicted by age differences in attentional control

Much is known about how age affects the brain during tightly controlled, though largely contrived, experiments, but do these effects extrapolate to everyday life? Naturalistic stimuli, such as movies, closely mimic the real world and provide a window onto the brain's ability to respond in a timely and measured fashion to complex, everyday events. Young adults respond to these stimuli in a highly synchronized fashion, but it remains to be seen how age affects neural responsiveness during naturalistic viewing. To this end, we scanned a large (N = 218), population-based sample from the Cambridge Centre for Ageing and Neuroscience (Cam-CAN) during movie-watching. Intersubject synchronization declined with age, such that older adults' response to the movie was more idiosyncratic. This decreased synchrony related to cognitive measures sensitive to attentional control. Our findings suggest that neural responsivity changes with age, which likely has important implications for real-world event comprehension and memory.

Neural Affective Mechanisms Predict Market-Level Microlending

Humans sometimes share with others whom they may never meet or know, in violation of the dictates of pure self-interest. Research has not established which neuropsychological mechanisms support lending decisions, nor whether their influence extends to markets involving significant financial incentives. In two studies, we found that neural affective mechanisms influence the success of requests for microloans. In a large Internet database of microloan requests (N = 13,500), we found that positive affective features of photographs promoted the success of those requests. We then established that neural activity (i.e., in the nucleus accumbens) and self-reported positive arousal in a neuroimaging sample (N = 28) predicted the success of loan requests on the Internet, above and beyond the effects of the neuroimaging sample's own choices (i.e., to lend or not). These findings suggest that elicitation of positive arousal can promote the success of loan requests, both in the laboratory and on the Internet. They also highlight affective neuroscience's potential to probe neuropsychological mechanisms that drive microlending, enhance the effectiveness of loan requests, and forecast market-level behavior.

Cortical Response Similarities Predict which Audiovisual Clips Individuals Viewed, but Are Unrelated to Clip Preference

Cortical responses to complex natural stimuli can be isolated by examining the relationship between neural measures obtained while multiple individuals view the same stimuli. These inter-subject correlation’s (ISC’s) emerge from similarities in individual’s cortical response to the shared audiovisual inputs, which may be related to their emergent cognitive and perceptual experience. Within the present study, our goal is to examine the utility of using ISC’s for predicting which audiovisual clips individuals viewed, and to examine the relationship between neural responses to natural stimuli and subjective reports. The ability to predict which clips individuals viewed depends on the relationship of the EEG response across subjects and the nature in which this information is aggregated. We conceived of three approaches for aggregating responses, i.e. three assignment algorithms, which we evaluated in Experiment 1A. The aggregate correlations algorithm generated the highest assignment accuracy (70.83% chance = 33.33%) and was selected as the assignment algorithm for the larger sample of individuals and clips within Experiment 1B. The overall assignment accuracy was 33.46% within Experiment 1B (chance = 06.25%), with accuracies ranging from 52.9% (Silver Linings Playbook) to 11.75% (Seinfeld) within individual clips. ISC’s were significantly greater than zero for 15 out of 16 clips, and fluctuations within the delta frequency band (i.e. 0-4 Hz) primarily contributed to response similarities across subjects. Interestingly, there was insufficient evidence to indicate that individuals with greater similarities in clip preference demonstrate greater similarities in cortical responses, suggesting a lack of association between ISC and clip preference. Overall these results demonstrate the utility of using ISC’s for prediction, and further characterize the relationship between ISC magnitudes and subjective reports.

From the eyes and the heart: a novel eye-gaze metric that predicts video preferences of a large audience

Eye-tracking has been extensively used to quantify audience preferences in the context of marketing and advertising research, primarily in methodologies involving static images or stimuli (i.e., advertising, shelf testing, and website usability). However, these methodologies do not generalize to narrative-based video stimuli where a specific storyline is meant to be communicated to the audience. In this paper, a novel metric based on eye-gaze dispersion (both within and across viewings) that quantifies the impact of narrative-based video stimuli to the preferences of large audiences is presented. The metric is validated in predicting the performance of video advertisements aired during the 2014 Super Bowl final. In particular, the metric is shown to explain 70% of the variance in likeability scores of the 2014 Super Bowl ads as measured by the USA TODAY Ad-Meter. In addition, by comparing the proposed metric with Heart Rate Variability (HRV) indices, we have associated the metric with biological processes relating to attention allocation. The underlying idea behind the proposed metric suggests a shift in perspective when it comes to evaluating narrative-based video stimuli. In particular, it suggests that audience preferences on video are modulated by the level of viewers lack of attention allocation. The proposed metric can be calculated on any narrative-based video stimuli (i.e., movie, narrative content, emotional content, etc.), and thus has the potential to facilitate the use of such stimuli in several contexts: prediction of audience preferences of movies, quantitative assessment of entertainment pieces, prediction of the impact of movie trailers, identification of group, and individual differences in the study of attention-deficit disorders, and the study of desensitization to media violence.

Evaluation of TV commercials using neurophysiological responses

Background

In recent years, neuroscientific knowledge has been applied to marketing as a novel and efficient means to comprehend the cognitive and behavioral aspects of consumers. A number of studies have attempted to evaluate media contents, especially TV commercials using various neuroimaging techniques such as electroencephalography (EEG). Yet neurophysiological examination of detailed cognitive and affective responses in viewers is still required to provide practical information to marketers. Here, this study develops a method to analyze temporal patterns of EEG data and extract affective and cognitive indices such as happiness, surprise, and attention for TV commercial evaluation.

Methods

Twenty participants participated in the study. We developed the neurophysiological indices for TV commercial evaluation using classification model. Specifically, these model-based indices were customized using individual EEG features. We used a video game for developing the index of attention and four video clips for developing indices of happiness and surprise. Statistical processes including one-way analyses of variance (ANOVA) and the cross validation scheme were used to select EEG features for each index. The EEG features were composed of the combinations of spectral power at selected channels from the cross validation for each individual. The Fisher’s linear discriminant classifier (FLDA) was used to estimate each neurophysiological index during viewing four different TV commercials. Post hoc behavioral responses of preference, short-term memory, and recall were measured.

Results

Behavioral results showed significant differences for all preference, short-term memory rates, and recall rates between commercials, leading to a ‘high-ranked’ commercial group and a ‘low-ranked’ group (P < 0.05). Neural estimation of happiness results revealed a significant difference between the high-ranked and the low-ranked commercials in happiness index (P < 0.01). The order of rankings based on happiness and attention matched well with the order of behavioral response rankings. In the elapsed-time analysis of the highest-ranked commercial, we could point to visual and auditory semantic structures of the commercial that induced increases in the happiness index.

Conclusions

Our results demonstrated that the neurophysiological indices developed in this study may provide a useful tool for evaluating TV commercials.

Engaged listeners: shared neural processing of powerful political speeches

Powerful speeches can captivate audiences, whereas weaker speeches fail to engage their listeners. What is happening in the brains of a captivated audience? Here, we assess audience-wide functional brain dynamics during listening to speeches of varying rhetorical quality. The speeches were given by German politicians and evaluated as rhetorically powerful or weak. Listening to each of the speeches induced similar neural response time courses, as measured by inter-subject correlation analysis, in widespread brain regions involved in spoken language processing. Crucially, alignment of the time course across listeners was stronger for rhetorically powerful speeches, especially for bilateral regions of the superior temporal gyri and medial prefrontal cortex. Thus, during powerful speeches, listeners as a group are more coupled to each other, suggesting that powerful speeches are more potent in taking control of the listeners' brain responses. Weaker speeches were processed more heterogeneously, although they still prompted substantially correlated responses. These patterns of coupled neural responses bear resemblance to metaphors of resonance, which are often invoked in discussions of speech impact, and contribute to the literature on auditory attention under natural circumstances. Overall, this approach opens up possibilities for research on the neural mechanisms mediating the reception of entertaining or persuasive messages.

Maximally reliable spatial filtering of steady state visual evoked potentials

Due to their high signal-to-noise ratio (SNR) and robustness to artifacts, steady state visual evoked potentials (SSVEPs) are a popular technique for studying neural processing in the human visual system. SSVEPs are conventionally analyzed at individual electrodes or linear combinations of electrodes which maximize some variant of the SNR. Here we exploit the fundamental assumption of evoked responses--reproducibility across trials--to develop a technique that extracts a small number of high SNR, maximally reliable SSVEP components. This novel spatial filtering method operates on an array of Fourier coefficients and projects the data into a low-dimensional space in which the trial-to-trial spectral covariance is maximized. When applied to two sample data sets, the resulting technique recovers physiologically plausible components (i.e., the recovered topographies match the lead fields of the underlying sources) while drastically reducing the dimensionality of the data (i.e., more than 90% of the trial-to-trial reliability is captured in the first four components). Moreover, the proposed technique achieves a higher SNR than that of the single-best electrode or the Principal Components. We provide a freely-available MATLAB implementation of the proposed technique, herein termed "Reliable Components Analysis".