The perceptual neural trace of memorable unseen scenes

Unveiling the neural dynamics of conscious perception in rapid object recognition

Our brain excels at recognizing objects, even when they flash by in a rapid sequence. However, the neural processes determining whether a target image in a rapid sequence can be recognized or not remains elusive. We used electroencephalography (EEG) to investigate the temporal dynamics of brain processes that shape perceptual outcomes in these challenging viewing conditions. Using naturalistic images and advanced multivariate pattern analysis (MVPA) techniques, we probed the brain dynamics governing conscious object recognition. Our results show that although initially similar, the processes for when an object can or cannot be recognized diverge around 180 ms post-appearance, coinciding with feedback neural processes. Decoding analyses indicate that gist perception (partial conscious perception) can occur at ∼120 ms through feedforward mechanisms. In contrast, object identification (full conscious perception of the image) is resolved at ∼190 ms after target onset, suggesting involvement of recurrent processing. These findings underscore the importance of recurrent neural connections in object recognition and awareness in rapid visual presentations.

Modeling short visual events through the BOLD moments video fMRI dataset and metadata

Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos' extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.

Perceptual encoding benefit of visual memorability on visual memory formation

Human observers often exhibit remarkable consistency in remembering specific visual details, such as certain face images. This phenomenon is commonly attributed to visual memorability, a collection of stimulus attributes that enhance the long-term retention of visual information. However, the exact contributions of visual memorability to visual memory formation remain elusive as these effects could emerge anywhere from early perceptual encoding to post-perceptual memory consolidation processes. To clarify this, we tested three key predictions from the hypothesis that visual memorability facilitates early perceptual encoding that supports the formation of visual short-term memory (VSTM) and the retention of visual long-term memory (VLTM). First, we examined whether memorability benefits in VSTM encoding manifest early, even within the constraints of a brief stimulus presentation (100-200 ms; Experiment 1). We achieved this by manipulating stimulus presentation duration in a VSTM change detection task using face images with high- or low-memorability while ensuring they were equally familiar to the participants. Second, we assessed whether this early memorability benefit increases the likelihood of VSTM retention, even with post-stimulus masking designed to interrupt post-perceptual VSTM consolidation processes (Experiment 2). Last, we investigated the durability of memorability benefits by manipulating memory retention intervals from seconds to 24 h (Experiment 3). Across experiments, our data suggest that visual memorability has an early impact on VSTM formation, persisting across variable retention intervals and predicting subsequent VLTM overnight. Combined, these findings highlight that visual memorability enhances visual memory within 100-200 ms following stimulus onset, resulting in robust memory traces resistant to post-perceptual interruption and long-term forgetting.

Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response

Behavioral and neuroscience studies in humans and primates have shown that memorability is an intrinsic property of an image that predicts its strength of encoding into and retrieval from memory. While previous work has independently probed when or where this memorability effect may occur in the human brain, a description of its spatiotemporal dynamics is missing. Here, we used representational similarity analysis (RSA) to combine functional magnetic resonance imaging (fMRI) with source-estimated magnetoencephalography (MEG) to simultaneously measure when and where the human cortex is sensitive to differences in image memorability. Results reveal that visual perception of High Memorable images, compared to Low Memorable images, recruits a set of regions of interest (ROIs) distributed throughout the ventral visual cortex: a late memorability response (from around 300 ms) in early visual cortex (EVC), inferior temporal cortex, lateral occipital cortex, fusiform gyrus, and banks of the superior temporal sulcus. Image memorability magnitude results are represented after high-level feature processing in visual regions and reflected in classical memory regions in the medial temporal lobe (MTL). Our results present, to our knowledge, the first unified spatiotemporal account of visual memorability effect across the human cortex, further supporting the levels-of-processing theory of perception and memory.

Memoir study: Investigating image memorability across developmental stages

Images have been shown to consistently differ in terms of their memorability in healthy adults: some images stick in one's mind while others are forgotten quickly. Studies have suggested that memorability is an intrinsic, continuous property of a visual stimulus that can be both measured and manipulated. Memory literature suggests that important developmental changes occur throughout adolescence that have an impact on recognition memory, yet the effect that these changes have on image memorability has not yet been investigated. In the current study, we recruited adolescents ages 11-18 (n = 273, mean = 16) to an online visual memory experiment to explore the effects of developmental changes throughout adolescence on image memorability, and determine if memorability findings in adults can be generalized to the adolescent age group. We used the online experiment to calculate adolescent memorability scores for 1,000 natural images, and compared the results to the MemCat dataset-a memorability dataset that is annotated with adult memorability scores (ages 19-27). Our study finds that memorability scores in adolescents and adults are strongly and significantly correlated (Spearman's rank correlation, r = 0.76, p < 0.001). This correlation persists even when comparing adults with developmentally different sub-groups of adolescents (ages 11-14: r = 0.67, p < 0.001; ages 15-18: r = 0.60, p < 0.001). Moreover, the rankings of image categories by mean memorability scores were identical in both adolescents and adults (including the adolescent sub-groups), indicating that broadly, certain image categories are more memorable for both adolescents and adults. Interestingly, however, adolescents experienced significantly higher false alarm rates than adults, supporting studies that show increased impulsivity and reward-seeking behaviour in adolescents. Our results reveal that the memorability of images remains consistent across individuals at different stages of development. This consistency aligns with and strengthens prior research, indicating that memorability is an intrinsic property of images. Our findings open new pathways for applying memorability studies in adolescent populations, with profound implications in fields such as education, marketing, and psychology. Our work paves the way for innovative approaches in these domains, leveraging the consistent nature of image memorability across age groups.

Tracing the emergence of the memorability benefit

Some visual stimuli are consistently better remembered than others across individuals, due to variations in memorability (the stimulus-intrinsic property that determines ease of encoding into visual long-term memory (VLTM)). However, it remains unclear what cognitive processes give rise to this mnemonic benefit. One possibility is that this benefit is imbued within the capacity-limited bottleneck of VLTM encoding, namely visual working memory (VWM). More precisely, memorable stimuli may be preferentially encoded into VLTM because fewer cognitive resources are required to store them in VWM (efficiency hypothesis). Alternatively, memorable stimuli may be more competitive in obtaining cognitive resources than forgettable stimuli, leading to more successful storage in VWM (competitiveness hypothesis). Additionally, the memorability benefit might emerge post-VWM, specifically, if memorable stimuli are less prone to be forgotten (i.e., are "stickier") than forgettable stimuli after they pass through the encoding bottleneck (stickiness hypothesis). To test this, we conducted two experiments to examine how memorability benefits emerge by manipulating the stimulus memorability, set size, and degree of competition among stimuli as participants encoded them in the context of a working memory task. Subsequently, their memory for the encoded stimuli was tested in a VLTM task. In the VWM task, performance was better for memorable stimuli compared to forgettable stimuli, supporting the efficiency hypothesis. In addition, we found that when in direct competition, memorable stimuli were also better at attracting limited VWM resources than forgettable stimuli, supporting the competitiveness hypothesis. However, only the efficiency advantage translated to a performance benefit in VLTM. Lastly, we found that memorable stimuli were less likely to be forgotten after they passed through the encoding bottleneck imposed by VWM, supporting the "stickiness" hypothesis. Thus, our results demonstrate that the memorability benefit develops across multiple cognitive processes.

The background assimilation effect: Facial emotional perception is affected by surrounding stimuli

The perception of facial emotion is not only determined by the physical features of the face itself but also be influenced by the emotional information of the background or surrounding information. However, the details of such effect are not fully understood. Here, the authors tested the perceived emotion of a target face surrounded by stimuli with different levels of emotional valence. In Experiment 1, four types of objects were divided into three groups (negative, unpleasant flowers and unpleasant animals; mildly negative (neutral), houses; positive, pleasant flowers). In Experiment 2, three groups of surrounding faces with different social-emotional valence (negative, neutral, and positive) were formed with the memory of affective personal knowledge. The data from two experiments showed that the perception of facial emotion can be influenced and modulated by the emotional valence of the surrounding stimuli, which can be explained by assimilation: the positive stimuli increased the valence of a target face, while the negative stimuli comparatively decreased it. Furthermore, the neutral stimuli also increased the valence of the target, which could be explained by the social positive effect. Therefore, the process of assimilation is likely to be a high-level emotional cognition rather than a low-level visual perception. The results of this study may help us better understand face perception in realistic scenarios.

The features underlying the memorability of objects

What makes certain images more memorable than others? While much of memory research has focused on participant effects, recent studies using a stimulus-centric perspective have sparked debate on the determinants of memory, including the roles of semantic and visual features and whether the most prototypical or atypical items are best remembered. Prior studies have typically relied on constrained stimulus sets, limiting a generalized view of the features underlying what we remember. Here, we collected more than 1 million memory ratings for a naturalistic dataset of 26,107 object images designed to comprehensively sample concrete objects. We establish a model of object features that is predictive of image memorability and examined whether memorability could be accounted for by the typicality of the objects. We find that semantic features exert a stronger influence than perceptual features on what we remember and that the relationship between memorability and typicality is more complex than a simple positive or negative association alone.

Dissociating the Impact of Memorability on Electrophysiological Correlates of Memory Encoding Success

Despite its unlimited capacity, not all visual information we encounter is encoded into visual long-term memory. Traditionally, variability in encoding success has been ascribed to variability in the types and efficacy of an individual's cognitive processes during encoding. Accordingly, past studies have identified several neural correlates of variability in encoding success, namely, frontal positivity, occipital alpha amplitude, and frontal theta amplitude, by contrasting the electrophysiological signals recorded during successful and failed encoding processes (i.e., subsequent memory). However, recent research demonstrated individuals remember and forget consistent sets of stimuli, thereby elucidating stimulus-intrinsic factors (i.e., memorability) that determine the ease of memory encoding independent of individual-specific variability in encoding processes. The existence of memorability raises the possibility that canonical EEG correlates of subsequent memory may reflect variability in stimulus-intrinsic factors rather than individual-specific encoding processes. To test this, we recorded the EEG correlates of subsequent memory while participants encoded 600 images of real-world objects and assessed the unique contribution of individual-specific and stimulus-intrinsic factors on each EEG correlate. Here, we found that frontal theta amplitude and occipital alpha amplitude were only influenced by individual-specific encoding success, whereas frontal positivity was influenced by stimulus-intrinsic and individual-specific encoding success. Overall, our results offer novel interpretations of canonical EEG correlates of subsequent memory by demonstrating a dissociable impact of stimulus-intrinsic and individual-specific factors of memory encoding success.

Radiogenomics of C9orf72 Expansion Carriers Reveals Global Transposable Element Derepression and Enables Prediction of Thalamic Atrophy and Clinical Impairment

Hexanucleotide repeat expansion (HRE) within C9orf72 is the most common genetic cause of frontotemporal dementia (FTD). Thalamic atrophy occurs in both sporadic and familial FTD but is thought to distinctly affect HRE carriers. Separately, emerging evidence suggests widespread derepression of transposable elements (TEs) in the brain in several neurodegenerative diseases, including C9orf72 HRE-mediated FTD (C9-FTD). Whether TE activation can be measured in peripheral blood and how the reduction in peripheral C9orf72 expression observed in HRE carriers relates to atrophy and clinical impairment remain unknown. We used FreeSurfer software to assess the effects of C9orf72 HRE and clinical diagnosis (n = 78 individuals, male and female) on atrophy of thalamic nuclei. We also generated a novel, human, whole-blood RNA-sequencing dataset to determine the relationships among peripheral C9orf72 expression, TE activation, thalamic atrophy, and clinical severity (n = 114 individuals, male and female). We confirmed global thalamic atrophy and reduced C9orf72 expression in HRE carriers. Moreover, we identified disproportionate atrophy of the right mediodorsal lateral nucleus in HRE carriers and showed that C9orf72 expression associated with clinical severity, independent of thalamic atrophy. Strikingly, we found global peripheral activation of TEs, including the human endogenous LINE-1 element L1HS L1HS levels were associated with atrophy of multiple pulvinar nuclei, a thalamic region implicated in C9-FTD. Integration of peripheral transcriptomic and neuroimaging data from human HRE carriers revealed atrophy of specific thalamic nuclei, demonstrated that C9orf72 levels relate to clinical severity, and identified marked derepression of TEs, including L1HS, which predicted atrophy of FTD-relevant thalamic nuclei.SIGNIFICANCE STATEMENT Pathogenic repeat expansion in C9orf72 is the most frequent genetic cause of FTD and amyotrophic lateral sclerosis (ALS; C9-FTD/ALS). The clinical, neuroimaging, and pathologic features of C9-FTD/ALS are well characterized, whereas the intersections of transcriptomic dysregulation and brain structure remain largely unexplored. Herein, we used a novel radiogenomic approach to examine the relationship between peripheral blood transcriptomics and thalamic atrophy, a neuroimaging feature disproportionately impacted in C9-FTD/ALS. We confirmed reduction of C9orf72 in blood and found broad dysregulation of transposable elements-genetic elements typically repressed in the human genome-in symptomatic C9orf72 expansion carriers, which associated with atrophy of thalamic nuclei relevant to FTD. C9orf72 expression was also associated with clinical severity, suggesting that peripheral C9orf72 levels capture disease-relevant information.

An Open-Source Cognitive Test Battery to Assess Human Attention and Memory

Cognitive test batteries are widely used in diverse research fields, such as cognitive training, cognitive disorder assessment, or brain mechanism understanding. Although they need flexibility according to their usage objectives, most test batteries are not available as open-source software and are not be tuned by researchers in detail. The present study introduces an open-source cognitive test battery to assess attention and memory, using a javascript library, p5.js. Because of the ubiquitous nature of dynamic attention in our daily lives, it is crucial to have tools for its assessment or training. For that purpose, our test battery includes seven cognitive tasks (multiple-objects tracking, enumeration, go/no-go, load-induced blindness, task-switching, working memory, and memorability), common in cognitive science literature. By using the test battery, we conducted an online experiment to collect the benchmark data. Results conducted on 2 separate days showed the high cross-day reliability. Specifically, the task performance did not largely change with the different days. Besides, our test battery captures diverse individual differences and can evaluate them based on the cognitive factors extracted from latent factor analysis. Since we share our source code as open-source software, users can expand and manipulate experimental conditions flexibly. Our test battery is also flexible in terms of the experimental environment, i.e., it is possible to experiment either online or in a laboratory environment.

Content-specific vulnerability of recent episodic memories in Alzheimer's disease

Endel Tulving's episodic memory framework emphasizes the multifaceted re-experiencing of personal events. Indeed, decades of research focused on the experiential nature of episodic memories, usually treating recent episodic memory as a coherent experiential quality. However, recent insights into the functional architecture of the medial temporal lobe show that different types of mnemonic information are segregated into distinct neural pathways in brain circuits empirically associated with episodic memory. Moreover, recent memories do not fade as a whole under conditions of progressive neurodegeneration in these brain circuits, notably in Alzheimer's disease. Instead, certain memory content seem particularly vulnerable from the moment of their encoding while other content can remain memorable consistently across individuals and contexts. We propose that these observations are related to the content-specific functional architecture of the medial temporal lobe and consequently to a content-specific impairment of memory at different stages of the neurodegeneration. To develop Endel Tulving's inspirational legacy further and to advance our understanding of how memory function is affected by neurodegenerative conditions such as Alzheimer's disease, we postulate that it is compelling to focus on the representational content of recent episodic memories.

•

The functional anatomy of episodic memory segregates different memory content.

•

Alzheimer's disease may cause content-specific loss of recent memories

•

Content-specific memorability across individuals changes with Alzheimer's disease.

•

Content-specific assessment could provide new insights into episodic memory in health and disease

Recognition Receiver Operating Characteristic Curves: The Complex Influence of Input Statistics, Memory, and Decision-making

Receiver operating characteristic (ROC) analysis is the standard tool for studying recognition memory. In particular, the curvilinearity and the y-offset of recognition ROC curves have been interpreted as indicative of either memory strength (single-process models) or different memory processes (dual-process model). The distinction between familiarity and recollection has been widely studied in cognitive neuroscience in a variety of conditions, including lesions of different brain regions. We develop a computational model that explicitly shows how performance in recognition memory is affected by a complex and, as yet, underappreciated interplay of various factors, such as stimulus statistics, memory processing, and decision-making. We demonstrate that (1) the factors in the model affect recognition ROC curves in unexpected ways, (2) fitting R and F parameters according to the dual-process model is not particularly useful for understanding the underlying processes, and (3) the variability of recognition ROC curves and the controversies they have caused might be due to the uncontrolled variability in the contributing factors. Although our model is abstract, its functional components can be mapped onto brain regions, which are involved in corresponding functions. This enables us to reproduce and interpret in a coherent framework the diverse effects on recognition memory that have been reported in patients with frontal and hippocampal lesions. To conclude, our work highlights the importance of the rich interplay of a variety of factors in driving recognition memory performance, which has to be taken into account when interpreting recognition ROC curves.

The effect of intrinsic image memorability on recollection and familiarity

Many photographs of real-life scenes are very consistently remembered or forgotten by most people, making these images intrinsically memorable or forgettable. Although machine vision algorithms can predict a given image's memorability very well, nothing is known about the subjective quality of these memories: are memorable images recognized based on strong feelings of familiarity or on recollection of episodic details? We tested people's recognition memory for memorable and forgettable scenes selected from image memorability databases, which contain memorability scores for each image, based on large-scale recognition memory experiments. Specifically, we tested the effect of intrinsic memorability on recollection and familiarity using cognitive computational models based on receiver operating characteristics (ROCs; Experiment 1 and 2) and on remember/know (R/K) judgments (Experiment 2). The ROC data of Experiment 2 indicated that image memorability boosted memory strength, but did not find a specific effect on recollection or familiarity. By contrast, ROC data from Experiment 2, which was designed to facilitate encoding and, in turn, recollection, found evidence for a specific effect of image memorability on recollection. Moreover, R/K judgments showed that, on average, memorability boosts recollection rather than familiarity. However, we also found a large degree of variability in these judgments across individual images: some images actually achieved high recognition rates by exclusively boosting familiarity rather than recollection. Together, these results show that current machine vision algorithms that can predict an image's intrinsic memorability in terms of hit rates fall short of describing the subjective quality of human memories.

Memorability of words in arbitrary verbal associations modulates memory retrieval in the anterior temporal lobe

Despite large individual differences in memory performance, people remember certain stimuli with overwhelming consistency. This phenomenon is referred to as the memorability of an individual item. However, it remains unknown whether memorability also affects our ability to retrieve associations between items. Here, using a paired-associates verbal memory task, we combine behavioural data, computational modelling and direct recordings from the human brain to examine how memorability influences associative memory retrieval. We find that certain words are correctly retrieved across participants irrespective of the cues used to initiate memory retrieval. These words, which share greater semantic similarity with other words, are more readily available during retrieval and lead to more intrusions when retrieval fails. Successful retrieval of these memorable items, relative to less memorable ones, results in faster reinstatement of neural activity in the anterior temporal lobe. Collectively, our data reveal how the brain prioritizes certain information to facilitate memory retrieval.

Neural activity underlying the detection of an object movement by an observer during forward self-motion: Dynamic decoding and temporal evolution of directional cortical connectivity

Relatively little is known about how the human brain identifies movement of objects while the observer is also moving in the environment. This is, ecologically, one of the most fundamental motion processing problems, critical for survival. To study this problem, we used a task which involved nine textured spheres moving in depth, eight simulating the observer's forward motion while the ninth, the target, moved independently with a different speed towards or away from the observer. Capitalizing on the high temporal resolution of magnetoencephalography (MEG) we trained a Support Vector Classifier (SVC) using the sensor-level data to identify correct and incorrect responses. Using the same MEG data, we addressed the dynamics of cortical processes involved in the detection of the independently moving object and investigated whether we could obtain confirmatory evidence for the brain activity patterns used by the classifier. Our findings indicate that response correctness could be reliably predicted by the SVC, with the highest accuracy during the blank period after motion and preceding the response. The spatial distribution of the areas critical for the correct prediction was similar but not exclusive to areas underlying the evoked activity. Importantly, SVC identified frontal areas otherwise not detected with evoked activity that seem to be important for the successful performance in the task. Dynamic connectivity further supported the involvement of frontal and occipital-temporal areas during the task periods. This is the first study to dynamically map cortical areas using a fully data-driven approach in order to investigate the neural mechanisms involved in the detection of moving objects during observer's self-motion.

MemCat: a new category-based image set quantified on memorability

Images differ in their memorability in consistent ways across observers. What makes an image memorable is not fully understood to date. Most of the current insight is in terms of high-level semantic aspects, related to the content. However, research still shows consistent differences within semantic categories, suggesting a role for factors at other levels of processing in the visual hierarchy. To aid investigations into this role as well as contributions to the understanding of image memorability more generally, we present MemCat. MemCat is a category-based image set, consisting of 10K images representing five broader, memorability-relevant categories (animal, food, landscape, sports, and vehicle) and further divided into subcategories (e.g., bear). They were sampled from existing source image sets that offer bounding box annotations or more detailed segmentation masks. We collected memorability scores for all 10 K images, each score based on the responses of on average 99 participants in a repeat-detection memory task. Replicating previous research, the collected memorability scores show high levels of consistency across observers. Currently, MemCat is the second largest memorability image set and the largest offering a category-based structure. MemCat can be used to study the factors underlying the variability in image memorability, including the variability within semantic categories. In addition, it offers a new benchmark dataset for the automatic prediction of memorability scores (e.g., with convolutional neural networks). Finally, MemCat allows the study of neural and behavioral correlates of memorability while controlling for semantic category.

Memorability of photographs in subjective cognitive decline and mild cognitive impairment: Implications for cognitive assessment

INTRODUCTION

Impaired long-term memory is a defining feature of mild cognitive impairment (MCI). We tested whether this impairment is item specific, limited to some memoranda, whereas some remain consistently memorable.

METHODS

We conducted item-based analyses of long-term visual recognition memory. Three hundred ninety-four participants (healthy controls, subjective cognitive decline [SCD], and MCI) in the multicentric DZNE-Longitudinal Cognitive Impairment and Dementia Study (DELCODE) were tested with images from a pool of 835 photographs.

RESULTS

We observed consistent memorability for images in healthy controls, SCD, and MCI, predictable by a neural network trained on another healthy sample. Looking at memorability differences between groups, we identified images that could successfully categorize group membership with higher success and a substantial image reduction than the original image set.

DISCUSSION

Individuals with SCD and MCI show consistent memorability for specific items, while other items show significant diagnosticity. Certain stimulus features could optimize diagnostic assessment, while others could support memory.

Population response magnitude variation in inferotemporal cortex predicts image memorability

Most accounts of image and object encoding in inferotemporal cortex (IT) focus on the distinct patterns of spikes that different images evoke across the IT population. By analyzing data collected from IT as monkeys performed a visual memory task, we demonstrate that variation in a complementary coding scheme, the magnitude of the population response, can largely account for how well images will be remembered. To investigate the origin of IT image memorability modulation, we probed convolutional neural network models trained to categorize objects. We found that, like the brain, different natural images evoked different magnitude responses from these networks, and in higher layers, larger magnitude responses were correlated with the images that humans and monkeys find most memorable. Together, these results suggest that variation in IT population response magnitude is a natural consequence of the optimizations required for visual processing, and that this variation has consequences for visual memory.