Visual properties and memorising scenes: Effects of image-space sparseness and uniformity

Navigating Memorability Landscapes: Hyperbolic Geometry Reveals Hierarchical Structures in Object Concept Memory

Why are some object concepts (e.g., birds, cars, vegetables, etc.) more memorable than others? Prior studies have suggested that features (e.g., color, animacy, etc.) and typicality (e.g., robin vs. penguin) of object images influences the likelihood of being remembered. However, a complete understanding of object memorability remains elusive. In this study, we examine whether the geometric relationship between object concepts explains differences in their memorability. Specifically, we hypothesize that image concepts will be geometrically arranged in hierarchical structures and that memorability will be explained by a concept's depth in these hierarchical trees. To test this hypothesis, we construct a Hyperbolic representation space of object concepts (N=1,854) from the THINGS database (Hebart et al., 2019), which consists of naturalistic images of concrete objects, and a space of 49 feature dimensions derived from data-driven models. Using ALBATROSS (Stier, A. J., Giusti, C., & Berman, M. G., In prep), a stochastic topological data analysis technique that detects underlying structures of data, we demonstrate that Hyperbolic geometry efficiently captures the hierarchical organization of object concepts above and beyond a traditional Euclidean geometry and that hierarchical organization is related to memorability. We find that concepts closer to the center of the representational space are more prototypical and also more memorable. Importantly, Hyperbolic distances are more predictive of memorability and prototypicality than Euclidean distances, suggesting that concept memorability and typicality are organized hierarchically. Taken together, our work presents a novel hierarchical representational structure of object concepts that explains memorability and typicality.

The influence of semantics on long-term visual memory capacity in children and adults

Human visual memory capacity has a rapid developmental progression. Here we examine whether image semantics modulate this progression. We assessed the performance of children (6-14 years) and young adults (19-36 years) on a visual memory task using real-world (or meaningful) as well as abstract image sets, which were matched in low-level image attributes. For real images, we find comparable performance across the two age groups, consistent with previously reported results. However, for abstract images, we find a clear age-related difference indicating greater reliance of children's memory processes on semantics, suggesting that strategies for encoding abstract patterns keep improving even into late childhood. We complemented these studies with computational experiments designed to examine the role of increasing experience with real-world images on real and abstract image encoding, to examine whether the observed age-related differences, as well as the general privilege of real over abstract images, can emerge directly through experience with meaningful images. Our results provide support for this possibility and set the stage for a finer-grained investigation of the timeline along which children's memory capacity for abstract images reaches adult levels.

Differential effects of intrinsic properties of natural scenes and interference mechanisms on recognition processes in long-term visual memory

Humans display remarkable long-term visual memory (LTVM) processes. Even though images may be intrinsically memorable, the fidelity of their visual representations, and consequently the likelihood of successfully retrieving them, hinges on their similarity when concurrently held in LTVM. In this debate, it is still unclear whether intrinsic features of images (perceptual and semantic) may be mediated by mechanisms of interference generated at encoding, or during retrieval, and how these factors impinge on recognition processes. In the current study, participants (32) studied a stream of 120 natural scenes from 8 semantic categories, which varied in frequencies (4, 8, 16 or 32 exemplars per category) to generate different levels of category interference, in preparation for a recognition test. Then they were asked to indicate which of two images, presented side by side (i.e. two-alternative forced-choice), they remembered. The two images belonged to the same semantic category but varied in their perceptual similarity (similar or dissimilar). Participants also expressed their confidence (sure/not sure) about their recognition response, enabling us to tap into their metacognitive efficacy (meta-d'). Additionally, we extracted the activation of perceptual and semantic features in images (i.e. their informational richness) through deep neural network modelling and examined their impact on recognition processes. Corroborating previous literature, we found that category interference and perceptual similarity negatively impact recognition processes, as well as response times and metacognitive efficacy. Moreover, images semantically rich were less likely remembered, an effect that trumped a positive memorability boost coming from perceptual information. Critically, we did not observe any significant interaction between intrinsic features of images and interference generated either at encoding or during retrieval. All in all, our study calls for a more integrative understanding of the representational dynamics during encoding and recognition enabling us to form, maintain and access visual information.

Memorability of line drawings of scenes: the role of contour properties

Why are some images more likely to be remembered than others? Previous work focused on the influence of global, low-level visual features as well as image content on memorability. To better understand the role of local, shape-based contours, we here investigate the memorability of photographs and line drawings of scenes. We find that the memorability of photographs and line drawings of the same scenes is correlated. We quantitatively measure the role of contour properties and their spatial relationships for scene memorability using a Random Forest analysis. To determine whether this relationship is merely correlational or if manipulating these contour properties causes images to be remembered better or worse, we split each line drawing into two half-images, one with high and the other with low predicted memorability according to the trained Random Forest model. In a new memorability experiment, we find that the half-images predicted to be more memorable were indeed remembered better, confirming a causal role of shape-based contour features, and, in particular, T junctions in scene memorability. We performed a categorization experiment on half-images to test for differential access to scene content. We found that half-images predicted to be more memorable were categorized more accurately. However, categorization accuracy for individual images was not correlated with their memorability. These results demonstrate that we can measure the contributions of individual contour properties to scene memorability and verify their causal involvement with targeted image manipulations, thereby bridging the gap between low-level features and scene semantics in our understanding of memorability.

The features underlying the memorability of objects

What makes certain images more memorable than others? While much of memory research has focused on participant effects, recent studies using a stimulus-centric perspective have sparked debate on the determinants of memory, including the roles of semantic and visual features and whether the most prototypical or atypical items are best remembered. Prior studies have typically relied on constrained stimulus sets, limiting a generalized view of the features underlying what we remember. Here, we collected more than 1 million memory ratings for a naturalistic dataset of 26,107 object images designed to comprehensively sample concrete objects. We establish a model of object features that is predictive of image memorability and examined whether memorability could be accounted for by the typicality of the objects. We find that semantic features exert a stronger influence than perceptual features on what we remember and that the relationship between memorability and typicality is more complex than a simple positive or negative association alone.

Judgments of learning reveal conscious access to stimulus memorability

Despite the massive capacity of visual long-term memory, individuals do not successfully encode all visual information they wish to remember. This variability in encoding success has been traditionally ascribed to fluctuations in individuals' cognitive states (e.g., sustained attention) and differences in memory encoding processes (e.g., depth of encoding). However, recent work has shown that a considerable amount of variability in encoding success stems from intrinsic stimulus properties that determine the ease of encoding across individuals. While researchers have identified several perceptual and semantic properties that contribute to stimulus memorability, much remains unknown, including whether individuals are aware of the memorability of stimuli they encounter. In the present study, we investigated whether individuals have conscious access to the memorability of real-world stimuli while forming self-referential judgments of learning (JOL) during explicit memory encoding (Experiments 1A-B) and when asked about the perceived memorability of a stimulus in the absence of attempted encoding (Experiments 2A-B). We found that JOLs and perceived memorability estimates (PME) were consistent across individuals and predictive of memorability, confirming that individuals can access memorability with or without stimulus encoding. At the same time, access to memorability was not comprehensive. We found that individuals unexpectedly remembered and forgot consistent sets of stimuli as well. When we compared access to memorability between JOLs and PMEs, we found that individuals had more access during JOLs. Thus, our findings demonstrate that individuals have partial access to stimulus memorability and that explicit encoding increases the amount of access that is available.

Radiogenomics of C9orf72 Expansion Carriers Reveals Global Transposable Element Derepression and Enables Prediction of Thalamic Atrophy and Clinical Impairment

Hexanucleotide repeat expansion (HRE) within C9orf72 is the most common genetic cause of frontotemporal dementia (FTD). Thalamic atrophy occurs in both sporadic and familial FTD but is thought to distinctly affect HRE carriers. Separately, emerging evidence suggests widespread derepression of transposable elements (TEs) in the brain in several neurodegenerative diseases, including C9orf72 HRE-mediated FTD (C9-FTD). Whether TE activation can be measured in peripheral blood and how the reduction in peripheral C9orf72 expression observed in HRE carriers relates to atrophy and clinical impairment remain unknown. We used FreeSurfer software to assess the effects of C9orf72 HRE and clinical diagnosis (n = 78 individuals, male and female) on atrophy of thalamic nuclei. We also generated a novel, human, whole-blood RNA-sequencing dataset to determine the relationships among peripheral C9orf72 expression, TE activation, thalamic atrophy, and clinical severity (n = 114 individuals, male and female). We confirmed global thalamic atrophy and reduced C9orf72 expression in HRE carriers. Moreover, we identified disproportionate atrophy of the right mediodorsal lateral nucleus in HRE carriers and showed that C9orf72 expression associated with clinical severity, independent of thalamic atrophy. Strikingly, we found global peripheral activation of TEs, including the human endogenous LINE-1 element L1HS L1HS levels were associated with atrophy of multiple pulvinar nuclei, a thalamic region implicated in C9-FTD. Integration of peripheral transcriptomic and neuroimaging data from human HRE carriers revealed atrophy of specific thalamic nuclei, demonstrated that C9orf72 levels relate to clinical severity, and identified marked derepression of TEs, including L1HS, which predicted atrophy of FTD-relevant thalamic nuclei.SIGNIFICANCE STATEMENT Pathogenic repeat expansion in C9orf72 is the most frequent genetic cause of FTD and amyotrophic lateral sclerosis (ALS; C9-FTD/ALS). The clinical, neuroimaging, and pathologic features of C9-FTD/ALS are well characterized, whereas the intersections of transcriptomic dysregulation and brain structure remain largely unexplored. Herein, we used a novel radiogenomic approach to examine the relationship between peripheral blood transcriptomics and thalamic atrophy, a neuroimaging feature disproportionately impacted in C9-FTD/ALS. We confirmed reduction of C9orf72 in blood and found broad dysregulation of transposable elements-genetic elements typically repressed in the human genome-in symptomatic C9orf72 expansion carriers, which associated with atrophy of thalamic nuclei relevant to FTD. C9orf72 expression was also associated with clinical severity, suggesting that peripheral C9orf72 levels capture disease-relevant information.

False memories when viewing overlapping scenes

Humans can memorize and later recognize many objects and complex scenes. In this study, we prepared large photographs and presented participants with only partial views to test the fidelity of their memories. The unpresented parts of the photographs were used as a source of distractors with similar semantic and perceptual information. Additionally, we presented overlapping views to determine whether the second presentation provided a memory advantage for later recognition tests. Experiment 1 (N = 28) showed that while people were good at recognizing presented content and identifying new foils, they showed a remarkable level of uncertainty about foils selected from the unseen parts of presented photographs (false alarm, 59%). The recognition accuracy was higher for the parts that were shown twice, irrespective of whether the same identical photograph was viewed twice or whether two photographs with overlapping content were observed. In Experiment 2 (N = 28), the memorability of the large image was estimated by a pre-trained deep neural network. Neither the recognition accuracy for an image part nor the tendency for false alarms correlated with the memorability. Finally, in Experiment 3 (N = 21), we repeated the experiment while measuring eye movements. Fixations were biased toward the center of the original large photograph in the first presentation, and this bias was repeated during the second presentation in both identical and overlapping views. Altogether, our experiments show that people recognize parts of remembered photographs, but they find it difficult to reject foils from unseen parts, suggesting that their memory representation is not sufficiently detailed to rule them out as distractors.

Test position effects on hit and false alarm rates in recognition memory for paintings and words

When old/new recognition memory is tested with equal numbers of studied and nonstudied items and no rewards or instructions that favour one response over the other, there is no obvious reason for response bias. In line with this, Canadian undergraduates have shown, on average, a neutral response bias when we tested them on recognition of common English words. By contrast, most subjects we have tested on recognition of richly detailed images have shown a conservative bias: they more often erred by missing a studied image than by judging a nonstudied image as studied. Here, in an effort to better understand these materials-based bias effects (MBBEs), we examined changes in hit and false alarm (FA) rates (and in sensitivity and bias) from the first to fourth quartile of a recognition memory test in eight experiments in which undergraduates studied words and/or images of paintings. Response bias for images tended to increase across quartiles, whereas bias for words showed no consistent pattern across quartiles. This pattern could be described as an increase in the MBBE over the course of the test, but the underlying patterns for hits and FAs are not easily reconciled with this interpretation. Hit rates decreased over the course of the test for both materials types, with that decline tending to be steeper for images than words. For words, FA rates tended to increase across quartiles, whereas for paintings FA rates did not increase across quartiles. We discuss implications of these findings for theoretical accounts of the MBBE.

Image memorability is predicted by discriminability and similarity in different stages of a convolutional neural network

The features of an image can be represented at multiple levels-from its low-level visual properties to high-level meaning. What drives some images to be memorable while others are forgettable? We address this question across two behavioral experiments. In the first, different layers of a convolutional neural network (CNN), which represent progressively higher levels of features, were used to select the images that would be shown to 100 participants through a form of prospective assignment. Here, the discriminability/similarity of an image with others, according to different CNN layers dictated the images presented to different groups, who made a simple indoor versus outdoor judgment for each scene. We found that participants remember more scene images that were selected based on their low-level discriminability or high-level similarity. A second experiment replicated these results in an independent sample of 50 participants, with a different order of postencoding tasks. Together, these experiments provide evidence that both discriminability and similarity, at different visual levels, predict image memorability.

The effect of intrinsic image memorability on recollection and familiarity

Many photographs of real-life scenes are very consistently remembered or forgotten by most people, making these images intrinsically memorable or forgettable. Although machine vision algorithms can predict a given image's memorability very well, nothing is known about the subjective quality of these memories: are memorable images recognized based on strong feelings of familiarity or on recollection of episodic details? We tested people's recognition memory for memorable and forgettable scenes selected from image memorability databases, which contain memorability scores for each image, based on large-scale recognition memory experiments. Specifically, we tested the effect of intrinsic memorability on recollection and familiarity using cognitive computational models based on receiver operating characteristics (ROCs; Experiment 1 and 2) and on remember/know (R/K) judgments (Experiment 2). The ROC data of Experiment 2 indicated that image memorability boosted memory strength, but did not find a specific effect on recollection or familiarity. By contrast, ROC data from Experiment 2, which was designed to facilitate encoding and, in turn, recollection, found evidence for a specific effect of image memorability on recollection. Moreover, R/K judgments showed that, on average, memorability boosts recollection rather than familiarity. However, we also found a large degree of variability in these judgments across individual images: some images actually achieved high recognition rates by exclusively boosting familiarity rather than recollection. Together, these results show that current machine vision algorithms that can predict an image's intrinsic memorability in terms of hit rates fall short of describing the subjective quality of human memories.

False memories for scenes using the DRM paradigm

People are remarkably good at remembering photographs. To further investigate the nature of the stored representations and the fidelity of human memories, it would be useful to evaluate the visual similarity of stimuli presented in experiments. Here, we explored the possible use of convolutional neural networks (CNN) as a measure of perceptual or representational similarity of visual scenes with respect to visual memory research. In Experiment 1, we presented participants with sets of nine images from the same scene category and tested whether they were able to detect the most distant scene in the image space defined by CNN. Experiment 2 was a visual variant of the Deese-Roediger-McDermott paradigm. We asked participants to remember a set of photographs from the same scene category. The photographs were preselected based on their distance to a particular visual prototype (defined as centroid of the image space). In the recognition test, we observed higher false alarm rates for scenes closer to this visual prototype. Our findings show that the similarity measured by CNN is reflected in human behavior: people can detect odd-one-out scenes or be lured to false alarms with similar stimuli. This method can be used for further studies regarding visual memory for complex scenes.

Get the Picture? Goodness of Image Organization Contributes to Image Memorability

According to Gestalt psychologists, goodness is a crucial variable for image organization. We hypothesized that these differences in goodness contribute to variability in image memorability. Building on this, we predicted that two characteristics of good organizations, (i) fast, efficient processing and (ii) robustness against transformations (e.g., shrinking), would be characteristic of memorable images. Two planned (Study 1, Study 2) and one follow-up (Study 3) study were conducted to test this. Study 1 operationalized fast processing as accuracy in a rapid-scene categorization task ("categorizability"). Study 2 operationalized robustness against shrinking as reaction time in a thumbnail search task ("shrinkability"). We used 44 real-life scene images of 14 semantic categories from a previous memorability study. Each image was assigned a categorizability and shrinkability score. The predicted positive relation between categorizability and memorability was not observed in Study 1. A post-hoc explanation attributed this null result to a masking role of image distinctiveness. Furthermore, memorable images were located faster in the thumbnail search task, as predicted, but Study 2 could not rule whether this was merely a result of their distinctiveness. To elucidate these results, Study 3 quantified the images on distinctiveness and statistically controlled for this variable in a reanalysis of Study 1 and Study 2. When distinctiveness was controlled for, categorizability and memorability did show a significant positive correlation. Moreover, the results also argued against the alternative explanation of the results of Study 2. Taken together, the results support the hypothesis that goodness of organization contributes to image memorability.

Dissociating neural markers of stimulus memorability and subjective recognition during episodic retrieval

While much of memory research takes an observer-centric focus looking at participant performance, recent work has pinpointed important item-centric effects on memory, or how intrinsically memorable a given stimulus is. However, little is known about the neural correlates of memorability during memory retrieval, or how such correlates relate to subjective memory behavior. Here, stimuli and blood-oxygen-level dependent data from a prior functional magnetic resonance imaging (fMRI) study were reanalyzed using a memorability-based framework. In that study, sixteen participants studied 200 novel face images and were scanned while making recognition memory judgments on those faces, interspersed with 200 unstudied faces. In the current investigation, memorability scores for those stimuli were obtained through an online crowd-sourced (N = 740) continuous recognition test that measured each image's corrected recognition rate. Representational similarity analyses were conducted across the brain to identify regions wherein neural pattern similarity tracked item-specific effects (stimulus memorability) versus observer-specific effects (individual memory performance). We find two non-overlapping sets of regions, with memorability-related information predominantly represented within ventral and medial temporal regions and memory retrieval outcome-related information within fronto-parietal regions. These memorability-based effects persist regardless of image history, implying that coding of stimulus memorability may be a continuous and automatic perceptual process.