Distributed representations of behaviorally-relevant object dimensions in the human visual system

Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioral goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of behaviorally-relevant dimensions derived from a large-scale analysis of human similarity judgments directly onto the brain. Our results reveal broadly distributed representations of behaviorally-relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behaviorally-relevant dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.

Biomimetic versus arbitrary motor control strategies for bionic hand skill learning

A long-standing engineering ambition has been to design anthropomorphic bionic limbs: devices that look like and are controlled in the same way as the biological body (biomimetic). The untested assumption is that biomimetic motor control enhances device embodiment, learning, generalization and automaticity. To test this, we compared biomimetic and non-biomimetic control strategies for non-disabled participants when learning to control a wearable myoelectric bionic hand operated by an eight-channel electromyography pattern-recognition system. We compared motor learning across days and behavioural tasks for two training groups: biomimetic (mimicking the desired bionic hand gesture with biological hand) and arbitrary control (mapping an unrelated biological hand gesture with the desired bionic gesture). For both trained groups, training improved bionic limb control, reduced cognitive reliance and increased embodiment over the bionic hand. Biomimetic users had more intuitive and faster control early in training. Arbitrary users matched biomimetic performance later in training. Furthermore, arbitrary users showed increased generalization to a new control strategy. Collectively, our findings suggest that biomimetic and arbitrary control strategies provide different benefits. The optimal strategy is probably not strictly biomimetic, but rather a flexible strategy within the biomimetic-to-arbitrary spectrum, depending on the user, available training opportunities and user requirements.

Stable Cortical Body Maps Before and After Arm Amputation

Neuroscientists have long debated the adult brain's capacity to reorganize itself in response to injury. A driving model for studying plasticity has been limb amputation. For decades, it was believed that amputation triggers large-scale reorganization of cortical body resources. However, these studies have relied on cross-sectional observations post-amputation, without directly tracking neural changes. Here, we longitudinally followed adult patients with planned arm amputations and measured hand and face representations, before and after amputation. By interrogating the representational structure elicited from movements of the hand (pre-amputation) and phantom hand (post-amputation), we demonstrate that hand representation is unaltered. Further, we observed no evidence for lower face (lip) reorganization into the deprived hand region. Collectively, our findings provide direct and decisive evidence that amputation does not trigger large-scale cortical reorganization.

Multidimensional object properties are dynamically represented in the human brain

Our visual world consists of an immense number of unique objects and yet, we are easily able to identify, distinguish, interact, and reason about the things we see within several hundred milliseconds. This requires that we flexibly integrate and focus on different object properties to support specific behavioral goals. In the current study, we examined how these rich object representations unfold in the human brain by modelling time-resolved MEG signals evoked by viewing thousands of objects. Using millions of behavioral judgments to guide our understanding of the neural representation of the object space, we find distinct temporal profiles across the object dimensions. These profiles fell into two broad types with either a distinct and early peak (~150 ms) or a slow rise to a late peak (~300 ms). Further, the early effects are stable across participants in contrast to later effects which show more variability across people. This highlights that early peaks may carry stimulus-specific and later peaks subject-specific information. Given that the dimensions with early peaks seem to be primarily visual dimensions and those with later peaks more conceptual, our results suggest that conceptual processing is more variable across people. Together, these data provide a comprehensive account of how a variety of object properties unfold in the human brain and contribute to the rich nature of object vision.

The features underlying the memorability of objects

What makes certain images more memorable than others? While much of memory research has focused on participant effects, recent studies using a stimulus-centric perspective have sparked debate on the determinants of memory, including the roles of semantic and visual features and whether the most prototypical or atypical items are best remembered. Prior studies have typically relied on constrained stimulus sets, limiting a generalized view of the features underlying what we remember. Here, we collected more than 1 million memory ratings for a naturalistic dataset of 26,107 object images designed to comprehensively sample concrete objects. We establish a model of object features that is predictive of image memorability and examined whether memorability could be accounted for by the typicality of the objects. We find that semantic features exert a stronger influence than perceptual features on what we remember and that the relationship between memorability and typicality is more complex than a simple positive or negative association alone.

A database of heterogeneous faces for studying naturalistic expressions

Facial expressions are thought to be complex visual signals, critical for communication between social agents. Most prior work aimed at understanding how facial expressions are recognized has relied on stimulus databases featuring posed facial expressions, designed to represent putative emotional categories (such as 'happy' and 'angry'). Here we use an alternative selection strategy to develop the Wild Faces Database (WFD); a set of one thousand images capturing a diverse range of ambient facial behaviors from outside of the laboratory. We characterized the perceived emotional content in these images using a standard categorization task in which participants were asked to classify the apparent facial expression in each image. In addition, participants were asked to indicate the intensity and genuineness of each expression. While modal scores indicate that the WFD captures a range of different emotional expressions, in comparing the WFD to images taken from other, more conventional databases, we found that participants responded more variably and less specifically to the wild-type faces, perhaps indicating that natural expressions are more multiplexed than a categorical model would predict. We argue that this variability can be employed to explore latent dimensions in our mental representation of facial expressions. Further, images in the WFD were rated as less intense and more genuine than images taken from other databases, suggesting a greater degree of authenticity among WFD images. The strong positive correlation between intensity and genuineness scores demonstrating that even the high arousal states captured in the WFD were perceived as authentic. Collectively, these findings highlight the potential utility of the WFD as a new resource for bridging the gap between the laboratory and real world in studies of expression recognition.

Motion-selective areas V5/MT and MST appear resistant to deterioration in choroideremia

Choroideremia (CHM) is an X-linked recessive form of hereditary retinal degeneration, which preserves only small islands of central retinal tissue. Previously, we demonstrated the relationship between central vision and structure and population receptive fields (pRF) using functional magnetic resonance imaging (fMRI) in untreated CHM subjects. Here, we replicate and extend this work, providing a more in-depth analysis of the visual responses in a cohort of CHM subjects who participated in a retinal gene therapy clinical trial. fMRI was conducted in six CHM subjects and six age-matched healthy controls (HC's) while they viewed drifting contrast pattern stimuli monocularly. A single ∼3-minute fMRI run was collected for each eye. Participants also underwent ophthalmic evaluations of visual acuity and static automatic perimetry (SAP). Consistent with our previous report, a single ∼ 3 min fMRI run accurately characterized ophthalmic evaluations of visual function in most CHM subjects. In-depth analyses of the cortical distribution of pRF responses revealed that the motion-selective regions V5/MT and MST appear resistant to progressive retinal degenerations in CHM subjects. This effect was restricted to V5/MT and MST and was not present in either primary visual cortex (V1), motion-selective V3A or regions within the ventral visual pathway. Motion-selective areas V5/MT and MST appear to be resistant to the continuous detrimental impact of CHM. Such resilience appears selective to these areas and may be mediated by independent retina-V5/MT anatomical connections that bypass V1. We did not observe any significant impact of gene therapy.

THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior

Understanding object representations requires a broad, comprehensive sampling of the objects in our visual world with dense measurements of brain activity and behavior. Here, we present THINGS-data, a multimodal collection of large-scale neuroimaging and behavioral datasets in humans, comprising densely sampled functional MRI and magnetoencephalographic recordings, as well as 4.70 million similarity judgments in response to thousands of photographic images for up to 1,854 object concepts. THINGS-data is unique in its breadth of richly annotated objects, allowing for testing countless hypotheses at scale while assessing the reproducibility of previous findings. Beyond the unique insights promised by each individual dataset, the multimodality of THINGS-data allows combining datasets for a much broader view into object processing than previously possible. Our analyses demonstrate the high quality of the datasets and provide five examples of hypothesis-driven and data-driven applications. THINGS-data constitutes the core public release of the THINGS initiative (https://things-initiative.org) for bridging the gap between disciplines and the advancement of cognitive neuroscience.

Children perceive illusory faces in objects as male more often than female

Face pareidolia is the experience of seeing illusory faces in inanimate objects. While children experience face pareidolia, it is unknown whether they perceive gender in illusory faces, as their face evaluation system is still developing in the first decade of life. In a sample of 412 children and adults from 4 to 80 years of age we found that like adults, children perceived many illusory faces in objects to have a gender and had a strong bias to see them as male rather than female, regardless of their own gender identification. These results provide evidence that the male bias for face pareidolia emerges early in life, even before the ability to discriminate gender from facial cues alone is fully developed. Further, the existence of a male bias in children suggests that any social context that elicits the cognitive bias to see faces as male has remained relatively consistent across generations.

Should bionic limb control mimic the human body? Impact of control strategy on bionic hand skill learning

A longstanding engineering ambition has been to design anthropomorphic bionic limbs: devices that look like and are controlled in the same way as the biological body (biomimetic). The untested assumption is that biomimetic motor control enhances device embodiment, learning, generalization, and automaticity. To test this, we compared biomimetic and non-biomimetic control strategies for able-bodied participants when learning to operate a wearable myoelectric bionic hand. We compared motor learning across days and behavioural tasks for two training groups: Biomimetic (mimicking the desired bionic hand gesture with biological hand) and Arbitrary control (mapping an unrelated biological hand gesture with the desired bionic gesture). For both trained groups, training improved bionic limb control, reduced cognitive reliance, and increased embodiment over the bionic hand. Biomimetic users had more intuitive and faster control early in training. Arbitrary users matched biomimetic performance later in training. Further, arbitrary users showed increased generalization to a novel control strategy. Collectively, our findings suggest that biomimetic and arbitrary control strategies provide different benefits. The optimal strategy is likely not strictly biomimetic, but rather a flexible strategy within the biomimetic to arbitrary spectrum, depending on the user, available training opportunities and user requirements.

Sustained neural representations of personally familiar people and places during cued recall

The recall and visualization of people and places from memory is an everyday occurrence, yet the neural mechanisms underpinning this phenomenon are not well understood. In particular, the temporal characteristics of the internal representations generated by active recall are unclear. Here, we used magnetoencephalography (MEG) and multivariate pattern analysis to measure the evolving neural representation of familiar places and people across the whole brain when human participants engage in active recall. To isolate self-generated imagined representations, we used a retro-cue paradigm in which participants were first presented with two possible labels before being cued to recall either the first or second item. We collected personalized labels for specific locations and people familiar to each participant. Importantly, no visual stimuli were presented during the recall period, and the retro-cue paradigm allowed the dissociation of responses associated with the labels from those corresponding to the self-generated representations. First, we found that following the retro-cue it took on average ∼1000 ms for distinct neural representations of freely recalled people or places to develop. Second, we found distinct representations of personally familiar concepts throughout the 4 s recall period. Finally, we found that these representations were highly stable and generalizable across time. These results suggest that self-generated visualizations and recall of familiar places and people are subserved by a stable neural mechanism that operates relatively slowly when under conscious control.

Multidimensional memory topography in the medial parietal cortex identified from neuroimaging of thousands of daily memory videos

Our memories form a tapestry of events, people, and places, woven across the decades of our lives. However, research has often been limited in assessing the nature of episodic memory by using artificial stimuli and short time scales. The explosion of social media enables new ways to examine the neural representations of naturalistic episodic memories, for features like the memory's age, location, memory strength, and emotions. We recruited 23 users of a video diary app ("1 s Everyday"), who had recorded 9266 daily memory videos spanning up to 7 years. During a 3 T fMRI scan, participants viewed 300 of their memory videos intermixed with 300 from another individual. We find that memory features are tightly interrelated, highlighting the need to test them in conjunction, and discover a multidimensional topography in medial parietal cortex, with subregions sensitive to a memory's age, strength, and the familiarity of the people and places involved.

Clutter Substantially Reduces Selectivity for Peripheral Faces in the Macaque Brain

According to a prominent view in neuroscience, visual stimuli are coded by discrete cortical networks that respond preferentially to specific categories, such as faces or objects. However, it remains unclear how these category-selective networks respond when viewing conditions are cluttered, i.e., when there is more than one stimulus in the visual field. Here, we asked three questions: (1) Does clutter reduce the response and selectivity for faces as a function of retinal location? (2) Is the preferential response to faces uniform across the visual field? And (3) Does the ventral visual pathway encode information about the location of cluttered faces? We used fMRI to measure the response of the face-selective network in awake, fixating macaques (two female, five male). Across a series of four experiments, we manipulated the presence and absence of clutter, as well as the location of the faces relative to the fovea. We found that clutter reduces the response to peripheral faces. When presented in isolation, without clutter, the selectivity for faces is fairly uniform across the visual field, but, when clutter is present, there is a marked decrease in the selectivity for peripheral faces. We also found no evidence of a contralateral visual field bias when faces were presented in clutter. Nonetheless, multivariate analyses revealed that the location of cluttered faces could be decoded from the multivoxel response of the face-selective network. Collectively, these findings demonstrate that clutter blunts the selectivity of the face-selective network to peripheral faces, although information about their retinal location is retained.SIGNIFICANCE STATEMENT Numerous studies that have measured brain activity in macaques have found visual regions that respond preferentially to faces. Although these regions are thought to be essential for social behavior, their responses have typically been measured while faces were presented in isolation, a situation atypical of the real world. How do these regions respond when faces are presented with other stimuli? We report that, when clutter is present, the preferential response to foveated faces is spared but preferential response to peripheral faces is reduced. Our results indicate that the presence of clutter changes the response of the face-selective network.

The nature of neural object representations during dynamic occlusion

Objects disappearing briefly from sight due to occlusion is an inevitable occurrence in everyday life. Yet we generally have a strong experience that occluded objects continue to exist, despite the fact that they objectively disappear. This indicates that neural object representations must be maintained during dynamic occlusion. However, it is unclear what the nature of such representation is and in particular whether it is perception-like or more abstract, for example, reflecting limited features such as position or movement direction only. In this study, we address this question by examining how different object features such as object shape, luminance, and position are represented in the brain when a moving object is dynamically occluded. We apply multivariate decoding methods to Magnetoencephalography (MEG) data to track how object representations unfold over time. Our methods allow us to contrast the representations of multiple object features during occlusion and enable us to compare the neural responses evoked by visible and occluded objects. The results show that object position information is represented during occlusion to a limited extent while object identity features are not maintained through the period of occlusion. Together, this suggests that the nature of object representations during dynamic occlusion is different from visual representations during perception.

Making sense of phantom limb pain

Phantom limb pain (PLP) impacts the majority of individuals who undergo limb amputation. The PLP experience is highly heterogenous in its quality, intensity, frequency and severity. This heterogeneity, combined with the low prevalence of amputation in the general population, has made it difficult to accumulate reliable data on PLP. Consequently, we lack consensus on PLP mechanisms, as well as effective treatment options. However, the wealth of new PLP research, over the past decade, provides a unique opportunity to re-evaluate some of the core assumptions underlying what we know about PLP and the rationale behind PLP treatments. The goal of this review is to help generate consensus in the field on how best to research PLP, from phenomenology to treatment. We highlight conceptual and methodological challenges in studying PLP, which have hindered progress on the topic and spawned disagreement in the field, and offer potential solutions to overcome these challenges. Our hope is that a constructive evaluation of the foundational knowledge underlying PLP research practices will enable more informed decisions when testing the efficacy of existing interventions and will guide the development of the next generation of PLP treatments.

Highly similar and competing visual scenes lead to diminished object but not spatial detail in memory drawings

Drawings of scenes made from memory can be highly detailed and spatially accurate, with little information not found in the observed stimuli. While prior work has focused on studying memory for distinct scenes, less is known about the specific detail recalled when episodes are highly similar and competing. Here, participants (N = 30) were asked to study and recall eight complex scene images using a drawing task. Importantly, four of these images were exemplars of different scene categories, while the other four images were from the same scene category. The resulting 213 drawings were judged by 1764 online scorers for a comprehensive set of measures, including scene and object diagnosticity, spatial information, and fixation and pen movement behaviour. We observed that competition in memory resulted in diminished object detail, with drawings and objects that were less diagnostic of their original image. However, repeated exemplars of a category did not result in differences in spatial memory accuracy, and there were no differences in fixations during study or pen movements during recall. These results reveal that while drawings for distinct categories of scenes can be highly detailed and accurate, drawings for scenes from repeated categories, creating competition in memory, show reduced object detail.

Multiple Adjoining Word- and Face-Selective Regions in Ventral Temporal Cortex Exhibit Distinct Dynamics

The map of category-selectivity in human ventral temporal cortex (VTC) provides organizational constraints to models of object recognition. One important principle is lateral-medial response biases to stimuli that are typically viewed in the center or periphery of the visual field. However, little is known about the relative temporal dynamics and location of regions that respond preferentially to stimulus classes that are centrally viewed, such as the face- and word-processing networks. Here, word- and face-selective regions within VTC were mapped using intracranial recordings from 36 patients. Partially overlapping, but also anatomically dissociable patches of face- and word-selectivity, were found in VTC. In addition to canonical word-selective regions along the left posterior occipitotemporal sulcus, selectivity was also located medial and anterior to face-selective regions on the fusiform gyrus at the group level and within individual male and female subjects. These regions were replicated using 7 Tesla fMRI in healthy subjects. Left hemisphere word-selective regions preceded right hemisphere responses by 125 ms, potentially reflecting the left hemisphere bias for language, with no hemispheric difference in face-selective response latency. Word-selective regions along the posterior fusiform responded first, then spread medially and laterally, then anteriorally. Face-selective responses were first seen in posterior fusiform regions bilaterally, then proceeded anteriorally from there. For both words and faces, the relative delay between regions was longer than would be predicted by purely feedforward models of visual processing. The distinct time courses of responses across these regions, and between hemispheres, suggest that a complex and dynamic functional circuit supports face and word perception.SIGNIFICANCE STATEMENT Representations of visual objects in the human brain have been shown to be organized by several principles, including whether those objects tend to be viewed centrally or peripherally in the visual field. However, it remains unclear how regions that process objects that are viewed centrally, such as words and faces, are organized relative to one another. Here, invasive and noninvasive neuroimaging suggests that there is a mosaic of regions in ventral temporal cortex that respond selectively to either words or faces. These regions display differences in the strength and timing of their responses, both within and between brain hemispheres, suggesting that they play different roles in perception. These results illuminate extended, bilateral, and dynamic brain pathways that support face perception and reading.

Representation of Contralateral Visual Space in the Human Hippocampus

The initial encoding of visual information primarily from the contralateral visual field is a fundamental organizing principle of the primate visual system. Recently, the presence of such retinotopic sensitivity has been shown to extend well beyond early visual cortex to regions not historically considered retinotopically sensitive. In particular, human scene-selective regions in parahippocampal and medial parietal cortex exhibit prominent biases for the contralateral visual field. Here, we used fMRI to test the hypothesis that the human hippocampus, which is thought to be anatomically connected with these scene-selective regions, would also exhibit a biased representation of contralateral visual space. First, population receptive field (pRF) mapping with scene stimuli revealed strong biases for the contralateral visual field in bilateral hippocampus. Second, the distribution of retinotopic sensitivity suggested a more prominent representation in anterior medial portions of the hippocampus. Finally, the contralateral bias was confirmed in independent data taken from the Human Connectome Project (HCP) initiative. The presence of contralateral biases in the hippocampus, a structure considered by many as the apex of the visual hierarchy, highlights the truly pervasive influence of retinotopy. Moreover, this finding has important implications for understanding how visual information relates to the allocentric global spatial representations known to be encoded therein.SIGNIFICANCE STATEMENT Retinotopic encoding of visual information is an organizing principle of visual cortex. Recent work demonstrates this sensitivity in structures far beyond early visual cortex, including those anatomically connected to the hippocampus. Here, using population receptive field (pRF) modeling in two independent sets of data we demonstrate a consistent bias for the contralateral visual field in bilateral hippocampus. Such a bias highlights the truly pervasive influence of retinotopy, with important implications for understanding how the presence of retinotopy relates to more allocentric spatial representations.

Quantifying aphantasia through drawing: Those without visual imagery show deficits in object but not spatial memory

Congenital aphantasia is a recently characterized variation of experience defined by the inability to form voluntary visual imagery, in individuals who are otherwise high performing. Because of this specific deficit to visual imagery, individuals with aphantasia serve as an ideal group for probing the nature of representations in visual memory, particularly the interplay of object, spatial, and symbolic information. Here, we conducted a large-scale online study of aphantasia and revealed a dissociation in object and spatial content in their memory representations. Sixty-one individuals with aphantasia and matched controls with typical imagery studied real-world scene images, and were asked to draw them from memory, and then later copy them during a matched perceptual condition. Drawings were objectively quantified by 2,795 online scorers for object and spatial details. Aphantasic participants recalled significantly fewer objects than controls, with less color in their drawings, and an increased reliance on verbal scaffolding. However, aphantasic participants showed high spatial accuracy equivalent to controls, and made significantly fewer memory errors. These differences between groups only manifested during recall, with no differences between groups during the matched perceptual condition. This object-specific memory impairment in individuals with aphantasia provides evidence for separate systems in memory that support object versus spatial information. The study also provides an important experimental validation for the existence of aphantasia as a variation in human imagery experience.

Theta-burst TMS of lateral occipital cortex reduces BOLD responses across category-selective areas in ventral temporal cortex

Human visual cortex contains three scene-selective regions in the lateral, medial and ventral cortex, termed the occipital place area (OPA), medial place area (MPA) and parahippocampal place area (PPA). Using functional magnetic resonance imaging (fMRI), all three regions respond more strongly when viewing visual scenes compared with isolated objects or faces. To determine how these regions are functionally and causally connected, we applied transcranial magnetic stimulation to OPA and measured fMRI responses before and after stimulation, using a theta-burst paradigm (TBS). To test for stimulus category-selectivity, we presented a range of visual categories (scenes, buildings, objects, faces). To test for specificity of any effects to TBS of OPA we employed two control conditions: Sham, with no TBS stimulation, and an active TBS-control with TBS to a proximal face-selective cortical region (occipital face area, or OFA). We predicted that TBS to OPA (but not OFA) would lead to decreased responses to scenes and buildings (but not other categories) in other scene-selective cortical regions. Across both ROI and whole-volume analyses, we observed decreased responses to scenes in PPA as a result of TBS. However, these effects were neither category specific, with decreased responses to all stimulus categories, nor limited to scene-selective regions, with decreases also observed in face-selective fusiform face area (FFA). Furthermore, similar effects were observed with TBS to OFA, thus effects were not specific to the stimulation site in the lateral occipital cortex. Whilst these data are suggestive of a causal, but non-specific relationship between lateral occipital and ventral temporal cortex, we discuss several factors that could have underpinned this result, such as the differences between TBS and online TMS, the role of anatomical distance between stimulated regions and how TMS effects are operationalised. Furthermore, our findings highlight the importance of active control conditions in brain stimulation experiments to accurately assess functional and causal connectivity between specific brain regions.

Reply to Intraub

Intraub posits the existence of two separate processes in scene memory: one in which we automatically extrapolate the visual information in a scene beyond its boundaries (scene construction), and one in which we normalize our memories to either a schema or an average (normalization). She claims that scene construction will lead to transformations exclusively in the direction of boundary extension (BE), while normalization will produce bidirectional transformations of both BE and the opposite effect of boundary contraction (BC). Thus, because we observed both BE and BC in our study [1], our paradigm must be tapping into additional cognitive processes than just scene construction. However, our paper [1] questions this premise - if BE and BC are equally common using large, representative stimulus sets in the same tasks used previously [2,3], then perhaps prior studies primarily found unidirectional BE effects due to limited stimulus sampling, and there was no privileged link between boundary extension and scene construction in memory to begin with.

Distinct Representational Structure and Localization for Visual Encoding and Recall during Visual Imagery

During memory recall and visual imagery, reinstatement is thought to occur as an echoing of the neural patterns during encoding. However, the precise information in these recall traces is relatively unknown, with previous work primarily investigating either broad distinctions or specific images, rarely bridging these levels of information. Using ultra-high-field (7T) functional magnetic resonance imaging with an item-based visual recall task, we conducted an in-depth comparison of encoding and recall along a spectrum of granularity, from coarse (scenes, objects) to mid (e.g., natural, manmade scenes) to fine (e.g., living room, cupcake) levels. In the scanner, participants viewed a trial-unique item, and after a distractor task, visually imagined the initial item. During encoding, we observed decodable information at all levels of granularity in category-selective visual cortex. In contrast, information during recall was primarily at the coarse level with fine-level information in some areas; there was no evidence of mid-level information. A closer look revealed segregation between voxels showing the strongest effects during encoding and those during recall, and peaks of encoding-recall similarity extended anterior to category-selective cortex. Collectively, these results suggest visual recall is not merely a reactivation of encoding patterns, displaying a different representational structure and localization from encoding, despite some overlap.

Revealing the multidimensional mental representations of natural objects underlying human similarity judgements

Objects can be characterized according to a vast number of possible criteria (such as animacy, shape, colour and function), but some dimensions are more useful than others for making sense of the objects around us. To identify these core dimensions of object representations, we developed a data-driven computational model of similarity judgements for real-world images of 1,854 objects. The model captured most explainable variance in similarity judgements and produced 49 highly reproducible and meaningful object dimensions that reflect various conceptual and perceptual properties of those objects. These dimensions predicted external categorization behaviour and reflected typicality judgements of those categories. Furthermore, humans can accurately rate objects along these dimensions, highlighting their interpretability and opening up a way to generate similarity estimates from object dimensions alone. Collectively, these results demonstrate that human similarity judgements can be captured by a fairly low-dimensional, interpretable embedding that generalizes to external behaviour.

Differential Representations of Perceived and Retrieved Visual Information in Hippocampus and Cortex

Memory retrieval is thought to depend on interactions between hippocampus and cortex, but the nature of representation in these regions and their relationship remains unclear. Here, we performed an ultra-high field fMRI (7T) experiment, comprising perception, learning and retrieval sessions. We observed a fundamental difference between representations in hippocampus and high-level visual cortex during perception and retrieval. First, while object-selective posterior fusiform cortex showed consistent responses that allowed us to decode object identity across both perception and retrieval one day after learning, object decoding in hippocampus was much stronger during retrieval than perception. Second, in visual cortex but not hippocampus, there was consistency in response patterns between perception and retrieval, suggesting that substantial neural populations are shared for both perception and retrieval. Finally, the decoding in hippocampus during retrieval was not observed when retrieval was tested on the same day as learning suggesting that the retrieval process itself is not sufficient to elicit decodable object representations. Collectively, these findings suggest that while cortical representations are stable between perception and retrieval, hippocampal representations are much stronger during retrieval, implying some form of reorganization of the representations between perception and retrieval.

Rapid and dynamic processing of face pareidolia in the human brain

The human brain is specialized for face processing, yet we sometimes perceive illusory faces in objects. It is unknown whether these natural errors of face detection originate from a rapid process based on visual features or from a slower, cognitive re-interpretation. Here we use a multifaceted approach to understand both the spatial distribution and temporal dynamics of illusory face representation in the brain by combining functional magnetic resonance imaging and magnetoencephalography neuroimaging data with model-based analysis. We find that the representation of illusory faces is confined to occipital-temporal face-selective visual cortex. The temporal dynamics reveal a striking evolution in how illusory faces are represented relative to human faces and matched objects. Illusory faces are initially represented more similarly to real faces than matched objects are, but within ~250 ms, the representation transforms, and they become equivalent to ordinary objects. This is consistent with the initial recruitment of a broadly-tuned face detection mechanism which privileges sensitivity over selectivity.

Memorability of words in arbitrary verbal associations modulates memory retrieval in the anterior temporal lobe

Despite large individual differences in memory performance, people remember certain stimuli with overwhelming consistency. This phenomenon is referred to as the memorability of an individual item. However, it remains unknown whether memorability also affects our ability to retrieve associations between items. Here, using a paired-associates verbal memory task, we combine behavioural data, computational modelling and direct recordings from the human brain to examine how memorability influences associative memory retrieval. We find that certain words are correctly retrieved across participants irrespective of the cues used to initiate memory retrieval. These words, which share greater semantic similarity with other words, are more readily available during retrieval and lead to more intrusions when retrieval fails. Successful retrieval of these memorable items, relative to less memorable ones, results in faster reinstatement of neural activity in the anterior temporal lobe. Collectively, our data reveal how the brain prioritizes certain information to facilitate memory retrieval.

Boundaries Extend and Contract in Scene Memory Depending on Image Properties

Boundary extension, a memory distortion in which observers consistently recall a scene with visual information beyond its boundaries, is widely accepted across the psychological sciences as a phenomenon revealing fundamental insight into memory representations [1-3], robust across paradigms [1, 4] and age groups [5-7]. This phenomenon has been taken to suggest that the mental representation of a scene consists of an intermingling of sensory information and a schema that extrapolates the views of a presented scene [8], and it has been used to provide evidence for the role of the neocortex [9] and hippocampus [10, 11] in the schematization of scenes during memory. However, the study of boundary extension has typically focused on object-oriented images that are not representative of our visuospatial world. Here, using a broad set of 1,000 images tested on 2,000 participants in a rapid recognition task, we discover "boundary contraction" as an equally robust phenomenon. Further, image composition largely drives whether extension or contraction is observed-although object-oriented images cause more boundary extension, scene-oriented images cause more boundary contraction. Finally, these effects also occur during drawing tasks, including a task with minimal memory load-when participants copy an image during viewing. Collectively, these results show that boundary extension is not a universal phenomenon and put into question the assumption that scene memory automatically combines visual information with additional context derived from internal schema. Instead, our memory for a scene may be largely driven by its visual composition, with a tendency to extend or contract the boundaries equally likely.

Memorability of photographs in subjective cognitive decline and mild cognitive impairment: Implications for cognitive assessment

INTRODUCTION

Impaired long-term memory is a defining feature of mild cognitive impairment (MCI). We tested whether this impairment is item specific, limited to some memoranda, whereas some remain consistently memorable.

METHODS

We conducted item-based analyses of long-term visual recognition memory. Three hundred ninety-four participants (healthy controls, subjective cognitive decline [SCD], and MCI) in the multicentric DZNE-Longitudinal Cognitive Impairment and Dementia Study (DELCODE) were tested with images from a pool of 835 photographs.

RESULTS

We observed consistent memorability for images in healthy controls, SCD, and MCI, predictable by a neural network trained on another healthy sample. Looking at memorability differences between groups, we identified images that could successfully categorize group membership with higher success and a substantial image reduction than the original image set.

DISCUSSION

Individuals with SCD and MCI show consistent memorability for specific items, while other items show significant diagnosticity. Certain stimulus features could optimize diagnostic assessment, while others could support memory.

THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images

In recent years, the use of a large number of object concepts and naturalistic object images has been growing strongly in cognitive neuroscience research. Classical databases of object concepts are based mostly on a manually curated set of concepts. Further, databases of naturalistic object images typically consist of single images of objects cropped from their background, or a large number of naturalistic images of varying quality, requiring elaborate manual image curation. Here we provide a set of 1,854 diverse object concepts sampled systematically from concrete picturable and nameable nouns in the American English language. Using these object concepts, we conducted a large-scale web image search to compile a database of 26,107 high-quality naturalistic images of those objects, with 12 or more object images per concept and all images cropped to square size. Using crowdsourcing, we provide higher-level category membership for the 27 most common categories and validate them by relating them to representations in a semantic embedding derived from large text corpora. Finally, by feeding images through a deep convolutional neural network, we demonstrate that they exhibit high selectivity for different object concepts, while at the same time preserving variability of different object images within each concept. Together, the THINGS database provides a rich resource of object concepts and object images and offers a tool for both systematic and large-scale naturalistic research in the fields of psychology, neuroscience, and computer science.

Similarity judgments and cortical visual responses reflect different properties of object and scene categories in naturalistic images

Numerous factors have been reported to underlie the representation of complex images in high-level human visual cortex, including categories (e.g. faces, objects, scenes), animacy, and real-world size, but the extent to which this organization reflects behavioral judgments of real-world stimuli is unclear. Here, we compared representations derived from explicit behavioral similarity judgments and ultra-high field (7T) fMRI of human visual cortex for multiple exemplars of a diverse set of naturalistic images from 48 object and scene categories. While there was a significant correlation between similarity judgments and fMRI responses, there were striking differences between the two representational spaces. Behavioral judgements primarily revealed a coarse division between man-made (including humans) and natural (including animals) images, with clear groupings of conceptually-related categories (e.g. transportation, animals), while these conceptual groupings were largely absent in the fMRI representations. Instead, fMRI responses primarily seemed to reflect a separation of both human and non-human faces/bodies from all other categories. Further, comparison of the behavioral and fMRI representational spaces with those derived from the layers of a deep neural network (DNN) showed a strong correspondence with behavior in the top-most layer and with fMRI in the mid-level layers. These results suggest a complex relationship between localized responses in high-level visual cortex and behavioral similarity judgments - each domain reflects different properties of the images, and responses in high-level visual cortex may correspond to intermediate stages of processing between basic visual features and the conceptual categories that dominate the behavioral response.

Distinct subdivisions of human medial parietal cortex support recollection of people and places

Human medial parietal cortex (MPC) is implicated in multiple cognitive processes including memory recall, visual scene processing and navigation, and is a core component of the default mode network. Here, we demonstrate distinct subdivisions of MPC that are selectively recruited during memory recall of either specific people or places. First, distinct regions of MPC exhibited differential functional connectivity with medial and lateral regions of ventral temporal cortex (VTC). Second, these same medial regions showed selective, but negative, responses to the visual presentation of different stimulus categories, with clear preferences for scenes and faces. Finally, and most critically, these regions were differentially recruited during memory recall of either people or places with a strong familiarity advantage. Taken together, these data suggest that the organizing principle defining the medial-lateral axis of VTC is reflected in MPC, but in the context of memory recall.

Scene Perception in the Human Brain

Humans are remarkably adept at perceiving and understanding complex real-world scenes. Uncovering the neural basis of this ability is an important goal of vision science. Neuroimaging studies have identified three cortical regions that respond selectively to scenes: parahippocampal place area, retrosplenial complex/medial place area, and occipital place area. Here, we review what is known about the visual and functional properties of these brain areas. Scene-selective regions exhibit retinotopic properties and sensitivity to low-level visual features that are characteristic of scenes. They also mediate higher-level representations of layout, objects, and surface properties that allow individual scenes to be recognized and their spatial structure ascertained. Challenges for the future include developing computational models of information processing in scene regions, investigating how these regions support scene perception under ecologically realistic conditions, and understanding how they operate in the context of larger brain networks.

Visual responsiveness in sensorimotor cortex is increased following amputation and reduced after mirror therapy

Phantom limb pain (PLP) following amputation, which is experienced by the vast majority of amputees, has been reported to be relieved with daily sessions of mirror therapy. During each session, a mirror is used to view the reflected image of the intact limb moving, providing visual feedback consistent with the movement of the missing/phantom limb. To investigate potential neural correlates of the treatment effect, we measured brain responses in volunteers with unilateral leg amputation using functional magnetic resonance imaging (fMRI) during a four-week course of mirror therapy. Mirror therapy commenced immediately following baseline scans, which were repeated after approximately two and four week intervals. We focused on responses in the region of sensorimotor cortex corresponding to primary somatosensory and motor representations of the missing leg. At baseline, prior to starting therapy, we found a strong and unexpected response in sensorimotor cortex of amputees to visually presented images of limbs. This response was stronger for images of feet compared to hands and there was no such response in matched controls. Further, this response to visually presented limbs was no longer present at the end of the four week mirror therapy treatment, when perceived phantom limb pain was also reduced. A similar pattern of results was also observed in extrastriate and parietal regions typically responsive to viewing hand actions, but not in regions corresponding to secondary somatosensory cortex. Finally, there was a significant correlation between initial visual responsiveness in sensorimotor cortex and reduction in PLP suggesting a potential marker for predicting efficacy of mirror therapy. Thus, enhanced visual responsiveness in sensorimotor cortex is associated with PLP and modulated over the course of mirror therapy.

•

Visual responsiveness to the sight of limbs in sensorimotor cortex of leg amputees but not matched controls

•

Consistent with prior studies, mirror therapy over 4 weeks reduced phantom limb pain

•

Visual responsiveness in sensorimotor cortex of amputees diminished following mirror therapy

•

Visual responsiveness in sensorimotor cortex might be useful in predicting the potential efficacy of mirror therapy