The Ferrier Lecture 1995 behind the seen: the functional specialization of the brain in space and time

Modality-Specific Perceptual Learning of Vocoded Auditory versus Lipread Speech: Different Effects of Prior Information

Traditionally, speech perception training paradigms have not adequately taken into account the possibility that there may be modality-specific requirements for perceptual learning with auditory-only (AO) versus visual-only (VO) speech stimuli. The study reported here investigated the hypothesis that there are modality-specific differences in how prior information is used by normal-hearing participants during vocoded versus VO speech training. Two different experiments, one with vocoded AO speech (Experiment 1) and one with VO, lipread, speech (Experiment 2), investigated the effects of giving different types of prior information to trainees on each trial during training. The training was for four ~20 min sessions, during which participants learned to label novel visual images using novel spoken words. Participants were assigned to different types of prior information during training: Word Group trainees saw a printed version of each training word (e.g., "tethon"), and Consonant Group trainees saw only its consonants (e.g., "t_th_n"). Additional groups received no prior information (i.e., Experiment 1, AO Group; Experiment 2, VO Group) or a spoken version of the stimulus in a different modality from the training stimuli (Experiment 1, Lipread Group; Experiment 2, Vocoder Group). That is, in each experiment, there was a group that received prior information in the modality of the training stimuli from the other experiment. In both experiments, the Word Groups had difficulty retaining the novel words they attempted to learn during training. However, when the training stimuli were vocoded, the Word Group improved their phoneme identification. When the training stimuli were visual speech, the Consonant Group improved their phoneme identification and their open-set sentence lipreading. The results are considered in light of theoretical accounts of perceptual learning in relationship to perceptual modality.

Blurred Lines: Memory, Perceptions, and Consciousness: Commentary on "Consciousness as a Memory System" by Budson et al (2022)

In the previous issue, Budson, Richman, and Kensinger (2022) put forth the intriguing proposal that consciousness may have evolved from the episodic memory system. In addition to providing a possible evolutionary trajectory for consciousness, I believe that viewing consciousness as an extension of memory in this way is particularly useful for understanding some of the puzzling temporal complexities that are inherent to consciousness. For example, due to neural transmission delays, our conscious experience must necessarily lag the outside world, which creates a paradox for both conscious perception (Do we see the past, rather than the present?) and action (How can we make rapid decisions if it takes so long to become conscious of something?). These paradoxes can be elegantly solved by treating consciousness as a memory system. Finally, the proposal put forth by Budson and colleagues (2022) aligns with the emerging perspective that consciousness, like memory, represents a narrative time line of events rather than any single instant. However, I believe that this conceptualization can be further extended to include not only the past, but also the future. In this way, consciousness can be provocatively viewed as the remembered past, present, and future.

Neural Mechanisms and Psychology of Psychedelic Ego Dissolution

Neuroimaging studies of psychedelics have advanced our understanding of hierarchical brain organization and the mechanisms underlying their subjective and therapeutic effects. The primary mechanism of action of classic psychedelics is binding to serotonergic 5-HT2A receptors. Agonist activity at these receptors leads to neuromodulatory changes in synaptic efficacy that can have a profound effect on hierarchical message-passing in the brain. Here, we review the cognitive and neuroimaging evidence for the effects of psychedelics: in particular, their influence on selfhood and subject-object boundaries-known as ego dissolution-surmised to underwrite their subjective and therapeutic effects. Agonism of 5-HT2A receptors, located at the apex of the cortical hierarchy, may have a particularly powerful effect on sentience and consciousness. These effects can endure well after the pharmacological half-life, suggesting that psychedelics may have effects on neural plasticity that may play a role in their therapeutic efficacy. Psychologically, this may be accompanied by a disarming of ego resistance that increases the repertoire of perceptual hypotheses and affords alternate pathways for thought and behavior, including those that undergird selfhood. We consider the interaction between serotonergic neuromodulation and sentience through the lens of hierarchical predictive coding, which speaks to the value of psychedelics in understanding how we make sense of the world and specific predictions about effective connectivity in cortical hierarchies that can be tested using functional neuroimaging. SIGNIFICANCE STATEMENT: Classic psychedelics bind to serotonergic 5-HT2A receptors. Their agonist activity at these receptors leads to neuromodulatory changes in synaptic efficacy, resulting in a profound effect on information processing in the brain. Here, we synthesize an abundance of brain imaging research with pharmacological and psychological interpretations informed by the framework of predictive coding. Moreover, predictive coding is suggested to offer more sophisticated interpretations of neuroimaging findings by bridging the role between the 5-HT2A receptors and large-scale brain networks.

The evolution of brain architectures for predictive coding and active inference

This article considers the evolution of brain architectures for predictive processing. We argue that brain mechanisms for predictive perception and action are not late evolutionary additions of advanced creatures like us. Rather, they emerged gradually from simpler predictive loops (e.g. autonomic and motor reflexes) that were a legacy from our earlier evolutionary ancestors-and were key to solving their fundamental problems of adaptive regulation. We characterize simpler-to-more-complex brains formally, in terms of generative models that include predictive loops of increasing hierarchical breadth and depth. These may start from a simple homeostatic motif and be elaborated during evolution in four main ways: these include the multimodal expansion of predictive control into an allostatic loop; its duplication to form multiple sensorimotor loops that expand an animal's behavioural repertoire; and the gradual endowment of generative models with hierarchical depth (to deal with aspects of the world that unfold at different spatial scales) and temporal depth (to select plans in a future-oriented manner). In turn, these elaborations underwrite the solution to biological regulation problems faced by increasingly sophisticated animals. Our proposal aligns neuroscientific theorising-about predictive processing-with evolutionary and comparative data on brain architectures in different animal species. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

Stochastic Chaos and Markov Blankets

In this treatment of random dynamical systems, we consider the existence-and identification-of conditional independencies at nonequilibrium steady-state. These independencies underwrite a particular partition of states, in which internal states are statistically secluded from external states by blanket states. The existence of such partitions has interesting implications for the information geometry of internal states. In brief, this geometry can be read as a physics of sentience, where internal states look as if they are inferring external states. However, the existence of such partitions-and the functional form of the underlying densities-have yet to be established. Here, using the Lorenz system as the basis of stochastic chaos, we leverage the Helmholtz decomposition-and polynomial expansions-to parameterise the steady-state density in terms of surprisal or self-information. We then show how Markov blankets can be identified-using the accompanying Hessian-to characterise the coupling between internal and external states in terms of a generalised synchrony or synchronisation of chaos. We conclude by suggesting that this kind of synchronisation may provide a mathematical basis for an elemental form of (autonomous or active) sentience in biology.

Advances in Neuroscience, Not Devices, Will Determine the Effectiveness of Visual Prostheses

Background: Innovations in engineering and neuroscience have enabled the development of sophisticated visual prosthetic devices. In clinical trials, these devices have provided visual acuities as high as 20/460, enabled coarse navigation, and even allowed for reading of short words. However, long-term commercial viability arguably rests on attaining even better vision and more definitive improvements in tasks of daily living and quality of life. Purpose: Here we review technological and biological obstacles in the implementation of visual prosthetics. Conclusions: Research in the visual prosthetic field has tackled significant technical challenges, including biocompatibility, signal spread through neural tissue, and inadvertent activation of passing axons; however, significant gaps in knowledge remain in the realm of neuroscience, including the neural code of vision and visual plasticity. We assert that further optimization of prosthetic devices alone will not provide markedly improved visual outcomes without significant advances in our understanding of neuroscience.

Neural and phenotypic representation under the free-energy principle

The aim of this paper is to leverage the free-energy principle and its corollary process theory, active inference, to develop a generic, generalizable model of the representational capacities of living creatures; that is, a theory of phenotypic representation. Given their ubiquity, we are concerned with distributed forms of representation (e.g., population codes), whereby patterns of ensemble activity in living tissue come to represent the causes of sensory input or data. The active inference framework rests on the Markov blanket formalism, which allows us to partition systems of interest, such as biological systems, into internal states, external states, and the blanket (active and sensory) states that render internal and external states conditionally independent of each other. In this framework, the representational capacity of living creatures emerges as a consequence of their Markovian structure and nonequilibrium dynamics, which together entail a dual-aspect information geometry. This entails a modest representational capacity: internal states have an intrinsic information geometry that describes their trajectory over time in state space, as well as an extrinsic information geometry that allows internal states to encode (the parameters of) probabilistic beliefs about (fictive) external states. Building on this, we describe here how, in an automatic and emergent manner, information about stimuli can come to be encoded by groups of neurons bound by a Markov blanket; what is known as the neuronal packet hypothesis. As a concrete demonstration of this type of emergent representation, we present numerical simulations showing that self-organizing ensembles of active inference agents sharing the right kind of probabilistic generative model are able to encode recoverable information about a stimulus array.

Temporal recalibration in vision requires location-based binding

Occupying the same location and occurring at the same time are the essential spatial and temporal factors for different features of a natural event or object to be integrated. Audio-visual temporal recalibration, as a temporal integration mechanism, refers to the brain's capacity to perceive simultaneity by adjusting for differential delays in the transmission of auditory and visual signals. Co-localization of auditory and visual information, however, is found not to be necessary for audio-visual temporal recalibration to occur. Here, we show that after exposure to a time lag between a visual flash and a visual collision, simultaneity responses were shifted toward an adapt lag in a bound condition where the flash and collision belonged to the same object but not in a separate condition where the flash and collision belonged to spatially separated objects. The results demonstrate that location-based binding is a requisite for temporal recalibration within the visual modality. Our finding suggests that the brain takes the modality difference in object localization into consideration when integrating temporally asynchronous signals.

Minds and Brains, Sleep and Psychiatry

Objective

This article offers a philosophical thesis for psychiatric disorders that rests upon some simple truths about the mind and brain. Specifically, it asks whether the dual aspect monism—that emerges from sleep research and theoretical neurobiology—can be applied to pathophysiology and psychopathology in psychiatry.

Methods

Our starting point is that the mind and brain are emergent aspects of the same (neuronal) dynamics; namely, the brain–mind. Our endpoint is that synaptic dysconnection syndromes inherit the same dual aspect; namely, aberrant inference or belief updating on the one hand, and a failure of neuromodulatory synaptic gain control on the other. We start with some basic considerations from sleep research that integrate the phenomenology of dreaming with the neurophysiology of sleep.

Results

We then leverage this treatment by treating the brain as an organ of inference. Our particular focus is on the role of precision (i.e., the representation of uncertainty) in belief updating and the accompanying synaptic mechanisms.

Conclusions

Finally, we suggest a dual aspect approach—based upon belief updating (i.e., mind processes) and its neurophysiological implementation (i.e., brain processes)—has a wide explanatory compass for psychiatry and various movement disorders. This approach identifies the kind of pathophysiology that underwrites psychopathology—and points to certain psychotherapeutic and psychopharmacological targets, which may stand in mechanistic relation to each other.

The ‘mind’ emerges from Bayesian belief updating in the ‘brain’

Psychopathology can be read as aberrant belief updating.

Aberrant belief updating follows from any neuromodulatory synaptopathy

The emergence of synchrony in networks of mutually inferring neurons

This paper considers the emergence of a generalised synchrony in ensembles of coupled self-organising systems, such as neurons. We start from the premise that any self-organising system complies with the free energy principle, in virtue of placing an upper bound on its entropy. Crucially, the free energy principle allows one to interpret biological systems as inferring the state of their environment or external milieu. An emergent property of this inference is synchronisation among an ensemble of systems that infer each other. Here, we investigate the implications of neuronal dynamics by simulating neuronal networks, where each neuron minimises its free energy. We cast the ensuing ensemble dynamics in terms of inference and show that cardinal behaviours of neuronal networks - both in vivo and in vitro - can be explained by this framework. In particular, we test the hypotheses that (i) generalised synchrony is an emergent property of free energy minimisation; thereby explaining synchronisation in the resting brain: (ii) desynchronisation is induced by exogenous input; thereby explaining event-related desynchronisation and (iii) structure learning emerges in response to causal structure in exogenous input; thereby explaining functional segregation in real neuronal systems.

Multiscale integration: beyond internalism and externalism

We present a multiscale integrationist interpretation of the boundaries of cognitive systems, using the Markov blanket formalism of the variational free energy principle. This interpretation is intended as a corrective for the philosophical debate over internalist and externalist interpretations of cognitive boundaries; we stake out a compromise position. We first survey key principles of new radical (extended, enactive, embodied) views of cognition. We then describe an internalist interpretation premised on the Markov blanket formalism. Having reviewed these accounts, we develop our positive multiscale account. We argue that the statistical seclusion of internal from external states of the system-entailed by the existence of a Markov boundary-can coexist happily with the multiscale integration of the system through its dynamics. Our approach does not privilege any given boundary (whether it be that of the brain, body, or world), nor does it argue that all boundaries are equally prescient. We argue that the relevant boundaries of cognition depend on the level being characterised and the explanatory interests that guide investigation. We approach the issue of how and where to draw the boundaries of cognitive systems through a multiscale ontology of cognitive systems, which offers a multidisciplinary research heuristic for cognitive science.

In vitro neural networks minimise variational free energy

In this work, we address the neuronal encoding problem from a Bayesian perspective. Specifically, we ask whether neuronal responses in an in vitro neuronal network are consistent with ideal Bayesian observer responses under the free energy principle. In brief, we stimulated an in vitro cortical cell culture with stimulus trains that had a known statistical structure. We then asked whether recorded neuronal responses were consistent with variational message passing based upon free energy minimisation (i.e., evidence maximisation). Effectively, this required us to solve two problems: first, we had to formulate the Bayes-optimal encoding of the causes or sources of sensory stimulation, and then show that these idealised responses could account for observed electrophysiological responses. We describe a simulation of an optimal neural network (i.e., the ideal Bayesian neural code) and then consider the mapping from idealised in silico responses to recorded in vitro responses. Our objective was to find evidence for functional specialisation and segregation in the in vitro neural network that reproduced in silico learning via free energy minimisation. Finally, we combined the in vitro and in silico results to characterise learning in terms of trajectories in a variational information plane of accuracy and complexity.

Consciousness, Representation, Action: The Importance of Being Goal-Directed

Recent years have witnessed fierce debates on the dependence of consciousness on interactions between a subject and the environment. Reviewing neuroscientific, computational, and clinical evidence, I will address three questions. First, does conscious experience necessarily depend on acute interactions between a subject and the environment? Second, does it depend on specific perception-action loops in the longer run? Third, which types of action does consciousness cohere with, if not with all of them? I argue that conscious contents do not necessarily depend on acute or long-term brain-environment interactions. Instead, consciousness is proposed to be specifically associated with, and subserve, deliberate, goal-directed behavior (GDB). Brain systems implied in conscious representation are highly connected to, but distinct from, neural substrates mediating GDB and declarative memory.

Diagnosing the Neural Circuitry of Reading

We summarize the current state of knowledge of the brain's reading circuits, and then we describe opportunities to use quantitative and reproducible methods for diagnosing these circuits. Neural circuit diagnostics-by which we mean identifying the locations and responses in an individual that differ significantly from measurements in good readers-can help parents and educators select the best remediation strategy. A sustained effort to develop and share diagnostic methods can support the societal goal of improving literacy.

White matter microstructure of attentional networks predicts attention and consciousness functional interactions

Attention is considered as one of the pre-requisites of conscious perception. Phasic alerting and exogenous orienting improve conscious perception of near-threshold information through segregated brain networks. Using a multimodal neuroimaging approach, combining data from functional MRI (fMRI) and diffusion-weighted imaging (DWI), we investigated the influence of white matter properties of the ventral branch of superior longitudinal fasciculus (SLF III) in functional interactions between attentional systems and conscious perception. Results revealed that (1) reduced integrity of the left hemisphere SLF III was predictive of the neural interactions observed between exogenous orienting and conscious perception, and (2) increased integrity of the left hemisphere SLF III was predictive of the neural interactions observed between phasic alerting and conscious perception. Our results combining fMRI and DWI data demonstrate that structural properties of the white matter organization determine attentional modulations over conscious perception.

Spatial Embedding and Wiring Cost Constrain the Functional Layout of the Cortical Network of Rodents and Primates

Mammals show a wide range of brain sizes, reflecting adaptation to diverse habitats. Comparing interareal cortical networks across brains of different sizes and mammalian orders provides robust information on evolutionarily preserved features and species-specific processing modalities. However, these networks are spatially embedded, directed, and weighted, making comparisons challenging. Using tract tracing data from macaque and mouse, we show the existence of a general organizational principle based on an exponential distance rule (EDR) and cortical geometry, enabling network comparisons within the same model framework. These comparisons reveal the existence of network invariants between mouse and macaque, exemplified in graph motif profiles and connection similarity indices, but also significant differences, such as fractionally smaller and much weaker long-distance connections in the macaque than in mouse. The latter lends credence to the prediction that long-distance cortico-cortical connections could be very weak in the much-expanded human cortex, implying an increased susceptibility to disconnection syndromes such as Alzheimer disease and schizophrenia. Finally, our data from tracer experiments involving only gray matter connections in the primary visual areas of both species show that an EDR holds at local scales as well (within 1.5 mm), supporting the hypothesis that it is a universally valid property across all scales and, possibly, across the mammalian class.

The Functional Anatomy of Time: What and When in the Brain

This Opinion article considers the implications for functional anatomy of how we represent temporal structure in our exchanges with the world. It offers a theoretical treatment that tries to make sense of the architectural principles seen in mammalian brains. Specifically, it considers a factorisation between representations of temporal succession and representations of content or, heuristically, a segregation into when and what. This segregation may explain the central role of the hippocampus in neuronal hierarchies while providing a tentative explanation for recent observations of how ordinal sequences are encoded. The implications for neuroanatomy and physiology may have something important to say about how self-organised cell assembly sequences enable the brain to exhibit purposeful behaviour that transcends the here and now.

Towards a Neuronal Gauge Theory

Given the amount of knowledge and data accruing in the neurosciences, is it time to formulate a general principle for neuronal dynamics that holds at evolutionary, developmental, and perceptual timescales? In this paper, we propose that the brain (and other self-organised biological systems) can be characterised via the mathematical apparatus of a gauge theory. The picture that emerges from this approach suggests that any biological system (from a neuron to an organism) can be cast as resolving uncertainty about its external milieu, either by changing its internal states or its relationship to the environment. Using formal arguments, we show that a gauge theory for neuronal dynamics--based on approximate Bayesian inference--has the potential to shed new light on phenomena that have thus far eluded a formal description, such as attention and the link between action and perception.

Neural pathways for visual speech perception

This paper examines the questions, what levels of speech can be perceived visually, and how is visual speech represented by the brain? Review of the literature leads to the conclusions that every level of psycholinguistic speech structure (i.e., phonetic features, phonemes, syllables, words, and prosody) can be perceived visually, although individuals differ in their abilities to do so; and that there are visual modality-specific representations of speech qua speech in higher-level vision brain areas. That is, the visual system represents the modal patterns of visual speech. The suggestion that the auditory speech pathway receives and represents visual speech is examined in light of neuroimaging evidence on the auditory speech pathways. We outline the generally agreed-upon organization of the visual ventral and dorsal pathways and examine several types of visual processing that might be related to speech through those pathways, specifically, face and body, orthography, and sign language processing. In this context, we examine the visual speech processing literature, which reveals widespread diverse patterns of activity in posterior temporal cortices in response to visual speech stimuli. We outline a model of the visual and auditory speech pathways and make several suggestions: (1) The visual perception of speech relies on visual pathway representations of speech qua speech. (2) A proposed site of these representations, the temporal visual speech area (TVSA) has been demonstrated in posterior temporal cortex, ventral and posterior to multisensory posterior superior temporal sulcus (pSTS). (3) Given that visual speech has dynamic and configural features, its representations in feedforward visual pathways are expected to integrate these features, possibly in TVSA.

Multisensory training can promote or impede visual perceptual learning of speech stimuli: visual-tactile vs. visual-auditory training

In a series of studies we have been investigating how multisensory training affects unisensory perceptual learning with speech stimuli. Previously, we reported that audiovisual (AV) training with speech stimuli can promote auditory-only (AO) perceptual learning in normal-hearing adults but can impede learning in congenitally deaf adults with late-acquired cochlear implants. Here, impeder and promoter effects were sought in normal-hearing adults who participated in lipreading training. In Experiment 1, visual-only (VO) training on paired associations between CVCVC nonsense word videos and nonsense pictures demonstrated that VO words could be learned to a high level of accuracy even by poor lipreaders. In Experiment 2, visual-auditory (VA) training in the same paradigm but with the addition of synchronous vocoded acoustic speech impeded VO learning of the stimuli in the paired-associates paradigm. In Experiment 3, the vocoded AO stimuli were shown to be less informative than the VO speech. Experiment 4 combined vibrotactile speech stimuli with the visual stimuli during training. Vibrotactile stimuli were shown to promote visual perceptual learning. In Experiment 5, no-training controls were used to show that training with visual speech carried over to consonant identification of untrained CVCVC stimuli but not to lipreading words in sentences. Across this and previous studies, multisensory training effects depended on the functional relationship between pathways engaged during training. Two principles are proposed to account for stimulus effects: (1) Stimuli presented to the trainee’s primary perceptual pathway will impede learning by a lower-rank pathway. (2) Stimuli presented to the trainee’s lower rank perceptual pathway will promote learning by a higher-rank pathway. The mechanisms supporting these principles are discussed in light of multisensory reverse hierarchy theory (RHT).

A moving overlay shrinks the attentional blink

This report describes a study examining the effects of overlaying a veil of spots on the letters in a central rapid serial visual presentation stream. Observers identified two target letters (T1 and T2, respectively) embedded in a stream of distractor letters printed in a different color. In Experiment 1, the attentional blink (AB) diminished when a different overlay veiled each letter, such that the spots appeared to move as the letters changed. Experiment 2 concerned whether the performance enhancement occurred because the overlay hampered processing of the lag 1 distractor, thus weakening the distractor's interference with T1. Experiment 3 focused on how changing the overlay at or around T1 affected the AB. The attention disengagement hypothesis was proposed to explain the common theme in the results-that performance was only enhanced when different overlays were applied to the T1 and lag 1 frames. The claim is that the AB reflects a failure of prompt attentional disengagement from T1, which, in turn, delays reengagement when T2 appears shortly thereafter. When T1's disappearance is accompanied by an overlay change, the perceptual system gets an additional cue signaling that the visual scene has changed, thereby inducing attentional disengagement. Apart from facilitating prompt reengagement at the next target, earlier disengagement also improves target recovery by excluding features of the trailing item, likely to be a distractor, from working memory.

Unimodal primary sensory cortices are directly connected by long-range horizontal projections in the rat sensory cortex

Research based on functional imaging and neuronal recordings in the barrel cortex subdivision of primary somatosensory cortex (SI) of the adult rat has revealed novel aspects of structure-function relationships in this cortex. Specifically, it has demonstrated that single whisker stimulation evokes subthreshold neuronal activity that spreads symmetrically within gray matter from the appropriate barrel area, crosses cytoarchitectural borders of SI and reaches deeply into other unimodal primary cortices such as primary auditory (AI) and primary visual (VI). It was further demonstrated that this spread is supported by a spatially matching underlying diffuse network of border-crossing, long-range projections that could also reach deeply into AI and VI. Here we seek to determine whether such a network of border-crossing, long-range projections is unique to barrel cortex or characterizes also other primary, unimodal sensory cortices and therefore could directly connect them. Using anterograde (BDA) and retrograde (CTb) tract-tracing techniques, we demonstrate that such diffuse horizontal networks directly and mutually connect VI, AI and SI. These findings suggest that diffuse, border-crossing axonal projections connecting directly primary cortices are an important organizational motif common to all major primary sensory cortices in the rat. Potential implications of these findings for topics including cortical structure-function relationships, multisensory integration, functional imaging, and cortical parcellation are discussed.

Audiovisual spoken word training can promote or impede auditory-only perceptual learning: prelingually deafened adults with late-acquired cochlear implants versus normal hearing adults

Training with audiovisual (AV) speech has been shown to promote auditory perceptual learning of vocoded acoustic speech by adults with normal hearing. In Experiment 1, we investigated whether AV speech promotes auditory-only (AO) perceptual learning in prelingually deafened adults with late-acquired cochlear implants. Participants were assigned to learn associations between spoken disyllabic C(=consonant)V(=vowel)CVC non-sense words and non-sense pictures (fribbles), under AV and then AO (AV-AO; or counter-balanced AO then AV, AO-AV, during Periods 1 then 2) training conditions. After training on each list of paired-associates (PA), testing was carried out AO. Across all training, AO PA test scores improved (7.2 percentage points) as did identification of consonants in new untrained CVCVC stimuli (3.5 percentage points). However, there was evidence that AV training impeded immediate AO perceptual learning: During Period-1, training scores across AV and AO conditions were not different, but AO test scores were dramatically lower in the AV-trained participants. During Period-2 AO training, the AV-AO participants obtained significantly higher AO test scores, demonstrating their ability to learn the auditory speech. Across both orders of training, whenever training was AV, AO test scores were significantly lower than training scores. Experiment 2 repeated the procedures with vocoded speech and 43 normal-hearing adults. Following AV training, their AO test scores were as high as or higher than following AO training. Also, their CVCVC identification scores patterned differently than those of the cochlear implant users. In Experiment 1, initial consonants were most accurate, and in Experiment 2, medial consonants were most accurate. We suggest that our results are consistent with a multisensory reverse hierarchy theory, which predicts that, whenever possible, perceivers carry out perceptual tasks immediately based on the experience and biases they bring to the task. We point out that while AV training could be an impediment to immediate unisensory perceptual learning in cochlear implant patients, it was also associated with higher scores during training.

Dr. Airy's "morbid affection of the eyesight": lessons from Teichopsia Circa 1870

Hubert Airy's iconic drawing of his own migraine visual aura for which he coined the term, "teichopsia," conveys important lessons for the contemporary clinician. His observations of the expansion ("build-up"), minification/magnification, and color/achromatopsia of migrainous teichopsia are consistent with (and possibly anticipatory of) the later discoveries of cortical spreading depression, cortical magnification of primary visual cortex (V1), and specialized cortical centers for color vision.

Cortical high-density counterstream architectures

Small-world networks provide an appealing description of cortical architecture owing to their capacity for integration and segregation combined with an economy of connectivity. Previous reports of low-density interareal graphs and apparent small-world properties are challenged by data that reveal high-density cortical graphs in which economy of connections is achieved by weight heterogeneity and distance-weight correlations. These properties define a model that predicts many binary and weighted features of the cortical network including a core-periphery, a typical feature of self-organizing information processing systems. Feedback and feedforward pathways between areas exhibit a dual counterstream organization, and their integration into local circuits constrains cortical computation. Here, we propose a bow-tie representation of interareal architecture derived from the hierarchical laminar weights of pathways between the high-efficiency dense core and periphery.

Life as we know it

This paper presents a heuristic proof (and simulations of a primordial soup) suggesting that life-or biological self-organization-is an inevitable and emergent property of any (ergodic) random dynamical system that possesses a Markov blanket. This conclusion is based on the following arguments: if the coupling among an ensemble of dynamical systems is mediated by short-range forces, then the states of remote systems must be conditionally independent. These independencies induce a Markov blanket that separates internal and external states in a statistical sense. The existence of a Markov blanket means that internal states will appear to minimize a free energy functional of the states of their Markov blanket. Crucially, this is the same quantity that is optimized in Bayesian inference. Therefore, the internal states (and their blanket) will appear to engage in active Bayesian inference. In other words, they will appear to model-and act on-their world to preserve their functional and structural integrity, leading to homoeostasis and a simple form of autopoiesis.

Auditory Perceptual Learning for Speech Perception Can be Enhanced by Audiovisual Training

Speech perception under audiovisual (AV) conditions is well known to confer benefits to perception such as increased speed and accuracy. Here, we investigated how AV training might benefit or impede auditory perceptual learning of speech degraded by vocoding. In Experiments 1 and 3, participants learned paired associations between vocoded spoken nonsense words and nonsense pictures. In Experiment 1, paired-associates (PA) AV training of one group of participants was compared with audio-only (AO) training of another group. When tested under AO conditions, the AV-trained group was significantly more accurate than the AO-trained group. In addition, pre- and post-training AO forced-choice consonant identification with untrained nonsense words showed that AV-trained participants had learned significantly more than AO participants. The pattern of results pointed to their having learned at the level of the auditory phonetic features of the vocoded stimuli. Experiment 2, a no-training control with testing and re-testing on the AO consonant identification, showed that the controls were as accurate as the AO-trained participants in Experiment 1 but less accurate than the AV-trained participants. In Experiment 3, PA training alternated AV and AO conditions on a list-by-list basis within participants, and training was to criterion (92% correct). PA training with AO stimuli was reliably more effective than training with AV stimuli. We explain these discrepant results in terms of the so-called “reverse hierarchy theory” of perceptual learning and in terms of the diverse multisensory and unisensory processing resources available to speech perception. We propose that early AV speech integration can potentially impede auditory perceptual learning; but visual top-down access to relevant auditory features can promote auditory perceptual learning.

Roles of contour and surface processing in microgenesis of object perception and visual consciousness

Developments in visual neuroscience and neural-network modeling indicate the existence of separate pathways for the processing of form and surface attributes of a visual object. In line with prior theoretical proposals, it is assumed that the processing of form can be explicit or conscious only as or after the surface property such as color is filled in. In conjunction with extant psychophysical findings, these developments point to interesting distinctions between nonconscious and conscious processing of these attributes, specifically in relation to distinguishable temporal dynamics. At nonconscious levels form processing proceeds faster than surface processing, whereas in contrast, at conscious levels form processing proceeds slower than surface processing. I mplications of separate form and surface processing for current and future psychophysical and neuroscientific research, particularly that relating cortical oscillations to conjunctions of surface and form features, and for cognitive science and philosophy of mind and consciousness are discussed.

Cortical responses to consciousness of schematic emotional facial expressions: a high-resolution EEG study

Is conscious perception of emotional face expression related to enhanced cortical responses? Electroencephalographic data (112 channels) were recorded in 15 normal adults during the presentation of cue stimuli with neutral, happy or sad schematic faces (duration: "threshold time" inducing about 50% of correct recognitions), masking stimuli (2 s), and go stimuli with happy or sad schematic faces (0.5 s). The subjects clicked left (right) mouse button in response to go stimuli with happy (sad) faces. After the response, they said "seen" or "not seen" with reference to previous cue stimulus. Electroencephalographic data formed visual event-related potentials (ERPs). Cortical sources of ERPs were estimated by LORETA software. Reaction time to go stimuli was generally shorter during "seen" than "not seen" trials, possibly due to covert attention and awareness. The cue stimuli evoked four ERP components (posterior N100, N170, P200, and P300), which had similar peak latency in the "not seen" and "seen" ERPs. Only N170 amplitude showed differences in amplitude in the "seen" versus "not seen" ERPs. Compared to the "not seen" ERPs, the "seen" ones showed prefrontal, premotor, and posterior parietal sources of N170 higher in amplitude with the sad cue stimuli and lower in amplitude with the neutral and happy cue stimuli. These results suggest that nonconscious and conscious processing of schematic emotional facial expressions shares a similar temporal evolution of cortical activity, and conscious processing induces an early enhancement of bilateral cortical activity for the schematic sad facial expressions (N170).

Learning to see words

Skilled reading requires recognizing written words rapidly; functional neuroimaging research has clarified how the written word initiates a series of responses in visual cortex. These responses are communicated to circuits in ventral occipitotemporal (VOT) cortex that learn to identify words rapidly. Structural neuroimaging has further clarified aspects of the white matter pathways that communicate reading signals between VOT and language systems. We review this circuitry, its development, and its deficiencies in poor readers. This review emphasizes data that measure the cortical responses and white matter pathways in individual subjects rather than group differences. Such methods have the potential to clarify why a child has difficulty learning to read and to offer guidance about the interventions that may be useful for that child.

Imaging retinotopic maps in the human brain

A quarter-century ago visual neuroscientists had little information about the number and organization of retinotopic maps in human visual cortex. The advent of functional magnetic resonance imaging (MRI), a non-invasive, spatially-resolved technique for measuring brain activity, provided a wealth of data about human retinotopic maps. Just as there are differences amongst non-human primate maps, the human maps have their own unique properties. Many human maps can be measured reliably in individual subjects during experimental sessions lasting less than an hour. The efficiency of the measurements and the relatively large amplitude of functional MRI signals in visual cortex make it possible to develop quantitative models of functional responses within specific maps in individual subjects. During this last quarter-century, there has also been significant progress in measuring properties of the human brain at a range of length and time scales, including white matter pathways, macroscopic properties of gray and white matter, and cellular and molecular tissue properties. We hope the next 25years will see a great deal of work that aims to integrate these data by modeling the network of visual signals. We do not know what such theories will look like, but the characterization of human retinotopic maps from the last 25years is likely to be an important part of future ideas about visual computations.

On the emergence of primary visual perception

We propose a concise novel conceptual and biological framework for the analysis of primary visual perception (PVP) that refers to the most basic levels of our awake subjective visual experiences. Neural representations for image content elaborated within V1/V2 and the early occipitotemporal (ventral) loop remain only latent with respect to PVP until spatially localized with respect to an attending observer. This process requires more than the downstream deployment of attentional resources onto targeted neurons. Additionally, the source neurons for such processes must be linked to a neural representation subserving a first-person perspective. We hypothesize that the simultaneous emergence of both the perceptual experience of image content and the personal inference of its ownership requires the resolution of any conflicting neuronal signaling between afferent and recurrent projections within and between both the ventral and dorsal streams. The V1/V2 complex and ventral cortical areas V3 and the V4 complex together with dorsal cortical areas LIP, VIP, and 7a with additional contributions from the motion areas V5/MT (middle temporal area), FST (fundus of superior temporal area), and MST (medial superior temporal area) together with their subcortical dependencies have the physiological properties required to constitute a "posterior perceptual core" that encodes the normal primary perceptual experience of image content, space, and sense of minimal self.

Spatially localized time shifts of the perceptual stream

Visual events trigger representations in different locations and times in the brain. In experience, however, these various neural responses refer to a single unified cause. To investigate how representations might be brought into temporal alignment, we attempted to locally manipulate neural processing in such a way that identical, simultaneous sequences would appear temporally misaligned. After adaptation to a 20 Hz sequentially expanding and contracting concentric grating, a running clock presented in the adapted region of the visual field appeared advanced relative to an identical clock presented simultaneously in an unadapted region. No such effect was observed following 5-Hz adaptation. Clock time reports following an exogenous cue showed the same effect of adaptation on perceived time, demonstrating that the apparent temporal misalignment was not mediated by differences in target selection or allocation of attention. This effect was not mediated by the apparent speed of the adapted clock: a clock in a 20-Hz-adapted spatial location appeared slower than a clock in a 5-Hz-adapted location, rather than faster. Furthermore, reaction times for a clock-hand orientation discrimination task were the same following 5- and 20-Hz adaptation, indicating that neural processing latencies were not differentially affected. Altogether, these findings suggest that the fragmented perceptual stream might be actively brought into temporal alignment through adaptive local mechanisms operating in spatially segregated regions of the visual field.

Cortical sources of visual evoked potentials during consciousness of executive processes

What is the timing of cortical activation related to consciousness of visuo-spatial executive functions? Electroencephalographic data (128 channels) were recorded in 13 adults. Cue stimulus briefly appeared on right or left (equal probability) monitor side for a period, inducing about 50% of recognitions. It was then masked and followed (2 s) by a central visual go stimulus. Left (right) mouse button had to be clicked after right (left) cue stimulus. This "inverted" response indexed executive processes. Afterward, subjects said "seen" if they had detected the cue stimulus or "not seen" when it was missed. Sources of event-related potentials (ERPs) were estimated by LORETA software. The inverted responses were about 95% in seen trials and about 60% in not seen trials. Cue stimulus evoked frontal-parietooccipital potentials, having the same peak latencies in the seen and not seen data. Maximal difference in amplitude of the seen and not seen ERPs was detected at about +300-ms post-stimulus (P3). P3 sources were higher in amplitude in the seen than not seen trials in dorsolateral prefrontal, premotor and parietooccipital areas. This was true in dorsolateral prefrontal and premotor cortex even when percentage of the inverted responses and reaction time were paired in the seen and not seen trials. These results suggest that, in normal subjects, the primary consciousness enhances the efficacy of visuo-spatial executive processes and is sub-served by a late (100- to 400-ms post-stimulus) enhancement of the neural synchronization in frontal areas.

Large-scale organization of rat sensorimotor cortex based on a motif of large activation spreads

Parcellation according to function (e.g., visual, somatosensory, auditory, motor) is considered a fundamental property of sensorimotor cortical organization, traditionally defined from cytoarchitectonics and mapping studies relying on peak evoked neuronal activity. In the adult rat, stimulation of single whiskers evokes peak activity at topographically appropriate locations within somatosensory cortex and provides an example of cortical functional specificity. Here, we show that single whisker stimulation also evokes symmetrical areas of suprathreshold and subthreshold neuronal activation that spread extensively away from peak activity, effectively ignoring cortical borders by spilling deeply into multiple cortical territories of different modalities (auditory, visual and motor), where they were blocked by localized neuronal activity blocker injections and thus ruled out as possibly caused by "volume conductance." These symmetrical activity spreads were supported by underlying border-crossing, long-range horizontal connections as confirmed with transection experiments and injections of anterograde neuronal tracer experiments. We found such large evoked activation spreads and their underlying connections regardless of whisker identity, cortical layer, or axis of recorded responses, thereby revealing a large scale nonspecific organization of sensorimotor cortex based on a motif of large symmetrical activation spreads. Because the large activation spreads and their underlying horizontal connections ignore anatomical borders between cortical modalities, sensorimotor cortex could therefore be viewed as a continuous entity rather than a collection of discrete, delineated unimodal regions, an organization that could coexist with established specificity of cortical organization and that could serve as a substrate for associative learning, direct multimodal integration and recovery of function after injury.

Relationship between color discrimination and neural responses in the inferior temporal cortex of the monkey

Earlier studies suggest that the inferior temporal (IT) cortex of the monkey plays a key role in color discrimination. Here, we examined the quantitative relationship between color judgment in monkeys and the responses of color-selective neurons in the anterior part of the IT cortex (area TE) by comparing neuronal activity and behavior recorded simultaneously while the monkeys performed a color-judgment task. We first compared the abilities of single neurons and monkeys to discriminate color. To calculate a neuron's ability to discriminate color, we computed a neurometric function using receiver-operating-characteristics analysis. We then compared the neural and behavioral thresholds for color discrimination and found that, in general, the neural threshold was higher than the behavioral threshold, although occasionally the reverse was true. Variation in the neural threshold across the color space corresponded well with that of the behavioral threshold. We then calculated the choice probability (CP), which is a measure of the correlation between the trial-to-trial fluctuations in neuronal responses and the monkeys' color judgment. On average, CPs were slightly but significantly greater than 0.5, indicating the activities of these TE neurons correlate positively with the monkeys' color judgment. This suggests that individual color-selective TE neurons only weakly contribute to color discrimination and that a large population of color-selective TE neurons contribute to the performance of color discrimination.

The encoding of temporally irregular and regular visual patterns in the human brain

In the work reported here, we set out to study the neural systems that detect predictable temporal patterns and departures from them. We used functional magnetic resonance imaging (fMRI) to locate activity in the brains of subjects when they viewed temporally regular and irregular patterns produced by letters, numbers, colors and luminance. Activity induced by irregular sequences was located within dorsolateral prefrontal cortex, including an area that was responsive to irregular patterns regardless of the type of visual stimuli producing them. Conversely, temporally regular arrangements resulted in activity in the right frontal lobe (medial frontal gyrus), in the left orbito-frontal cortex and in the left pallidum. The results show that there is an abstractive system in the brain for detecting temporal irregularity, regardless of the source producing it.

Reciprocal Benefits of Mass-Univariate and Multivariate Modeling in Brain Mapping: Applications to Event-Related Functional MRI, H(2) (15)O-, and FDG-PET

In brain mapping studies of sensory, cognitive, and motor operations, specific waveforms of dynamic neural activity are predicted based on theoretical models of human information processing. For example in event-related functional MRI (fMRI), the general linear model (GLM) is employed in mass-univariate analyses to identify the regions whose dynamic activity closely matches the expected waveforms. By comparison multivariate analyses based on PCA or ICA provide greater flexibility in detecting spatiotemporal properties of experimental data that may strongly support alternative neuroscientific explanations. We investigated conjoint multivariate and mass-univariate analyses that combine the capabilities to (1) verify activation of neural machinery we already understand and (2) discover reliable signatures of new neural machinery. We examined combinations of GLM and PCA that recover latent neural signals (waveforms and footprints) with greater accuracy than either method alone. Comparative results are illustrated with analyses of real fMRI data, adding to Monte Carlo simulation support.