The effect of head motion on brain age prediction using deep convolutional neural networks

Deep learning can be used effectively to predict participants' age from brain magnetic resonance imaging (MRI) data, and a growing body of evidence suggests that the difference between predicted and chronological age-referred to as brain-predicted age difference (brain-PAD)-is related to various neurological and neuropsychiatric disease states. A crucial aspect of the applicability of brain-PAD as a biomarker of individual brain health is whether and how brain-predicted age is affected by MR image artifacts commonly encountered in clinical settings. To investigate this issue, we trained and validated two different 3D convolutional neural network architectures (CNNs) from scratch and tested the models on a separate dataset consisting of motion-free and motion-corrupted T1-weighted MRI scans from the same participants, the quality of which were rated by neuroradiologists from a clinical diagnostic point of view. Our results revealed a systematic increase in brain-PAD with worsening image quality for both models. This effect was also observed for images that were deemed usable from a clinical perspective, with brains appearing older in medium than in good quality images. These findings were also supported by significant associations found between the brain-PAD and standard image quality metrics indicating larger brain-PAD for lower-quality images. Our results demonstrate a spurious effect of advanced brain aging as a result of head motion and underline the importance of controlling for image quality when using brain-predicted age based on structural neuroimaging data as a proxy measure for brain health.

Finding Pattern in the Noise: Persistent Implicit Statistical Knowledge Impacts the Processing of Unpredictable Stimuli

Humans can extract statistical regularities of the environment to predict upcoming events. Previous research recognized that implicitly acquired statistical knowledge remained persistent and continued to influence behavior even when the regularities were no longer present in the environment. Here, in an fMRI experiment, we investigated how the persistence of statistical knowledge is represented in the brain. Participants (n = 32) completed a visual, four-choice, RT task consisting of statistical regularities. Two types of blocks constantly alternated with one another throughout the task: predictable statistical regularities in one block type and unpredictable ones in the other. Participants were unaware of the statistical regularities and their changing distribution across the blocks. Yet, they acquired the statistical regularities and showed significant statistical knowledge at the behavioral level not only in the predictable blocks but also in the unpredictable ones, albeit to a smaller extent. Brain activity in a range of cortical and subcortical areas, including early visual cortex, the insula, the right inferior frontal gyrus, and the right globus pallidus/putamen contributed to the acquisition of statistical regularities. The right insula, inferior frontal gyrus, and hippocampus as well as the bilateral angular gyrus seemed to play a role in maintaining this statistical knowledge. The results altogether suggest that statistical knowledge could be exploited in a relevant, predictable context as well as transmitted to and retrieved in an irrelevant context without a predictable structure.

Study protocol of the Hungarian Longitudinal Study of Healthy Brain Aging (HuBA)

Background and purpose

Neuro­cog­nitive aging and the associated brain diseases impose a major social and economic burden. Therefore, substantial efforts have been put into revealing the lifestyle, the neurobiological and the genetic underpinnings of healthy neurocognitive aging. However, these studies take place almost exclusively in a limited number of highly-developed countries. Thus, it is an important open question to what extent their findings may generalize to neurocognitive aging in other, not yet investigated regions. The purpose of the Hungarian Longitudinal Study of Healthy Brain Aging (HuBA) is to collect multi-modal longitudinal data on healthy neurocognitive aging to address the data gap in this field in Central and Eastern Europe.

.

Methods

We adapted the Australian Ima­ging, Biomarkers and Lifestyle (AIBL) study of aging study protocol to local circumstances and collected demographic, lifestyle, men­tal and physical health, medication and medical history related information as well as re­cor­ded a series of magnetic resonance imaging (MRI) data. In addition, participants were al­so offered to participate in the collection of blood samples to assess circulating in­flam­matory biomarkers as well as a sleep study aimed at evaluating the general sleep quality based on multi-day collection of subjective sleep questionnaires and whole-night elec­troencephalographic (EEG) data.

.

Results

Baseline data collection has al­ready been accomplished for more than a hundred participants and data collection in the se­cond
session is on the way. The collected data might reveal specific local trends or could also indicate the generalizability of previous findings. Moreover, as the HuBA protocol al­so offers a sleep study designed for tho­rough characterization of participants’ sleep quality and related factors, our extended multi-modal dataset might provide a base for incorporating these measures into healthy and clinical aging research. 

.

Conclusion

Besides its straightforward na­tional benefits in terms of health ex­pen­di­ture, we hope that this Hungarian initiative could provide results valid for the whole Cent­ral and Eastern European region and could also promote aging and Alzheimer’s disease research in these countries.

.

Automatic brain MRI motion artifact detection based on end-to-end deep learning is similarly effective as traditional machine learning trained on image quality metrics

Head motion artifacts in magnetic resonance imaging (MRI) are an important confounding factor concerning brain research as well as clinical practice. For this reason, several machine learning-based methods have been developed for the automatic quality control of structural MRI scans. Deep learning offers a promising solution to this problem, however, given its data-hungry nature and the scarcity of expert-annotated datasets, its advantage over traditional machine learning methods in identifying motion-corrupted brain scans is yet to be determined. In the present study, we investigated the relative advantage of the two methods in structural MRI quality control. To this end, we collected publicly available T1-weighted images and scanned subjects in our own lab under conventional and active head motion conditions. The quality of the images was rated by a team of radiologists from the point of view of clinical diagnostic use. We present a relatively simple, lightweight 3D convolutional neural network trained in an end-to-end manner that achieved a test set (N = 411) balanced accuracy of 94.41% in classifying brain scans into clinically usable or unusable categories. A support vector machine trained on image quality metrics achieved a balanced accuracy of 88.44% on the same test set. Statistical comparison of the two models yielded no significant difference in terms of confusion matrices, error rates, or receiver operating characteristic curves. Our results suggest that these machine learning methods are similarly effective in identifying severe motion artifacts in brain MRI scans, and underline the efficacy of end-to-end deep learning-based systems in brain MRI quality control, allowing the rapid evaluation of diagnostic utility without the need for elaborate image pre-processing.

Lateralization of orthographic processing in fixed-gaze and natural reading conditions

Lateralized processing of orthographic information is a hallmark of proficient reading. However, how this finding obtained for fixed-gaze processing of orthographic stimuli translates to ecologically valid reading conditions remained to be clarified. To address this shortcoming, here we assessed the lateralization of early orthographic processing in fixed-gaze and natural reading conditions using concurrent eye-tracking and EEG data recorded from young adults without reading difficulties. Sensor-space analyses confirmed the well-known left-lateralized negative-going deflection of fixed-gaze EEG activity throughout the period of early orthographic processing. At the same time, fixation-related EEG activity exhibited left-lateralized followed by right-lateralized processing of text stimuli during natural reading. A strong positive relationship was found between the early leftward lateralization in fixed-gaze and natural reading conditions. Using source-space analyses, early left-lateralized brain activity was obtained in lateraloccipital and posterior ventral occipito-temporal cortices reflecting letter-level processing in both conditions. In addition, in the same time interval, left-lateralized source activity was found also in premotor and parietal brain regions during natural reading. While brain activity remained left-lateralized in later stages representing word-level processing in posterior and middle ventral temporal regions in the fixed-gaze condition, fixation-related source activity became stronger in the right hemisphere in medial and more anterior ventral temporal brain regions indicating higher-level processing of orthographic information. Although our results show a strong positive relationship between the lateralization of letter-level processing in the two reading modes and suggest lateralized brain activity as a general marker for processing of orthographic information, they also clearly indicate the need for reading research in ecologically valid conditions to identify the neural basis of visuospatial attentional, oculomotor and higher-level processes specific to natural reading.

Movement-related artefacts (MR-ART) dataset of matched motion-corrupted and clean structural MRI brain scans

Magnetic Resonance Imaging (MRI) provides a unique opportunity to investigate neural changes in healthy and clinical conditions. Its large inherent susceptibility to motion, however, often confounds the measurement. Approaches assessing, correcting, or preventing motion corruption of MRI measurements are under active development, and such efforts can greatly benefit from carefully controlled datasets. We present a unique dataset of structural brain MRI images collected from 148 healthy adults which includes both motion-free and motion-affected data acquired from the same participants. This matched dataset allows direct evaluation of motion artefacts, their impact on derived data, and testing approaches to correct for them. Our dataset further stands out by containing images with different levels of motion artefacts from the same participants, is enriched with expert scoring characterizing the image quality from a clinical point of view and is also complemented with standard image quality metrics obtained from MRIQC. The goal of the dataset is to raise awareness of the issue and provide a useful resource to assess and improve current motion correction approaches.

Lateralization of early orthographic processing during natural reading is impaired in developmental dyslexia

Skilled reading requires specialized visual cortical processing of orthographic information and its impairment has been proposed as a potential correlate of compromised reading in dyslexia. However, which stage of orthographic information processing during natural reading is disturbed in dyslexics remains unexplored. Here we addressed this question by simultaneously measuring the eye movements and EEG of dyslexic and control young adults during natural reading. Isolated meaningful sentences were presented at five inter-letter spacing levels spanning the range from minimal to extra-large spacing, and participants were instructed to read the text silently at their own pace. Control participants read faster, performed larger saccades and shorter fixations compared to dyslexics. While reading speed peaked around the default letter spacing, saccade amplitude increased and fixation duration decreased with the increase of letter spacing in both groups. Lateralization of occipito-temporal fixation-related EEG activity (FREA) was found in three consecutive time intervals corresponding to early orthographic processing in control readers. Importantly, the lateralization in the time range of the first negative left occipito-temporal FREA peak was specific for first fixations and exhibited an interaction effect between reading ability and letter spacing. The interaction originated in the significant decrease of FREA lateralization at extra-large compared to default letter spacing in control readers and the lack of lateralization in both letter spacing conditions in the case of dyslexics. These findings suggest that expertise-driven hemispheric functional specialization for early orthographic processing thought to be responsible for letter identity extraction during natural reading is compromised in dyslexia.

Neural basis of distractor resistance during visual working memory maintenance

Visual working memory representations must be protected from the intervening irrelevant visual input. While it is well known that interference resistance is most challenging when distractors match the prioritised mnemonic information, its neural mechanisms remain poorly understood. Here, we identify two top-down attentional control processes that have opposing effects on distractor resistance. We reveal an early selection negativity in the EEG responses to matching as compared to non-matching distractors, the magnitude of which is negatively associated with behavioural distractor resistance. Additionally, matching distractors lead to reduced post-stimulus alpha power as well as increased fMRI responses in the object-selective visual cortical areas and the inferior frontal gyrus. However, the congruency effect found on the post-stimulus periodic alpha power and the inferior frontal gyrus fMRI responses show a positive association with distractor resistance. These findings suggest that distractor interference is enhanced by proactive memory content-guided selection processes and diminished by reactive allocation of top-down attentional resources to protect memorandum representations within visual cortical areas retaining the most selective mnemonic code.

Investigation of the relationship between visual feature binding in short- and long-term memory in healthy aging

Binding visual features into coherent object representations is essential both in short- and long-term memory. However, the relationship between feature binding processes at different memory delays remains unexplored. Here, we addressed this question by using the Mnemonic Similarity Task and a delayed-estimation working memory task on a large sample of older adults. The results revealed that higher propensity to misbind object features in working memory is associated with lower lure discrimination performance in the mnemonic similarity task, suggesting that shared feature binding processes underlie the formation of coherent short- and long-term visual object memory representations.

An empirical investigation of the benefit of increasing the temporal resolution of task-evoked fMRI data with multi-band imaging

Objective

There is a tendency for reducing TR in MRI experiments with multi-band imaging. We empirically investigate its benefit for the group-level statistical outcome in task-evoked fMRI.

Methods

Three visual fMRI data sets were collected from 17 healthy adult participants. Multi-band acquisition helped vary the TR (2000/1000/410 ms, respectively). Because these data sets capture different temporal aspects of the haemodynamic response (HRF), we tested several HRF models. We computed a composite descriptive statistic, H, from β’s of each first-level model fit and carried it to the group-level analysis. The number of activated voxels and the t value of the group-level analysis as well as a goodness-of-fit measure were used as surrogate markers of data quality for comparison.

Results

Increasing the temporal sampling rate did not provide a universal improvement in the group-level statistical outcome. Rather, both the voxel-wise and ROI-averaged group-level results varied widely with anatomical location, choice of HRF and the setting of the TR. Correspondingly, the goodness-of-fit of HRFs became worse with increasing the sampling frequency.

Conclusion

Rather than universally increasing the temporal sampling rate in cognitive fMRI experiments, these results advocate the performance of a pilot study for the specific ROIs of interest to identify the appropriate temporal sampling rate for the acquisition and the correspondingly suitable HRF for the analysis of the data.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10334-021-00918-z.

Dichoptic Spatial Contrast Sensitivity Reflects Binocular Balance in Normal and Stereoanomalous Subjects

Purpose

To study binocular balance by comparing dichoptic and standard monocular contrast sensitivity function (CSF) in stereonormal and stereoanomalous/stereoblind amblyopic subjects.

Methods

Sixteen amblyopes and 17 controls participated. Using the capability of the passive three-dimensional display, we measured their CSF both monocularly and dichoptically at spatial frequencies 0.5, 1, 2, 4, and 8 cpds using achromatic Gabor patches on a luminance noise background. During monocular stimulation, the untested eye was covered, while for the dichoptic stimulation the untested eye viewed background noise. Dichoptic CSF of both eyes was acquired within one block.

Results

In patients with central fixation, dichoptic viewing had a large negative impact on the CSF of the amblyopic eye, although it hardly affected that of the dominant eye. In contrast, dichoptic viewing had a small but significant effect on both eyes for controls. In addition, all participants lay along a continuum in terms of how much their two eyes were affected by dichoptic stimulation: by using two predefined contrast sensitivity ratios, namely, amblyopic sensitivity decrement and dichoptic sensitivity decrement, not only did we find a significant correlation between these variables among all participants, but also the two groups were identified with minimum error using a cluster analysis.

Conclusions

Dichoptic CSF may be considered to measure visual performance in patients with altered binocular vision, because it better reflects the visual capacity of the amblyopic eye than the standard monocular examinations. It may also be a more reliable parameter to assess the efficacy of modern approaches to treat amblyopia.

The influence of anticipated monetary incentives on visual working memory performance in healthy younger and older adults

Motivation exerts substantial control over cognitive functions, including working memory. Although it is well known that both motivational control and working memory processes undergo a progressive decline with ageing, whether and to what extent their interaction is altered in old age remain unexplored. Here we aimed at uncovering the effect of reward anticipation on visual working memory performance in a large cohort of younger and older adults using a delayed-estimation task. We applied a three-component probabilistic model to dissociate the reward effects on three possible sources of error corrupting working memory performance: variability in recall, misbinding of object features and random guessing. The results showed that monetary incentives have a significant beneficial effect on overall working memory recall precision only in the group of younger adults. However, our model-based analysis resulted in significant reward effects on all three working memory component processes, which did not differ between the age groups, suggesting that model-based analysis is more sensitive to small reward-induced modulations in the case of older participants. These findings revealed that monetary incentives have a global boosting effect on working memory performance, which is deteriorated to some extent but still present in healthy older adults.

Expectations about word stress modulate neural activity in speech-sensitive cortical areas

A recent dual-stream model of language processing proposed that the postero-dorsal stream performs predictive sequential processing of linguistic information via hierarchically organized internal models. However, it remains unexplored whether the prosodic segmentation of linguistic information involves predictive processes. Here, we addressed this question by investigating the processing of word stress, a major component of speech segmentation, using probabilistic repetition suppression (RS) modulation as a marker of predictive processing. In an event-related acoustic fMRI RS paradigm, we presented pairs of pseudowords having the same (Rep) or different (Alt) stress patterns, in blocks with varying Rep and Alt trial probabilities. We found that the BOLD signal was significantly lower for Rep than for Alt trials, indicating RS in the posterior and middle superior temporal gyrus (STG) bilaterally, and in the anterior STG in the left hemisphere. Importantly, the magnitude of RS was modulated by repetition probability in the posterior and middle STG. These results reveal the predictive processing of word stress in the STG areas and raise the possibility that words stress processing is related to the dorsal "where" auditory stream.

Predicting Body Mass Index From Structural MRI Brain Images Using a Deep Convolutional Neural Network

In recent years, deep learning (DL) has become more widespread in the fields of cognitive and clinical neuroimaging. Using deep neural network models to process neuroimaging data is an efficient method to classify brain disorders and identify individuals who are at increased risk of age-related cognitive decline and neurodegenerative disease. Here we investigated, for the first time, whether structural brain imaging and DL can be used for predicting a physical trait that is of significant clinical relevance—the body mass index (BMI) of the individual. We show that individual BMI can be accurately predicted using a deep convolutional neural network (CNN) and a single structural magnetic resonance imaging (MRI) brain scan along with information about age and sex. Localization maps computed for the CNN highlighted several brain structures that strongly contributed to BMI prediction, including the caudate nucleus and the amygdala. Comparison to the results obtained via a standard automatic brain segmentation method revealed that the CNN-based visualization approach yielded complementary evidence regarding the relationship between brain structure and BMI. Taken together, our results imply that predicting BMI from structural brain scans using DL represents a promising approach to investigate the relationship between brain morphological variability and individual differences in body weight and provide a new scope for future investigations regarding the potential clinical utility of brain-predicted BMI.

Efficacy of weight loss intervention can be predicted based on early alterations of fMRI food cue reactivity in the striatum

Increased fMRI food cue reactivity in obesity, i.e. higher responses to high- vs. low-calorie food images, is a promising marker of the dysregulated brain reward system underlying enhanced susceptibility to obesogenic environmental cues. Recently, it has also been shown that weight loss interventions might affect fMRI food cue reactivity and that there is a close association between the alteration of cue reactivity and the outcome of the intervention. Here we tested whether fMRI food cue reactivity could be used as a marker of diet-induced early changes of neural processing in the striatum that are predictive of the outcome of the weight loss intervention. To this end we investigated the relationship between food cue reactivity in the striatum measured one month after the onset of the weight loss program and weight changes obtained at the end of the six-month intervention. We observed a significant correlation between BMI change measured after six months and early alterations of fMRI food cue reactivity in the striatum, including the bilateral putamen, right pallidum, and left caudate. Our findings provide evidence for diet-induced early alterations of fMRI food cue reactivity in the striatum that can predict the outcome of the weight loss intervention.

Transfer learning improves resting-state functional connectivity pattern analysis using convolutional neural networks

Background

Deep learning is gaining importance in the prediction of cognitive states and brain pathology based on neuroimaging data. Including multiple hidden layers in artificial neural networks enables unprecedented predictive power; however, the proper training of deep neural networks requires thousands of exemplars. Collecting this amount of data is not feasible in typical neuroimaging experiments. A handy solution to this problem, which has largely fallen outside the scope of deep learning applications in neuroimaging, is to repurpose deep networks that have already been trained on large datasets by fine-tuning them to target datasets/tasks with fewer exemplars. Here, we investigated how this method, called transfer learning, can aid age category classification and regression based on brain functional connectivity patterns derived from resting-state functional magnetic resonance imaging. We trained a connectome-convolutional neural network on a larger public dataset and then examined how the knowledge learned can be used effectively to perform these tasks on smaller target datasets collected with a different type of scanner and/or imaging protocol and pre-processing pipeline.

Results

Age classification on the target datasets benefitted from transfer learning. Significant improvement (∼9%–13% increase in accuracy) was observed when the convolutional layers’ weights were initialized based on the values learned on the public dataset and then fine-tuned to the target datasets. Transfer learning also appeared promising in improving the otherwise poor prediction of chronological age.

Conclusions

Transfer learning is a plausible solution to adapt convolutional neural networks to neuroimaging data with few exemplars and different data acquisition and pre-processing protocols.

Correction to: Reducing task-based fMRI scanning time using simultaneous multislice echo planar imaging

The original version of this article contained a mistake. The correct Affiliation 2 is Semmelweis University, János Szentágothai PhD School, MR Research Centre, Balassa Street 6, Budapest 1083, Hungary.

Reducing task-based fMRI scanning time using simultaneous multislice echo planar imaging

PURPOSE

To maintain alertness and to remain motionless during scanning represent a substantial challenge for patients/subjects involved in both clinical and research functional magnetic resonance imaging (fMRI) examinations. Therefore, availability and application of new data acquisition protocols allowing the shortening of scan time without compromising the data quality and statistical power are of major importance.

METHODS

Higher order category-selective visual cortical areas were identified individually, and rapid event-related fMRI design was used to compare three different sampling rates (TR = 2000, 1000, and 410 ms, using state-of-the-art simultaneous multislice imaging) and four different scanning lengths to match the statistical power of the traditional scanning methods to high sampling-rate design.

RESULTS

The results revealed that ~ 4 min of the scan time with 1 Hz (TR = 1000 ms) sampling rate and ~ 2 min scanning at ~ 2.5 Hz (TR = 410 ms) sampling rate provide similar localization sensitivity and selectivity to that obtained with 11-min session at conventional, 0.5 Hz (TR = 2000 ms) sampling rate.

CONCLUSION

Our findings suggest that task-based fMRI examination of clinical population prone to distress such as presurgical mapping experiments might substantially benefit from the reduced (20-40%) scanning time that can be achieved by the application of simultaneous multislice sequences.

Resting State fMRI Functional Connectivity-Based Classification Using a Convolutional Neural Network Architecture

Machine learning techniques have become increasingly popular in the field of resting state fMRI (functional magnetic resonance imaging) network based classification. However, the application of convolutional networks has been proposed only very recently and has remained largely unexplored. In this paper we describe a convolutional neural network architecture for functional connectome classification called connectome-convolutional neural network (CCNN). Our results on simulated datasets and a publicly available dataset for amnestic mild cognitive impairment classification demonstrate that our CCNN model can efficiently distinguish between subject groups. We also show that the connectome-convolutional network is capable to combine information from diverse functional connectivity metrics and that models using a combination of different connectivity descriptors are able to outperform classifiers using only one metric. From this flexibility follows that our proposed CCNN model can be easily adapted to a wide range of connectome based classification or regression tasks, by varying which connectivity descriptor combinations are used to train the network.

Resting State fMRI Functional Connectivity Analysis Using Dynamic Time Warping

Traditional resting-state network concept is based on calculating linear dependence of spontaneous low frequency fluctuations of the BOLD signals of different brain areas, which assumes temporally stable zero-lag synchrony across regions. However, growing amount of experimental findings suggest that functional connectivity exhibits dynamic changes and a complex time-lag structure, which cannot be captured by the static zero-lag correlation analysis. Here we propose a new approach applying Dynamic Time Warping (DTW) distance to evaluate functional connectivity strength that accounts for non-stationarity and phase-lags between the observed signals. Using simulated fMRI data we found that DTW captures dynamic interactions and it is less sensitive to linearly combined global noise in the data as compared to traditional correlation analysis. We tested our method using resting-state fMRI data from repeated measurements of an individual subject and showed that DTW analysis results in more stable connectivity patterns by reducing the within-subject variability and increasing robustness for preprocessing strategies. Classification results on a public dataset revealed a superior sensitivity of the DTW analysis to group differences by showing that DTW based classifiers outperform the zero-lag correlation and maximal lag cross-correlation based classifiers significantly. Our findings suggest that analysing resting-state functional connectivity using DTW provides an efficient new way for characterizing functional networks.

Investigating the Group-Level Impact of Advanced Dual-Echo fMRI Combinations

Multi-echo fMRI data acquisition has been widely investigated and suggested to optimize sensitivity for detecting the BOLD signal. Several methods have also been proposed for the combination of data with different echo times. The aim of the present study was to investigate whether these advanced echo combination methods provide advantages over the simple averaging of echoes when state-of-the-art group-level random-effect analyses are performed. Both resting-state and task-based dual-echo fMRI data were collected from 27 healthy adult individuals (14 male, mean age = 25.75 years) using standard echo-planar acquisition methods at 3T. Both resting-state and task-based data were subjected to a standard image pre-processing pipeline. Subsequently the two echoes were combined as a weighted average, using four different strategies for calculating the weights: (1) simple arithmetic averaging, (2) BOLD sensitivity weighting, (3) temporal-signal-to-noise ratio weighting and (4) temporal BOLD sensitivity weighting. Our results clearly show that the simple averaging of data with the different echoes is sufficient. Advanced echo combination methods may provide advantages on a single-subject level but when considering random-effects group level statistics they provide no benefit regarding sensitivity (i.e., group-level t-values) compared to the simple echo-averaging approach. One possible reason for the lack of clear advantages may be that apart from increasing the average BOLD sensitivity at the single-subject level, the advanced weighted averaging methods also inflate the inter-subject variance. As the echo combination methods provide very similar results, the recommendation is to choose between them depending on the availability of time for collecting additional resting-state data or whether subject-level or group-level analyses are planned.

Visual processing during natural reading

Reading is a unique human ability that plays a pivotal role in the development and functioning of our modern society. However, its neural basis remains poorly understood since previous research was focused on reading words with fixed gaze. Here we developed a methodological framework for single-trial analysis of fixation onset-related EEG activity (FOREA) that enabled us to investigate visual information processing during natural reading. To reveal the effect of reading skills on orthographic processing during natural reading, we measured how altering the configural properties of the written text by modifying inter-letter spacing affects FOREA. We found that orthographic processing is reflected in FOREA in three consecutive time windows (120–175 ms, 230–265 ms, 345–380 ms after fixation onset) and the magnitude of FOREA effects in the two later time intervals showed a close association with the participants’ reading speed: FOREA effects were larger in fast than in slow readers. Furthermore, these expertise-driven configural effects were clearly dissociable from the FOREA signatures of visual perceptual processes engaged to handle the increased crowding (155–220 ms) as a result of decreasing letter spacing. Our findings revealed that with increased reading skills orthographic processing becomes more sensitive to the configural properties of the written text.

Object-based attentional selection modulates anticipatory alpha oscillations

Visual cortical alpha oscillations are involved in attentional gating of incoming visual information. It has been shown that spatial and feature-based attentional selection result in increased alpha oscillations over the cortical regions representing sensory input originating from the unattended visual field and task-irrelevant visual features, respectively. However, whether attentional gating in the case of object based selection is also associated with alpha oscillations has not been investigated before. Here we measured anticipatory electroencephalography (EEG) alpha oscillations while participants were cued to attend to foveal face or word stimuli, the processing of which is known to have right and left hemispheric lateralization, respectively. The results revealed that in the case of simultaneously displayed, overlapping face and word stimuli, attending to the words led to increased power of parieto-occipital alpha oscillations over the right hemisphere as compared to when faces were attended. This object category-specific modulation of the hemispheric lateralization of anticipatory alpha oscillations was maintained during sustained attentional selection of sequentially presented face and word stimuli. These results imply that in the case of object-based attentional selection—similarly to spatial and feature-based attention—gating of visual information processing might involve visual cortical alpha oscillations.

Amblyopic deficit beyond the fovea: delayed and variable single-trial ERP response latencies, but unaltered amplitudes

PURPOSE

Amblyopia was first described as a deficit of central vision. However, it has long been debated whether this dysfunction is limited to the fovea or whether extrafoveal vision is also affected, as studies concerning the latter are equivocal. The purpose of the study was to resolve this issue.

METHODS

We investigated the amblyopic effect on event-related potentials (ERPs) with foveal and perifoveal stimuli, either matched in size based on cortical magnification or presented as large annular stimuli. In two separate experiments we measured ERPs on amblyopic patients and control subjects using face images. Latency and amplitude of averaged ERPs and their single-trial distributions were analyzed.

RESULTS

When the fovea was stimulated, latency and amplitude of the early averaged ERP components increased and were reduced, respectively, in the amblyopic compared with the fellow eye. Importantly, perifoveal stimulation also elicited similar amblyopic deficits, which were clearly significant in the case of using cortical magnification scaled stimuli. However, single-trial peak analysis revealed that foveal and perifoveal effects differed in nature: Peak amplitudes were reduced only in foveal stimulation, while latencies were delayed and jittered at both the fovea and perifovea. Event-related potentials obtained from fellow eyes were not significantly different from those of normal observers.

CONCLUSIONS

Our findings revealed the existence of amblyopic deficits at the perifovea when the stimulated cortical area was matched in size to that of foveal stimulation. These deficits manifested themselves only in the temporal structure of the responses, unlike foveal deficits, which affected both component amplitude and latency.

How the visual cortex handles stimulus noise: insights from amblyopia

Adding noise to a visual image makes object recognition more effortful and has a widespread effect on human electrophysiological responses. However, visual cortical processes directly involved in handling the stimulus noise have yet to be identified and dissociated from the modulation of the neural responses due to the deteriorated structural information and increased stimulus uncertainty in the case of noisy images. Here we show that the impairment of face gender categorization performance in the case of noisy images in amblyopic patients correlates with amblyopic deficits measured in the noise-induced modulation of the P1/P2 components of single-trial event-related potentials (ERP). On the other hand, the N170 ERP component is similarly affected by the presence of noise in the two eyes and its modulation does not predict the behavioral deficit. These results have revealed that the efficient processing of noisy images depends on the engagement of additional processing resources both at the early, feature-specific as well as later, object-level stages of visual cortical processing reflected in the P1 and P2 ERP components, respectively. Our findings also suggest that noise-induced modulation of the N170 component might reflect diminished face-selective neuronal responses to face images with deteriorated structural information.

FMRI repetition suppression for voices is modulated by stimulus expectations

According to predictive coding models of sensory processing, stimulus expectations have a profound effect on sensory cortical responses. This was supported by experimental results, showing that fMRI repetition suppression (fMRI RS) for face stimuli is strongly modulated by the probability of stimulus repetitions throughout the visual cortical processing hierarchy. To test whether processing of voices is also affected by stimulus expectations, here we investigated the effect of repetition probability on fMRI RS in voice-selective cortical areas. Changing ('alt') and identical ('rep') voice stimulus pairs were presented to the listeners in blocks, with a varying probability of alt and rep trials across blocks. We found auditory fMRI RS in the nonprimary voice-selective cortical regions, including the bilateral posterior STS, the right anterior STG and the right IFC, as well as in the IPL. Importantly, fMRI RS effects in all of these areas were strongly modulated by the probability of stimulus repetition: auditory fMRI RS was reduced or not present in blocks with low repetition probability. Our results revealed that auditory fMRI RS in higher-level voice-selective cortical regions is modulated by repetition probabilities and thus suggest that in audition, similarly to the visual modality, processing of sensory information is shaped by stimulus expectation processes.

Visual cortical responses to the input from the amblyopic eye are suppressed during binocular viewing

Amblyopia is a visual disorder caused by an anomalous early visual experience. It has been suggested that suppression of the visual input from the weaker eye might be a primary underlying mechanism of the amblyopic syndrome. However, it is still an unresolved question to what extent neural responses to the visual information coming from the amblyopic eye are suppressed during binocular viewing. To address this question we measured event-related potentials (ERP) to foveal face stimuli in amblyopic patients, both in monocular and binocular viewing conditions. The results revealed no difference in the amplitude and latency of early components of the ERP responses between the binocular and fellow eye stimulation. On the other hand, early ERP components were reduced and delayed in the case of monocular stimulation of the amblyopic eye as compared to the fellow eye stimulation or to binocular viewing. The magnitude of the amblyopic effect measured on the ERP amplitudes was comparable to that found on the fMRI responses in the fusiform face area using the same face stimuli and task conditions. Our findings showing that the amblyopic effects present on the early ERP components in the case of monocular stimulation are not manifested in the ERP responses during binocular viewing suggest that input from the amblyopic eye is completely suppressed already at the earliest stages of visual cortical processing when stimuli are viewed by both eyes.

Amblyopic deficits in the timing and strength of visual cortical responses to faces

Behavioral research revealed that object vision is impaired in amblyopia. Nevertheless, neurophysiological research in humans has focused on the amblyopic effects at the earliest stage of visual cortical processing, leaving the question of later, object-specific neural processing deficits unexplored. By measuring event-related potentials (ERPs) to foveal face stimuli we characterized the amblyopic effects on the N170 component, reflecting higher-level structural face processing. Single trial analysis revealed that latencies of the ERP components increased and were more variable in the amblyopic eye compared to the fellow eye both in strabismic and anisometropic patent groups. Moreover, there was an additional delay of N170 relative to the early P1 component over the right hemisphere, which was absent in the fellow eye, suggesting a slower evolution of face specific cortical responses in amblyopia. On the other hand, distribution of single trial N170 peak amplitudes differed between the amblyopic and fellow eye only in the strabismic but not in the anisometropic patients. Furthermore, the amblyopic N170 latency increment but not the amplitude reduction correlated with the interocular differences in visual acuity and fixation stability. We found no difference in the anticipatory neural oscillations between stimulation of the amblyopic and the fellow eye implying that impairment of the neural processes underlying generation of stimulus-driven visual cortical responses might be the primary reason behind the observed amblyopic effects. These findings provide evidence that amblyopic disruption of early visual experience leads to deficits in the strength and timing of higher-level, face specific visual cortical responses, reflected in the N170 component.

Neural mechanisms for voice recognition

We investigated neural mechanisms that support voice recognition in a training paradigm with fMRI. The same listeners were trained on different weeks to categorize the mid-regions of voice-morph continua as an individual's voice. Stimuli implicitly defined a voice-acoustics space, and training explicitly defined a voice-identity space. The pre-defined centre of the voice category was shifted from the acoustic centre each week in opposite directions, so the same stimuli had different training histories on different tests. Cortical sensitivity to voice similarity appeared over different time-scales and at different representational stages. First, there were short-term adaptation effects: increasing acoustic similarity to the directly preceding stimulus led to haemodynamic response reduction in the middle/posterior STS and in right ventrolateral prefrontal regions. Second, there were longer-term effects: response reduction was found in the orbital/insular cortex for stimuli that were most versus least similar to the acoustic mean of all preceding stimuli, and, in the anterior temporal pole, the deep posterior STS and the amygdala, for stimuli that were most versus least similar to the trained voice-identity category mean. These findings are interpreted as effects of neural sharpening of long-term stored typical acoustic and category-internal values. The analyses also reveal anatomically separable voice representations: one in a voice-acoustics space and one in a voice-identity space. Voice-identity representations flexibly followed the trained identity shift, and listeners with a greater identity effect were more accurate at recognizing familiar voices. Voice recognition is thus supported by neural voice spaces that are organized around flexible 'mean voice' representations.

Retention Interval Affects Visual Short-Term Memory Encoding

Humans can efficiently store fine-detailed facial emotional information in visual short-term memory for several seconds. However, an unresolved question is whether the same neural mechanisms underlie high-fidelity short-term memory for emotional expressions at different retention intervals. Here we show that retention interval affects the neural processes of short-term memory encoding using a delayed facial emotion discrimination task. The early sensory P100 component of the event-related potentials (ERP) was larger in the 1-s interstimulus interval (ISI) condition than in the 6-s ISI condition, whereas the face-specific N170 component was larger in the longer ISI condition. Furthermore, the memory-related late P3b component of the ERP responses was also modulated by retention interval: it was reduced in the 1-s ISI as compared with the 6-s condition. The present findings cannot be explained based on differences in sensory processing demands or overall task difficulty because there was no difference in the stimulus information and subjects' performance between the two different ISI conditions. These results reveal that encoding processes underlying high-precision short-term memory for facial emotional expressions are modulated depending on whether information has to be stored for one or for several seconds.

Electrophysiological correlates of learning-induced modulation of visual motion processing in humans

Training on a visual task leads to increased perceptual and neural responses to visual features that were attended during training as well as decreased responses to neglected distractor features. However, the time course of these attention-based modulations of neural sensitivity for visual features has not been investigated before. Here we measured event related potentials (ERP) in response to motion stimuli with different coherence levels before and after training on a speed discrimination task requiring object-based attentional selection of one of the two competing motion stimuli. We found that two peaks on the ERP waveform were modulated by the strength of the coherent motion signal; the response amplitude associated with motion directions that were neglected during training was smaller than the response amplitude associated with motion directions that were attended during training. The first peak of motion coherence-dependent modulation of the ERP responses was at 300 ms after stimulus onset and it was most pronounced over the occipitotemporal cortex. The second peak was around 500 ms and was focused over the parietal cortex. A control experiment suggests that the earlier motion coherence-related response modulation reflects the extraction of the coherent motion signal whereas the later peak might index accumulation and readout of motion signals by parietal decision mechanisms. These findings suggest that attention-based learning affects neural responses both at the sensory and decision processing stages.

Attention-based perceptual learning increases binocular rivalry suppression of irrelevant visual features

Perceptual learning refers to an improvement on a perceptual task after repeated exposure to a stimulus. It has been shown that attention can play an important role in perceptual learning. Recently, it has been suggested that training can lead to increased suppression of information that is continuously irrelevant, and that this attention-based suppression plays an important role in more efficient noise exclusion. Here we investigate this claim. Observers performed a visual speed-discrimination task for 5 consecutive days. After training, sensitivity to motion directions that were relevant, irrelevant, or neutral toward the training task was assessed by measuring motion coherence thresholds. In addition, perceptual dominance during binocular rivalry was assessed for combinations of the three motion directions. The results showed that sensitivity to the task-relevant feature increased due to training. That is, motion coherence thresholds were selectively lowered for the task-relevant feature. Interestingly, the feature that was task-irrelevant during training was more strongly suppressed during binocular rivalry: The mean perceptual dominance of this feature was selectively decreased. Our results show that task-irrelevant information that potentially interferes with the primary task during learning gets more strongly suppressed. Furthermore, our results add new evidence in support of the claim that mechanisms involved in visual attention and binocular rivalry overlap.

Adaptation duration affects the spatial selectivity of facial aftereffects

Adaptation processes in human early visual cortical areas are sensitive to the exposure time of the adaptor stimulus. Here we investigated the effect of adaptation duration at the higher, shape-specific stages of visual processing using facial adaptation. It was found that long-term (5s) adaptation evokes facial aftereffects consisting of a position invariant as well as a position-specific component. As a result of adaptation to a female face, test faces were judged more masculine when they were displayed in the same location as the female adaptor face, as compared to that when they were presented in the opposite visual hemifield. However, aftereffects evoked by short-term (500 ms) adaptation were found to be entirely position invariant. In accordance with these behavioral results, we found that the adaptation effects, measured on the amplitude of the N170 ERP component consisted of a position-specific component only after long-term, but not after short-term adaptation conditions. These results suggest that both short and long exposure to a face stimulus leads to adaptation of position invariant face-selective processes, whereas adaptation of position-specific neural mechanisms of face processing requires long-term adaptation. Our findings imply that manipulating adaptation duration provides an opportunity to specifically adapt different neural processes of shape-specific coding and to investigate their stimulus selectivity.

How well do patients with schizophrenia track multiple moving targets?

It has been demonstrated that patients with schizophrenia perform poorly on tasks that require orienting, focusing, maintaining, and shifting attention. However, it is unknown how patients with schizophrenia can track multiple moving targets. To elucidate this issue, the authors investigated fast and slow multiple-object tracking in patients with schizophrenia (n = 30) and in matched healthy control participants (n = 30) and assessed their relationship with motion perception (velocity discrimination), sustained attention and context processing (Continuous Performance Test, 1-9 version; J. R. Finkelstein, T. D. Cannon, R. E. Gur, R. C. Gur, & P. Moberg, 1997), and object and spatial working memory. Results revealed that patients with schizophrenia displayed impaired performances on multiple-object tracking tasks. Linear regression analysis revealed a specific relationship among object tracking, velocity discrimination, and spatial working memory. In patients with schizophrenia, velocity discrimination and spatial working memory were the predictive factors of multiple-object tracking, whereas in healthy control participants, the single predictive factor was velocity discrimination. Probabilistic regression analysis revealed that only the Continuous Performance Test made significant contribution to discriminating between patients and control participants. These results suggest that multiple-object tracking is impaired in schizophrenia, and that it is specifically associated with motion perception and spatial working memory.

Subthreshold features of visual objects: Unseen but not unbound

The object is a basic unit that is thought to organize the way in which we perceive and think about the world. According to theories of object-based attention, perception of unified objects depends on the binding together of the disparate features of each object via attention. Here we show that a visual feature that is not consciously perceived is nonetheless modulated by object-based attention: the influence of a subthreshold motion signal (prime) on subsequent motion perception depended critically on whether it was associated with the attended object or another, spatially overlapping object. These results show that invisibly weak features of attended objects are not lost, but are organized by and selected together with the object by attention.

Back-to-front: Improved tactile discrimination performance in the space you cannot see

We investigated any differences in people's ability to reconstruct the appropriate spatiotemporal ordering of multiple tactile stimuli, when presented in frontal space (a region where visual inputs tend to dominate) versus in the space behind the back (a region of space that we rarely see) in professional piano players and in non-musicians. Even though tactile temporal order judgments were much better in the musicians overall, both groups showed a much reduced crossed-hands deficit when their hands were crossed behind their backs rather than at the front. These results suggest that because of differences in the availability of visual input, the spatiotemporal representation of non-visual stimuli in front versus rear space is different.

Position-specificity of facial adaptation

We investigated the representation of objects' position at the higher, shape-selective stages of visual processing by testing the position-specificity of the behavioural and neural effects of facial adaptation. Here, we show that facial after-effects evoked by adaptation to both upright and upside-down faces are significantly larger when the adaptor and test faces are presented on the same retinal position than when they are displayed in different hemifields. Our event-related potential recordings revealed that adaptation effects measured on the amplitude of the N170 event-related potential component over the hemisphere that was contralateral to the test face stimulus also show strong position-specificity. These findings suggest that face adaptation effects are only partially translation invariant and facial after-effects measured with peripheral test stimuli primarily reflect the adaptation processes in the contralateral hemisphere.

Cross-feature spread of global attentional modulation in human area MT+

Feature-based attention affects the processing of the selected feature throughout the visual field. Here, we show that such global attentional modulation is not restricted to the attended feature but spreads to task-irrelevant features that are bound to the attended one. Attention to a color in one of the visual hemifields affected the processing of task-irrelevant motion in the other hemifield when it was associated with a stimulus that shared the attended color. This cross-feature global attentional selection increased the duration of the motion aftereffect and the strength of functional magnetic resonance imaging responses in the motion-sensitive area MT+, evoked by the task-irrelevant motion. These findings imply that features belonging to the same object are bound and selected jointly even outside the focus of attention.

Electrophysiological Correlates of Visual Adaptation to Faces and Body Parts in Humans

The existence of facial aftereffects suggests that shape-selective mechanisms at the higher stages of visual object coding -- similarly to the early processing of low-level visual features -- are adaptively recalibrated. Our goal was to uncover the ERP correlates of shape-selective adaptation and to test whether it is also involved in the visual processing of human body parts. We found that prolonged adaptation to female hands -- similarly to adaptation to female faces -- biased the judgements about the subsequently presented hand test stimuli: they were perceived more masculine than in the control conditions. We also showed that these hand aftereffects are size and orientation invariant. However, no aftereffects were found when the adaptor and test stimuli belonged to different categories (i.e. face adaptor and hand test, or vice versa), suggesting that the underlying adaptation mechanisms are category-specific. In accordance with the behavioral results, both adaptation to faces and hands resulted in a strong and category-specific modulation -- reduced amplitude and increased latency -- of the N170 component of ERP responses. Our findings suggest that shape-selective adaptation is a general mechanism of visual object processing and its neural effects are primarily reflected in the N170 component of the ERP responses.

Implicit Attentional Selection of Bound Visual Features

Traditionally, research on visual attention has been focused on the processes involved in conscious, explicit selection of task-relevant sensory input. Recently, however, it has been shown that attending to a specific feature of an object automatically increases neural sensitivity to this feature throughout the visual field. Here we show that directing attention to a specific color of an object results in attentional modulation of the processing of task-irrelevant and not consciously perceived motion signals that are spatiotemporally associated with this color throughout the visual field. Such implicit cross-feature spreading of attention takes place according to the veridical physical associations between the color and motion signals, even under special circumstances when they are perceptually misbound. These results imply that the units of implicit attentional selection are spatiotemporally colocalized feature clusters that are automatically bound throughout the visual field.

Binding of motion and colour is early and automatic

At what stages of the human visual hierarchy different features are bound together, and whether this binding requires attention, is still highly debated. We used a colour-contingent motion after-effect (CCMAE) to study the binding of colour and motion signals. The logic of our approach was as follows: if CCMAEs can be evoked by targeted adaptation of early motion processing stages, without allowing for feedback from higher motion integration stages, then this would support our hypothesis that colour and motion are bound automatically on the basis of spatiotemporally local information. Our results show for the first time that CCMAE's can be evoked by adaptation to a locally paired opposite-motion dot display, a stimulus that, importantly, is known to trigger direction-specific responses in the primary visual cortex yet results in strong inhibition of the directional responses in area MT of macaques as well as in area MT+ in humans and, indeed, is perceived only as motionless flicker. The magnitude of the CCMAE in the locally paired condition was not significantly different from control conditions where the different directions were spatiotemporally separated (i.e. not locally paired) and therefore perceived as two moving fields. These findings provide evidence that adaptation at an early, local motion stage, and only adaptation at this stage, underlies this CCMAE, which in turn implies that spatiotemporally coincident colour and motion signals are bound automatically, most probably as early as cortical area V1, even when the association between colour and motion is perceptually inaccessible.

Learning to suppress task-irrelevant visual stimuli with attention

While the importance of attention in perceptual learning is widely recognized, the mechanisms through which it affects learning are poorly understood. Here we show that attentional mechanisms themselves are modified during learning. Attentional suppression of task-irrelevant stimuli becomes more efficient with practice. Attentional learning was found to be stimulus-specific and to persist for several weeks, suggesting that the plasticity of attentional mechanisms is an inherent component of visual perceptual learning.