Test–retest reliability in fMRI: Or how I learned to stop worrying and love the variability

A systematic review of neuroimaging approaches to mapping language in individuals

Although researchers often rely on group-level fMRI results to draw conclusions about the neurobiology of language, doing so without accounting for the complexities of individual brains may reduce the validity of our findings. Furthermore, understanding brain organization in individuals is critically important for both basic science and clinical translation. To assess the state of single-subject language localization in the functional neuroimaging literature, we carried out a systematic review of studies published through April 2020. Out of 977 papers identified through our search, 121 met our inclusion criteria for reporting single-subject fMRI results (fMRI studies of language in adults that report task-based single-subject statistics). Of these, 20 papers reported using a single-subject test-retest analysis to assess reliability. Thus, we found that a relatively modest number of papers reporting single-subject results quantified single-subject reliability. These varied substantially in acquisition parameters, task design, and reliability measures, creating significant challenges for making comparisons across studies. Future endeavors to optimize the localization of language networks in individuals will benefit from the standardization and broader reporting of reliability metrics for different tasks and acquisition parameters.

Planning ahead: Predictable switching recruits task-active and resting-state networks

Switching is a difficult cognitive process characterised by costs in task performance; specifically, slowed responses and reduced accuracy. It is associated with the recruitment of a large coalition of task-positive regions including those referred to as the multiple demand cortex (MDC). The neural correlates of switching not only include the MDC, but occasionally the default mode network (DMN), a characteristically task-negative network. To unpick the role of the DMN during switching we collected fMRI data from 24 participants playing a switching paradigm that perturbed predictability (i.e., cognitive load) across three switch dimensions-sequential, perceptual, and spatial predictability. We computed the activity maps unique to switch vs. stay trials and all switch dimensions, then evaluated functional connectivity under these switch conditions by computing the pairwise mutual information functional connectivity (miFC) between regional timeseries. Switch trials exhibited an expected cost in reaction time while sequential predictability produced a significant benefit to task accuracy. Our results showed that switch trials recruited a broader activity map than stay trials, including regions of the DMN, the MDC, and task-positive networks such as visual, somatomotor, dorsal, salience/ventral attention networks. More sequentially predictable trials recruited increased activity in the somatomotor and salience/ventral attention networks. Notably, changes in sequential and perceptual predictability, but not spatial predictability, had significant effects on miFC. Increases in perceptual predictability related to decreased miFC between control, visual, somatomotor, and DMN regions, whereas increases in sequential predictability increased miFC between regions in the same networks, as well as regions within ventral attention/ salience, dorsal attention, limbic, and temporal parietal networks. These results provide novel clues as to how DMN may contribute to executive task performance. Specifically, the improved task performance, unique activity, and increased miFC associated with increased sequential predictability suggest that the DMN may coordinate more strongly with the MDC to generate a temporal schema of upcoming task events, which may attenuate switching costs.

The Atypical Effective Connectivity of Right Temporoparietal Junction in Autism Spectrum Disorder: A Multi-Site Study

Social function impairment is the core deficit of autism spectrum disorder (ASD). Although many studies have investigated ASD through a variety of neuroimaging tools, its brain mechanism of social function remains unclear due to its complex and heterogeneous symptoms. The present study aimed to use resting-state functional magnetic imaging data to explore effective connectivity between the right temporoparietal junction (RTPJ), one of the key brain regions associated with social impairment of individuals with ASD, and the whole brain to further deepen our understanding of the neuropathological mechanism of ASD. This study involved 1,454 participants from 23 sites from the Autism Brain Imaging Data Exchange (ABIDE) public dataset, which included 618 individuals with ASD and 836 with typical development (TD). First, a voxel-wise Granger causality analysis (GCA) was conducted with the RTPJ selected as the region of interest (ROI) to investigate the differences in effective connectivity between the ASD and TD groups in every site. Next, to obtain further accurate and representative results, an image-based meta-analysis was implemented to further analyze the GCA results of each site. Our results demonstrated abnormal causal connectivity between the RTPJ and the widely distributed brain regions and that the connectivity has been associated with social impairment in individuals with ASD. The current study could help to further elucidate the pathological mechanisms of ASD and provides a new perspective for future research.

Test-retest reliability of brain responses to risk-taking during the balloon analogue risk task

The Balloon Analogue Risk Task (BART) provides a reliable and ecologically valid model for the assessment of individual risk-taking propensity and is frequently used in neuroimaging and developmental research. Although the test-retest reliability of risk-taking behavior during the BART is well established, the reliability of brain activation patterns in response to risk-taking during the BART remains elusive. In this study, we used functional magnetic resonance imaging (fMRI) and evaluated the test-retest reliability of brain responses in 34 healthy adults during a modified BART by calculating the intraclass correlation coefficients (ICC) and Dice’s similarity coefficients (DSC). Analyses revealed that risk-induced brain activation patterns showed good test-retest reliability (median ICC = 0.62) and moderate to high spatial consistency, while brain activation patterns associated with win or loss outcomes only had poor to fair reliability (median ICC = 0.33 for win and 0.42 for loss). These findings have important implications for future utility of the BART in fMRI to examine brain responses to risk-taking and decision-making.

The individuality index: a measure to quantify the degree of inter-individual, spatial variability in intra-cerebral brain electric and metabolic activity

Contemporary neuroscience research primarily focuses on the identification of brain activation patterns commonly deviant across participant groups or experimental conditions. This approach inherently underestimates potentially meaningful intra- and inter-individual variability present in brain physiological measures. We propose a parameter referred to as 'individuality index (II)' that takes individual variability into account. It quantifies the degree of individual variance of brain activation patterns for different brain regions and participants. IIs can be computed based on intra-cerebral source strength values such as the ones derived from the exact low resolution electromagnetic tomography source localization software. We exemplary estimated IIs for simulated datasets. Our results illustrate how IIs are affected by different spatial activation patterns across participants and quantify their distributional properties. They suggest that the proposed indices can meaningfully quantify inter- and intra-individuality of brain activation patterns. Their application to realistic datasets will allow the identification of (1) those brain regions that show particularly heterogeneous activation patterns, the contribution of which is particularly likely to be underestimated by conventional group statistics, (2) those brain regions that can alternatively be recruited by different participants for the same tasks, and (3) their associations with potentially decisive behavioral variables such as individually applied mental strategy.

Assessing inter-individual differences with task-related functional neuroimaging

Explaining and predicting individual behavioural differences induced by clinical and social factors constitutes one of the most promising applications of neuroimaging. In this Perspective, we discuss the theoretical and statistical foundations of the analyses of inter-individual differences in task-related functional neuroimaging. Leveraging a five-year literature review (July 2013-2018), we show that researchers often assess how activations elicited by a variable of interest differ between individuals. We argue that the rationale for such analyses, typically grounded in resource theory, offers an over-large analytical and interpretational flexibility that undermines their validity. We also recall how, in the established framework of the general linear model, inter-individual differences in behaviour can act as hidden moderators and spuriously induce differences in activations. We conclude with a set of recommendations and directions, which we hope will contribute to improving the statistical validity and the neurobiological interpretability of inter-individual difference analyses in task-related functional neuroimaging.

Brain response to food brands correlates with increased intake from branded meals in children: an fMRI study

Food branding is ubiquitous, however, not all children are equally susceptible to its effects. The objectives of this study were to 1) determine whether food brands evoke differential response than non-food brands in brain areas related to motivation and inhibitory control using blood oxygen level dependent (BOLD) functional magnetic resonance imaging (fMRI) and 2) determine the association between brain response and energy intake at test-meals presented with or without brands. Twenty-eight 7-10 year-old children completed four visits as part of a within-subjects design where they consumed three multi-item test-meals presented with familiar food brands, novel food brand, and no brand. On the fourth visit an fMRI was performed where children passively viewed food brands, non-food brands and control images. A whole-brain analysis was conducted to compare BOLD response between conditions. Pearson's correlations were calculated to determine the association between brain response and meal intake. Relative to non-food brands, food brand images were associated with increased activity in the right lingual gyrus. Relative to control, food and non-food brand images were associated with greater response in bilateral fusiform gyri and decreased response in the cuneus, precuneus, lingual gyrus, and supramarginal gyrus. Less activation in the bilateral fusiform gyrus to both food and non-food brands was associated with greater energy intake of the branded vs unbranded meal. These findings may help explain differences in the susceptibility to the intake-promoting effects of food advertising in children.

Novel Imaging Biomarkers for Huntington's Disease and Other Hereditary Choreas

PURPOSE OF THE REVIEW

Imaging biomarkers for neurodegenerative disorders are primarily developed with the goal to aid diagnosis, to monitor disease progression, and to assess the efficacy of disease-modifying therapies in support to clinical outcomes that may either show limited sensitivity or need extended time for their evaluation. This article will review the most recent concepts and findings in the field of neuroimaging applied to Huntington's disease and Huntington-like syndromes. Emphasis will be given to the discussion of potential pharmacodynamic biomarkers for clinical trials in Huntington's disease (HD) and of neuroimaging tools that can be used as diagnostic biomarkers in HD-like syndromes.

RECENT FINDINGS

Several magnetic resonance (MR) and positron emission tomography (PET) molecular imaging tools have been identified as potential pharmacodynamic biomarkers and others are in the pipeline after preclinical validation. MRI and 18F-fluorodeoxyglucose PET can be considered useful supportive diagnostic tools for the differentiation of other HD-like syndromes. New trials in HD have the primary goal to lower mutant huntingtin (mHTT) protein levels in the brain in order to reduce or alter the progression of the disease. MR and PET molecular imaging markers have been developed as tools to monitor disease progression and to evaluate treatment outcomes of disease-modifying trials in HD. These markers could be used alone or in combination for detecting structural and pharmacodynamic changes potentially associated with the lowering of mHTT.

Thought experiment: Decoding cognitive processes from the fMRI data of one individual

Cognitive processes, such as the generation of language, can be mapped onto the brain using fMRI. These maps can in turn be used for decoding the respective processes from the brain activation patterns. Given individual variations in brain anatomy and organization, analyzes on the level of the single person are important to improve our understanding of how cognitive processes correspond to patterns of brain activity. They also allow to advance clinical applications of fMRI, because in the clinical setting making diagnoses for single cases is imperative. In the present study, we used mental imagery tasks to investigate language production, motor functions, visuo-spatial memory, face processing, and resting-state activity in a single person. Analysis methods were based on similarity metrics, including correlations between training and test data, as well as correlations with maps from the NeuroSynth meta-analysis. The goal was to make accurate predictions regarding the cognitive domain (e.g. language) and the specific content (e.g. animal names) of single 30-second blocks. Four teams used the dataset, each blinded regarding the true labels of the test data. Results showed that the similarity metrics allowed to reach the highest degrees of accuracy when predicting the cognitive domain of a block. Overall, 23 of the 25 test blocks could be correctly predicted by three of the four teams. Excluding the unspecific rest condition, up to 10 out of 20 blocks could be successfully decoded regarding their specific content. The study shows how the information contained in a single fMRI session and in each of its single blocks can allow to draw inferences about the cognitive processes an individual engaged in. Simple methods like correlations between blocks of fMRI data can serve as highly reliable approaches for cognitive decoding. We discuss the implications of our results in the context of clinical fMRI applications, with a focus on how decoding can support functional localization.

Statistical harmonization corrects site effects in functional connectivity measurements from multi-site fMRI data

Acquiring resting-state functional magnetic resonance imaging (fMRI) datasets at multiple MRI scanners and clinical sites can improve statistical power and generalizability of results. However, multi-site neuroimaging studies have reported considerable nonbiological variability in fMRI measurements due to different scanner manufacturers and acquisition protocols. These undesirable sources of variability may limit power to detect effects of interest and may even result in erroneous findings. Until now, there has not been an approach that removes unwanted site effects. In this study, using a relatively large multi-site (4 sites) fMRI dataset, we investigated the impact of site effects on functional connectivity and network measures estimated by widely used connectivity metrics and brain parcellations. The protocols and image acquisition of the dataset used in this study had been homogenized using identical MRI phantom acquisitions from each of the neuroimaging sites; however, intersite acquisition effects were not completely eliminated. Indeed, in this study, we found that the magnitude of site effects depended on the choice of connectivity metric and brain atlas. Therefore, to further remove site effects, we applied ComBat, a harmonization technique previously shown to eliminate site effects in multi-site diffusion tensor imaging (DTI) and cortical thickness studies. In the current work, ComBat successfully removed site effects identified in connectivity and network measures and increased the power to detect age associations when using optimal combinations of connectivity metrics and brain atlases. Our proposed ComBat harmonization approach for fMRI-derived connectivity measures facilitates reliable and efficient analysis of retrospective and prospective multi-site fMRI neuroimaging studies.

Test-retest reliability of prefrontal transcranial Direct Current Stimulation (tDCS) effects on functional MRI connectivity in healthy subjects

Transcranial Direct Current Stimulation (tDCS) of the prefrontal cortex (PFC) can be used for probing functional brain connectivity and meets general interest as novel therapeutic intervention in psychiatric and neurological disorders. Along with a more extensive use, it is important to understand the interplay between neural systems and stimulation protocols requiring basic methodological work. Here, we examined the test-retest (TRT) characteristics of tDCS-induced modulations in resting-state functional-connectivity MRI (RS fcMRI). Twenty healthy subjects received 20minutes of either active or sham tDCS of the dorsolateral PFC (2mA, anode over F3 and cathode over F4, international 10-20 system), preceded and ensued by a RS fcMRI (10minutes each). All subject underwent three tDCS sessions with one-week intervals in between. Effects of tDCS on RS fcMRI were determined at an individual as well as at a group level using both ROI-based and independent-component analyses (ICA). To evaluate the TRT reliability of individual active-tDCS and sham effects on RS fcMRI, voxel-wise intra-class correlation coefficients (ICC) of post-tDCS maps between testing sessions were calculated. For both approaches, results revealed low reliability of RS fcMRI after active tDCS (ICC_(2,1) = -0.09 - 0.16). Reliability of RS fcMRI (baselines only) was low to moderate for ROI-derived (ICC_(2,1) = 0.13 - 0.50) and low for ICA-derived connectivity (ICC_(2,1) = 0.19 - 0.34). Thus, for ROI-based analyses, the distribution of voxel-wise ICC was shifted to lower TRT reliability after active, but not after sham tDCS, for which the distribution was similar to baseline. The intra-individual variation observed here resembles variability of tDCS effects in motor regions and may be one reason why in this study robust tDCS effects at a group level were missing. The data can be used for appropriately designing large scale studies investigating methodological issues such as sources of variability and localisation of tDCS effects.

A Hitchhiker's Guide to Functional Magnetic Resonance Imaging

Functional Magnetic Resonance Imaging (fMRI) studies have become increasingly popular both with clinicians and researchers as they are capable of providing unique insights into brain functions. However, multiple technical considerations (ranging from specifics of paradigm design to imaging artifacts, complex protocol definition, and multitude of processing and methods of analysis, as well as intrinsic methodological limitations) must be considered and addressed in order to optimize fMRI analysis and to arrive at the most accurate and grounded interpretation of the data. In practice, the researcher/clinician must choose, from many available options, the most suitable software tool for each stage of the fMRI analysis pipeline. Herein we provide a straightforward guide designed to address, for each of the major stages, the techniques, and tools involved in the process. We have developed this guide both to help those new to the technique to overcome the most critical difficulties in its use, as well as to serve as a resource for the neuroimaging community.

Variance decomposition for single-subject task-based fMRI activity estimates across many sessions

Here we report an exploratory within-subject variance decomposition analysis conducted on a task-based fMRI dataset with an unusually large number of repeated measures (i.e., 500 trials in each of three different subjects) distributed across 100 functional scans and 9 to 10 different sessions. Within-subject variance was segregated into four primary components: variance across-sessions, variance across-runs within a session, variance across-blocks within a run, and residual measurement/modeling error. Our results reveal inhomogeneous and distinct spatial distributions of these variance components across significantly active voxels in grey matter. Measurement error is dominant across the whole brain. Detailed evaluation of the remaining three components shows that across-session variance is the second largest contributor to total variance in occipital cortex, while across-runs variance is the second dominant source for the rest of the brain. Network-specific analysis revealed that across-block variance contributes more to total variance in higher-order cognitive networks than in somatosensory cortex. Moreover, in some higher-order cognitive networks across-block variance can exceed across-session variance. These results help us better understand the temporal (i.e., across blocks, runs and sessions) and spatial distributions (i.e., across different networks) of within-subject natural variability in estimates of task responses in fMRI. They also suggest that different brain regions will show different natural levels of test-retest reliability even in the absence of residual artifacts and sufficiently high contrast-to-noise measurements. Further confirmation with a larger sample of subjects and other tasks is necessary to ensure generality of these results.

Improving fMRI reliability in presurgical mapping for brain tumours

PURPOSE

Functional MRI (fMRI) is becoming increasingly integrated into clinical practice for presurgical mapping. Current efforts are focused on validating data quality, with reliability being a major factor. In this paper, we demonstrate the utility of a recently developed approach that uses receiver operating characteristic-reliability (ROC-r) to: (1) identify reliable versus unreliable data sets; (2) automatically select processing options to enhance data quality; and (3) automatically select individualised thresholds for activation maps.

METHODS

Presurgical fMRI was conducted in 16 patients undergoing surgical treatment for brain tumours. Within-session test-retest fMRI was conducted, and ROC-reliability of the patient group was compared to a previous healthy control cohort. Individually optimised preprocessing pipelines were determined to improve reliability. Spatial correspondence was assessed by comparing the fMRI results to intraoperative cortical stimulation mapping, in terms of the distance to the nearest active fMRI voxel.

RESULTS

The average ROC-r reliability for the patients was 0.58±0.03, as compared to 0.72±0.02 in healthy controls. For the patient group, this increased significantly to 0.65±0.02 by adopting optimised preprocessing pipelines. Co-localisation of the fMRI maps with cortical stimulation was significantly better for more reliable versus less reliable data sets (8.3±0.9 vs 29±3 mm, respectively).

CONCLUSIONS

We demonstrated ROC-r analysis for identifying reliable fMRI data sets, choosing optimal postprocessing pipelines, and selecting patient-specific thresholds. Data sets with higher reliability also showed closer spatial correspondence to cortical stimulation. ROC-r can thus identify poor fMRI data at time of scanning, allowing for repeat scans when necessary. ROC-r analysis provides optimised and automated fMRI processing for improved presurgical mapping.

A Rotational Cylindrical fMRI Phantom for Image Quality Control

Purpose

A novel phantom for image quality testing for functional magnetic resonance imaging (fMRI) scans is described.

Methods

The cylindrical, rotatable, ~4.5L phantom, with eight wedge-shaped compartments, is used to simulate rest and activated states. The compartments contain NiCl₂ doped agar gel with alternating concentrations of agar (1.4%, 1.6%) to produce T₁ and T₂ values approximating brain grey matter. The Jacard index was used to compare the image distortions for echo planar imaging (EPI) and gradient recalled echo (GRE) scans. Contrast to noise ratio (_CNR) was compared across the imaging volume for GRE and EPI.

Results

The mean T₂ for the two agar concentrations were found to be 106.5±4.8, 94.5±4.7 ms, and T₁ of 1500±40 and 1485±30 ms, respectively. The Jacard index for GRE was generally found to be higher than for EPI (0.95 versus 0.8). The CNR varied from 20 to 50 across the slices and echo times used for EPI scans, and from 20 to 40 across the slices for the GRE scans. The phantom provided a reproducible CNR over 25 days.

Conclusions

The phantom provides a quantifiable signal change over a head-size imaging volume with EPI and GRE sequences, which was used for image quality assessment.

How cognitive neuroscience could be more biological-and what it might learn from clinical neuropsychology

Three widespread assumptions of Cognitive-affective Neuroscience are discussed: first, mental functions are assumed to be localized in circumscribed brain areas which can be exactly determined, at least in principle (localizationism). Second, this assumption is associated with the more general claim that these functions (and dysfunctions, such as in neurological or mental diseases) are somehow generated inside the brain (internalism). Third, these functions are seen to be “biological” in the sense that they can be decomposed and finally explained on the basis of elementary biological causes (i.e., genetic, molecular, neurophysiological etc.), causes that can be identified by experimental methods as the gold standard (isolationism). Clinical neuropsychology is widely assumed to support these tenets. However, by making reference to the ideas of Kurt Goldstein (1878–1965), one of its most important founders, I argue that none of these assumptions is sufficiently supported. From the perspective of a clinical-neuropsychological practitioner, assessing and treating brain damage sequelae reveals a quite different picture of the brain as well as of us “brain carriers”, making the organism (or person) in its specific environment the crucial reference point. This conclusion can be further elaborated: all experimental and clinical research on humans presupposes the notion of a situated, reflecting, and interacting subject, which precedes all kinds of scientific decomposition, however useful. These implications support the core assumptions of the embodiment approach to brain and mind, and, as I argue, Goldstein and his clinical-neuropsychological observations are part of its very origin, for both theoretical and historical reasons.

Number of events and reliability in fMRI

Relatively early in the history of fMRI, research focused on issues of power and reliability, with an important line concerning the establishment of optimal procedures for experimental design in order to maximize the various statistical properties of such designs. However, in recent years, tasks wherein events are defined only a posteriori, on the basis of behavior, have become increasingly common. Although these designs enable a much wider array of questions to be answered, they are not amenable to the tight control afforded by designs with events defined entirely a priori, and little work has assessed issues of power and reliability in such designs. We demonstrate how differences in numbers of events-as can occur with a posteriori event definition-affect reliability, both through simulation and in real data.

Which "neural activity" do you mean? fMRI, MEG, oscillations and neurotransmitters

Over the last 20 years, BOLD-FMRI has proved itself to be a powerful and versatile tool for the study of the neural substrate underpinning many of our cognitive and perceptual functions. However, exactly how it is coupled to the underlying neurophysiology, and how this coupling varies across the brain, across tasks and across individuals is still unclear. The story is further complicated by the fact that within the same cortical region, multiple evoked and induced oscillatory effects may be modulated during task execution, supporting different cognitive roles, and any or all of these may have metabolic demands that then drive the BOLD response. In this paper I shall concentrate on one experimental approach to shedding light on this problem i.e. the execution of the same experimental tasks using MEG and fMRI in order to reveal which electrophysiological responses best match the BOLD response spatially, temporally and functionally. The results demonstrate a rich and complex story that does not fit with a simplistic view of BOLD reflecting "neural activity" and suggests that we could consider the coupling between BOLD and the various parameters of neural function as an ill-posed inverse problem. Finally, I describe recent work linking individual variability in both cortical oscillations and the BOLD-fMRI response to variability in endogenous GABA concentration.