A method for generating reproducible evidence in fMRI studies

The empirical replicability of task-based fMRI as a function of sample size

Replicating results (i.e. obtaining consistent results using a new independent dataset) is an essential part of good science. As replicability has consequences for theories derived from empirical studies, it is of utmost importance to better understand the underlying mechanisms influencing it. A popular tool for non-invasive neuroimaging studies is functional magnetic resonance imaging (fMRI). While the effect of underpowered studies is well documented, the empirical assessment of the interplay between sample size and replicability of results for task-based fMRI studies remains limited. In this work, we extend existing work on this assessment in two ways. Firstly, we use a large database of 1400 subjects performing four types of tasks from the IMAGEN project to subsample a series of independent samples of increasing size. Secondly, replicability is evaluated using a multi-dimensional framework consisting of 3 different measures: (un)conditional test-retest reliability, coherence and stability. We demonstrate not only a positive effect of sample size, but also a trade-off between spatial resolution and replicability. When replicability is assessed voxelwise or when observing small areas of activation, a larger sample size than typically used in fMRI is required to replicate results. On the other hand, when focussing on clusters of voxels, we observe a higher replicability. In addition, we observe variability in the size of clusters of activation between experimental paradigms or contrasts of parameter estimates within these.

Quality assurance of human functional magnetic resonance imaging: a literature review

Functional magnetic resonance imaging (fMRI) has been a popular approach in brain research over the past 20 years. It offers a noninvasive method to probe the brain and uses blood oxygenation level dependent (BOLD) signal changes to access brain function. However, the BOLD signal only represents a small fraction of the total MR signal. System instability and various noise have a strong impact on the BOLD signal. Additionally, fMRI applies fast imaging technique to record brain cognitive process over time, requiring high temporal stability of MR scanners. Furthermore, data acquisition, image quality, processing, and statistical analysis methods also have a great effect on the results of fMRI studies. Quality assurance (QA) programs for fMRI can test the stability of MR scanners, evaluate the quality of fMRI and help to find errors during fMRI scanning, thereby greatly enhancing the success rate of fMRI. In this review, we focus on previous studies which developed QA programs and methods in SCI/SCIE citation peer-reviewed publications over the last 20 years, including topics on existing fMRI QA programs, QA phantoms, image QA metrics, quality evaluation of existing preprocessing pipelines and fMRI statistical analysis methods. The summarized studies were classified into four categories: QA of fMRI systems, QA of fMRI data, quality evaluation of data processing pipelines and statistical methods and QA of task-related fMRI. Summary tables and figures of QA programs and metrics have been developed based on the comprehensive review of the literature.

Improving fMRI reliability in presurgical mapping for brain tumours

PURPOSE

Functional MRI (fMRI) is becoming increasingly integrated into clinical practice for presurgical mapping. Current efforts are focused on validating data quality, with reliability being a major factor. In this paper, we demonstrate the utility of a recently developed approach that uses receiver operating characteristic-reliability (ROC-r) to: (1) identify reliable versus unreliable data sets; (2) automatically select processing options to enhance data quality; and (3) automatically select individualised thresholds for activation maps.

METHODS

Presurgical fMRI was conducted in 16 patients undergoing surgical treatment for brain tumours. Within-session test-retest fMRI was conducted, and ROC-reliability of the patient group was compared to a previous healthy control cohort. Individually optimised preprocessing pipelines were determined to improve reliability. Spatial correspondence was assessed by comparing the fMRI results to intraoperative cortical stimulation mapping, in terms of the distance to the nearest active fMRI voxel.

RESULTS

The average ROC-r reliability for the patients was 0.58±0.03, as compared to 0.72±0.02 in healthy controls. For the patient group, this increased significantly to 0.65±0.02 by adopting optimised preprocessing pipelines. Co-localisation of the fMRI maps with cortical stimulation was significantly better for more reliable versus less reliable data sets (8.3±0.9 vs 29±3 mm, respectively).

CONCLUSIONS

We demonstrated ROC-r analysis for identifying reliable fMRI data sets, choosing optimal postprocessing pipelines, and selecting patient-specific thresholds. Data sets with higher reliability also showed closer spatial correspondence to cortical stimulation. ROC-r can thus identify poor fMRI data at time of scanning, allowing for repeat scans when necessary. ROC-r analysis provides optimised and automated fMRI processing for improved presurgical mapping.

Reliability of Task-Based fMRI for Preoperative Planning: A Test-Retest Study in Brain Tumor Patients and Healthy Controls

Background

Functional magnetic resonance imaging (fMRI) continues to develop as a clinical tool for patients with brain cancer, offering data that may directly influence surgical decisions. Unfortunately, routine integration of preoperative fMRI has been limited by concerns about reliability. Many pertinent studies have been undertaken involving healthy controls, but work involving brain tumor patients has been limited. To develop fMRI fully as a clinical tool, it will be critical to examine these reliability issues among patients with brain tumors. The present work is the first to extensively characterize differences in activation map quality between brain tumor patients and healthy controls, including the effects of tumor grade and the chosen behavioral testing paradigm on reliability outcomes.

Method

Test-retest data were collected for a group of low-grade (n = 6) and high-grade glioma (n = 6) patients, and for matched healthy controls (n = 12), who performed motor and language tasks during a single fMRI session. Reliability was characterized by the spatial overlap and displacement of brain activity clusters, BOLD signal stability, and the laterality index. Significance testing was performed to assess differences in reliability between the patients and controls, and low-grade and high-grade patients; as well as between different fMRI testing paradigms.

Results

There were few significant differences in fMRI reliability measures between patients and controls. Reliability was significantly lower when comparing high-grade tumor patients to controls, or to low-grade tumor patients. The motor task produced more reliable activation patterns than the language tasks, as did the rhyming task in comparison to the phonemic fluency task.

Conclusion

In low-grade glioma patients, fMRI data are as reliable as healthy control subjects. For high-grade glioma patients, further investigation is required to determine the underlying causes of reduced reliability. To maximize reliability outcomes, testing paradigms should be carefully selected to generate robust activation patterns.

Additional evidence for the sustained effect of acupuncture at the vision-related acupuncture point, GB37

OBJECTIVE

To investigate the dynamics underlying the sustained effect of acupuncture as a possible explanation of earlier findings that acupuncture stimulation at the vision-related acupuncture point, GB37, cannot specifically change the functional MRI (fMRI) signals of the visual cortex compared with stimulation at an adjacent non-meridian point.

METHODS

The 'on-off' experimental design was separated into four series conditions: 1 min of baseline scanning at the beginning, then two stimulation epochs separated by a 50 s 'rest' period, and then a 1 min 'rest' epoch. The standard General Linear Model (GLM) approach and multi-conditions analysis were used.

RESULTS

Results from the multi-conditions analysis were different from those from the standard GLM analysis. We found that the neural signal of the limbic-paralimbic-neocortical system after acupuncture stimulus lasted longer than the putative period. Furthermore, the fMRI signal changes in the occipital cortex showed different temporal patterns between GB37 and the non-meridian point.

CONCLUSIONS

Owing to the sustained effect of acupuncture, standard GLM analysis may be unsuitable for 'on-off' design acupuncture studies and lead to uncertain and contradictory results. The findings from this study suggest that acupuncture at GB37 can induce complex brain activity in the vision cortex. The state-related neural signal may reflect one of the significant characteristics underlying acupuncture.

Combining spatial priors and anatomical information for fMRI detection

In this paper, we analyze Markov Random Field (MRF) as a spatial regularizer in fMRI detection. The low signal-to-noise ratio (SNR) in fMRI images presents a serious challenge for detection algorithms, making regularization necessary to achieve good detection accuracy. Gaussian smoothing, traditionally employed to boost SNR, often produces over-smoothed activation maps. Recently, the use of MRF priors has been suggested as an alternative regularization approach. However, solving for an optimal configuration of the MRF is NP-hard in general. In this work, we investigate fast inference algorithms based on the Mean Field approximation in application to MRF priors for fMRI detection. Furthermore, we propose a novel way to incorporate anatomical information into the MRF-based detection framework and into the traditional smoothing methods. Intuitively speaking, the anatomical evidence increases the likelihood of activation in the gray matter and improves spatial coherency of the resulting activation maps within each tissue type. Validation using the receiver operating characteristic (ROC) analysis and the confusion matrix analysis on simulated data illustrates substantial improvement in detection accuracy using the anatomically guided MRF spatial regularizer. We further demonstrate the potential benefits of the proposed method in real fMRI signals of reduced length. The anatomically guided MRF regularizer enables significant reduction of the scan length while maintaining the quality of the resulting activation maps.

Is it time to re-prioritize neuroimaging databases and digital repositories?

The development of in vivo brain imaging has lead to the collection of large quantities of digital information. In any individual research article, several tens of gigabytes-worth of data may be represented-collected across normal and patient samples. With the ease of collecting such data, there is increased desire for brain imaging datasets to be openly shared through sophisticated databases. However, very often the raw and pre-processed versions of these data are not available to researchers outside of the team that collected them. A range of neuroimaging databasing approaches has streamlined the transmission, storage, and dissemination of data from such brain imaging studies. Though early sociological and technical concerns have been addressed, they have not been ameliorated altogether for many in the field. In this article, we review the progress made in neuroimaging databases, their role in data sharing, data management, potential for the construction of brain atlases, recording data provenance, and value for re-analysis, new publication, and training. We feature the LONI IDA as an example of an archive being used as a source for brain atlas workflow construction, list several instances of other successful uses of image databases, and comment on archive sustainability. Finally, we suggest that, given these developments, now is the time for the neuroimaging community to re-prioritize large-scale databases as a valuable component of brain imaging science.

Test-retest and between-site reliability in a multicenter fMRI study

In the present report, estimates of test-retest and between-site reliability of fMRI assessments were produced in the context of a multicenter fMRI reliability study (FBIRN Phase 1, www.nbirn.net). Five subjects were scanned on 10 MRI scanners on two occasions. The fMRI task was a simple block design sensorimotor task. The impulse response functions to the stimulation block were derived using an FIR-deconvolution analysis with FMRISTAT. Six functionally-derived ROIs covering the visual, auditory and motor cortices, created from a prior analysis, were used. Two dependent variables were compared: percent signal change and contrast-to-noise-ratio. Reliability was assessed with intraclass correlation coefficients derived from a variance components analysis. Test-retest reliability was high, but initially, between-site reliability was low, indicating a strong contribution from site and site-by-subject variance. However, a number of factors that can markedly improve between-site reliability were uncovered, including increasing the size of the ROIs, adjusting for smoothness differences, and inclusion of additional runs. By employing multiple steps, between-site reliability for 3T scanners was increased by 123%. Dropping one site at a time and assessing reliability can be a useful method of assessing the sensitivity of the results to particular sites. These findings should provide guidance toothers on the best practices for future multicenter studies.

Evaluation and comparison of GLM- and CVA-based fMRI processing pipelines with Java-based fMRI processing pipeline evaluation system

Activation patterns identified by fMRI processing pipelines or fMRI software packages are usually determined by the preprocessing options, parameters, and statistical models used. Previous studies that evaluated options of GLM (general linear model)--based fMRI processing pipelines are mainly based on simulated data with receiver operating characteristics (ROC) analysis, but evaluation of such fMRI processing pipelines on real fMRI data is rare. To understand the effect of processing options on performance of GLM-based fMRI processing pipelines with real fMRI data, we investigated the impact of commonly-used fMRI preprocessing steps; optimized the associated GLM-based single-subject processing pipelines; and quantitatively compared univariate GLM (in FSL.FEAT and NPAIRS.GLM) and multivariate CVA (canonical variates analysis) (in NPAIRS.CVA)-based analytic models in single-subject analysis with a recently developed fMRI processing pipeline evaluation system based on prediction accuracy (classification accuracy) and reproducibility performance metrics. For block-design data, we found that with GLM analysis (1) slice timing correction and global intensity normalization have little consistent impact on fMRI processing pipelines, spatial smoothing and high-pass filtering or temporal detrending significantly increases pipeline performance and thus are essential for robust fMRI statistical analysis; (2) combined optimization of spatial smoothing and temporal detrending improves pipeline performance; and (3) in general, the prediction performance of multivariate CVA is higher than that of the univariate GLM, while univariate GLM is more reproducible than multivariate CVA. Because of the different bias-variance trade-offs of univariate and multivariate models, it may be necessary to consider a consensus approach to obtain more accurate activation patterns in fMRI data.

Mapping the human brain: new insights from FMRI data sharing

The sharing of primary data in the field of neuroscience has received considerable scrutiny from scientific societies and from science journals. Many see this as value added for science publishing that can enhance and inform secondary examination of data and results. Still others worry that data sharing is an undue burden for researchers with little long term value to science. But examples of how data sharing can be done successfully do exist. The fMRI Data Center, established at Dartmouth College in 2000 and now based at the University of California Santa Barbara, has worked to facilitate the open sharing of neuroimaging data from peer-reviewed papers to foster progress in cognitive science. The fMRI study on the representation of objects in the human occipital and temporal cortex, published in 2000 in the Journal of Cognitive Neuroscience (JOCN), marked the first deposition in the new database. Despite initial concerns about fMRI data sharing, this data set was frequently downloaded. We describe the original results of distributed brain activation patterns elicited by faces and objects in the human visual system, and overview several secondary analyses by independent investigators. A philosopher tested Husserl's temporal components of consciousness, whereas other brain imagers deployed new analytic tools, from Dynamic Causal Modeling, which estimates the neural interactions between cortical regions, to a novel method for constructing reproducibility maps. These re-analyses revealed new findings not reported in the original study, provided new perspectives on visual perception, generated new predictions, and resulted in new collaborations and publications in high profile journals.

WSPM: Wavelet-based statistical parametric mapping

Recently, we have introduced an integrated framework that combines wavelet-based processing with statistical testing in the spatial domain. In this paper, we propose two important enhancements of the framework. First, we revisit the underlying paradigm; i.e., that the effect of the wavelet processing can be considered as an adaptive denoising step to "improve" the parameter map, followed by a statistical detection procedure that takes into account the non-linear processing of the data. With an appropriate modification of the framework, we show that it is possible to reduce the spatial bias of the method with respect to the best linear estimate, providing conservative results that are closer to the original data. Second, we propose an extension of our earlier technique that compensates for the lack of shift-invariance of the wavelet transform. We demonstrate experimentally that both enhancements have a positive effect on performance. In particular, we present a reproducibility study for multi-session data that compares WSPM against SPM with different amounts of smoothing. The full approach is available as a toolbox, named WSPM, for the SPM2 software; it takes advantage of multiple options and features of SPM such as the general linear model.

Divide and conquer: a defense of functional localizers

Numerous functionally distinct regions of cortex (e.g., V1, MT, the fusiform face area) can be easily identified in any normal human subject in just a few minutes of fMRI scanning. However, the locations of these regions vary across subjects. Investigations of these regions have therefore often used a functional region of interest (fROI) approach in which the region is first identified functionally in each subject individually, before subsequent scans in the same subjects test specific hypotheses concerning that region. This fROI method, which resembled long-established practice in visual neurophysiology, has methodological, statistical, and theoretical advantages over standard alternatives (such as whole-brain analyses of group data): (i) because functional properties are more consistently and robustly associated with fROIs than with locations in stereotaxic space, functional hypotheses concerning fROIs are often the most straightforward to frame, motivate, and test, (ii) because hypotheses are tested in only a handful of fROIs (instead of in tens of thousands of voxels), advance specification of fROIs provides a massive increase in statistical power over whole-brain analyses, and (iii) some fROIs may serve as candidate distinct components of the mind/brain worth investigation as such. Of course fROIs can be productively used in conjunction with other complementary methods. Here, we explain the motivation for and advantages of the fROI approach, and we rebut the criticism of this method offered by Friston et al. (Friston, K., Rotshtein, P., Geng, J., Sterzer, P., Henson, R., in press. A critique of functional localizers. NeuroImage).