Numerical uncertainty in analytical pipelines lead to impactful variability in brain networks

Moving beyond processing- and analysis-related variation in resting-state functional brain imaging

When fields lack consensus standard methods and accessible ground truths, reproducibility can be more of an ideal than a reality. Such has been the case for functional neuroimaging, where there exists a sprawling space of tools and processing pipelines. We provide a critical evaluation of the impact of differences across five independently developed minimal preprocessing pipelines for functional magnetic resonance imaging. We show that, even when handling identical data, interpipeline agreement was only moderate, critically shedding light on a factor that limits cross-study reproducibility. We show that low interpipeline agreement can go unrecognized until the reliability of the underlying data is high, which is increasingly the case as the field progresses. Crucially we show that, when interpipeline agreement is compromised, so too is the consistency of insights from brain-wide association studies. We highlight the importance of comparing analytic configurations, because both widely discussed and commonly overlooked decisions can lead to marked variation.

Reproducibility in Neuroimaging Analysis: Challenges and Solutions

Recent years have marked a renaissance in efforts to increase research reproducibility in psychology, neuroscience, and related fields. Reproducibility is the cornerstone of a solid foundation of fundamental research-one that will support new theories built on valid findings and technological innovation that works. The increased focus on reproducibility has made the barriers to it increasingly apparent, along with the development of new tools and practices to overcome these barriers. Here, we review challenges, solutions, and emerging best practices with a particular emphasis on neuroimaging studies. We distinguish 3 main types of reproducibility, discussing each in turn. Analytical reproducibility is the ability to reproduce findings using the same data and methods. Replicability is the ability to find an effect in new datasets, using the same or similar methods. Finally, robustness to analytical variability refers to the ability to identify a finding consistently across variation in methods. The incorporation of these tools and practices will result in more reproducible, replicable, and robust psychological and brain research and a stronger scientific foundation across fields of inquiry.