1
|
Rosenblatt M, Tejavibulya L, Sun H, Camp CC, Khaitova M, Adkinson BD, Jiang R, Westwater ML, Noble S, Scheinost D. Power and reproducibility in the external validation of brain-phenotype predictions. Nat Hum Behav 2024:10.1038/s41562-024-01931-7. [PMID: 39085406 DOI: 10.1038/s41562-024-01931-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 06/18/2024] [Indexed: 08/02/2024]
Abstract
Brain-phenotype predictive models seek to identify reproducible and generalizable brain-phenotype associations. External validation, or the evaluation of a model in external datasets, is the gold standard in evaluating the generalizability of models in neuroimaging. Unlike typical studies, external validation involves two sample sizes: the training and the external sample sizes. Thus, traditional power calculations may not be appropriate. Here we ran over 900 million resampling-based simulations in functional and structural connectivity data to investigate the relationship between training sample size, external sample size, phenotype effect size, theoretical power and simulated power. Our analysis included a wide range of datasets: the Healthy Brain Network, the Adolescent Brain Cognitive Development Study, the Human Connectome Project (Development and Young Adult), the Philadelphia Neurodevelopmental Cohort, the Queensland Twin Adolescent Brain Project, and the Chinese Human Connectome Project; and phenotypes: age, body mass index, matrix reasoning, working memory, attention problems, anxiety/depression symptoms and relational processing. High effect size predictions achieved adequate power with training and external sample sizes of a few hundred individuals, whereas low and medium effect size predictions required hundreds to thousands of training and external samples. In addition, most previous external validation studies used sample sizes prone to low power, and theoretical power curves should be adjusted for the training sample size. Furthermore, model performance in internal validation often informed subsequent external validation performance (Pearson's r difference <0.2), particularly for well-harmonized datasets. These results could help decide how to power future external validation studies.
Collapse
Affiliation(s)
- Matthew Rosenblatt
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA.
| | - Link Tejavibulya
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Huili Sun
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Chris C Camp
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Milana Khaitova
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
| | - Brendan D Adkinson
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Rongtao Jiang
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
| | - Margaret L Westwater
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
| | - Stephanie Noble
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
- Department of Bioengineering, Northeastern University, Boston, MA, USA
- Department of Psychology, Northeastern University, Boston, MA, USA
| | - Dustin Scheinost
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
- Child Study Center, Yale School of Medicine, New Haven, CT, USA
- Department of Statistics and Data Science, Yale University, New Haven, CT, USA
| |
Collapse
|
2
|
Tetereva A, Pat N. Brain age has limited utility as a biomarker for capturing fluid cognition in older individuals. eLife 2024; 12:RP87297. [PMID: 38869938 PMCID: PMC11175613 DOI: 10.7554/elife.87297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/14/2024] Open
Abstract
One well-known biomarker candidate that supposedly helps capture fluid cognition is Brain Age, or a predicted value based on machine-learning models built to predict chronological age from brain MRI. To formally evaluate the utility of Brain Age for capturing fluid cognition, we built 26 age-prediction models for Brain Age based on different combinations of MRI modalities, using the Human Connectome Project in Aging (n=504, 36-100 years old). First, based on commonality analyses, we found a large overlap between Brain Age and chronological age: Brain Age could uniquely add only around 1.6% in explaining variation in fluid cognition over and above chronological age. Second, the age-prediction models that performed better at predicting chronological age did NOT necessarily create better Brain Age for capturing fluid cognition over and above chronological age. Instead, better-performing age-prediction models created Brain Age that overlapped larger with chronological age, up to around 29% out of 32%, in explaining fluid cognition. Third, Brain Age missed around 11% of the total variation in fluid cognition that could have been explained by the brain variation. That is, directly predicting fluid cognition from brain MRI data (instead of relying on Brain Age and chronological age) could lead to around a 1/3-time improvement of the total variation explained. Accordingly, we demonstrated the limited utility of Brain Age as a biomarker for fluid cognition and made some suggestions to ensure the utility of Brain Age in explaining fluid cognition and other phenotypes of interest.
Collapse
Affiliation(s)
- Alina Tetereva
- Department of Psychology, University of OtagoDunedinNew Zealand
| | - Narun Pat
- Department of Psychology, University of OtagoDunedinNew Zealand
| |
Collapse
|
3
|
Molloy MF, Saygin ZM, Osher DE. Predicting high-level visual areas in the absence of task fMRI. Sci Rep 2024; 14:11376. [PMID: 38762549 PMCID: PMC11102456 DOI: 10.1038/s41598-024-62098-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Accepted: 05/13/2024] [Indexed: 05/20/2024] Open
Abstract
The ventral visual stream is organized into units, or functional regions of interest (fROIs), specialized for processing high-level visual categories. Task-based fMRI scans ("localizers") are typically used to identify each individual's nuanced set of fROIs. The unique landscape of an individual's functional activation may rely in large part on their specialized connectivity patterns; recent studies corroborate this by showing that connectivity can predict individual differences in neural responses. We focus on the ventral visual stream and ask: how well can an individual's resting state functional connectivity localize their fROIs for face, body, scene, and object perception? And are the neural processors for any particular visual category better predicted by connectivity than others, suggesting a tighter mechanistic relationship between connectivity and function? We found, among 18 fROIs predicted from connectivity for each subject, all but one were selective for their preferred visual category. Defining an individual's fROIs based on their connectivity patterns yielded regions that were more selective than regions identified from previous studies or atlases in nearly all cases. Overall, we found that in the absence of a domain-specific localizer task, a 10-min resting state scan can be reliably used for defining these fROIs.
Collapse
Affiliation(s)
- M Fiona Molloy
- Department of Psychology, The Ohio State University, 1835 Neil Avenue, Columbus, OH, 43210, USA
- Department of Psychiatry, University of Michigan, Ann Arbor, MI, USA
| | - Zeynep M Saygin
- Department of Psychology, The Ohio State University, 1835 Neil Avenue, Columbus, OH, 43210, USA
| | - David E Osher
- Department of Psychology, The Ohio State University, 1835 Neil Avenue, Columbus, OH, 43210, USA.
| |
Collapse
|
4
|
Tetereva A, Knodt AR, Melzer TR, van der Vliet W, Gibson B, Hariri AR, Whitman ET, Li J, Deng J, Ireland D, Ramrakha S, Pat N. Improving Predictability, Test-Retest Reliability and Generalisability of Brain-Wide Associations for Cognitive Abilities via Multimodal Stacking. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.05.03.589404. [PMID: 38746222 PMCID: PMC11092590 DOI: 10.1101/2024.05.03.589404] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
Brain-wide association studies (BWASs) have attempted to relate cognitive abilities with brain phenotypes, but have been challenged by issues such as predictability, test-retest reliability, and cross-cohort generalisability. To tackle these challenges, we proposed "stacking" that combines brain magnetic resonance imaging of different modalities, from task-fMRI contrasts and functional connectivity during tasks and rest to structural measures, into one prediction model. We benchmarked the benefits of stacking, using the Human Connectome Projects: Young Adults and Aging and the Dunedin Multidisciplinary Health and Development Study. For predictability, stacked models led to out-of-sample r ∼.5-.6 when predicting cognitive abilities at the time of scanning and 36 years earlier. For test-retest reliability, stacked models reached an excellent level of reliability (ICC>.75), even when we stacked only task-fMRI contrasts together. For generalisability, a stacked model with non-task MRI built from one dataset significantly predicted cognitive abilities in other datasets. Altogether, stacking is a viable approach to undertake the three challenges of BWAS for cognitive abilities.
Collapse
|
5
|
Rosenblatt M, Tejavibulya L, Jiang R, Noble S, Scheinost D. Data leakage inflates prediction performance in connectome-based machine learning models. Nat Commun 2024; 15:1829. [PMID: 38418819 PMCID: PMC10901797 DOI: 10.1038/s41467-024-46150-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Accepted: 02/15/2024] [Indexed: 03/02/2024] Open
Abstract
Predictive modeling is a central technique in neuroimaging to identify brain-behavior relationships and test their generalizability to unseen data. However, data leakage undermines the validity of predictive models by breaching the separation between training and test data. Leakage is always an incorrect practice but still pervasive in machine learning. Understanding its effects on neuroimaging predictive models can inform how leakage affects existing literature. Here, we investigate the effects of five forms of leakage-involving feature selection, covariate correction, and dependence between subjects-on functional and structural connectome-based machine learning models across four datasets and three phenotypes. Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects. Furthermore, small datasets exacerbate the effects of leakage. Overall, our results illustrate the variable effects of leakage and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.
Collapse
Affiliation(s)
- Matthew Rosenblatt
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA.
| | - Link Tejavibulya
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Rongtao Jiang
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
| | - Stephanie Noble
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
- Department of Bioengineering, Northeastern University, Boston, MA, USA
- Department of Psychology, Northeastern University, Boston, MA, USA
| | - Dustin Scheinost
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
- Child Study Center, Yale School of Medicine, New Haven, CT, USA
- Department of Statistics & Data Science, Yale University, New Haven, CT, USA
| |
Collapse
|
6
|
Dadashkarimi J, Karbasi A, Liang Q, Rosenblatt M, Noble S, Foster M, Rodriguez R, Adkinson B, Ye J, Sun H, Camp C, Farruggia M, Tejavibulya L, Dai W, Jiang R, Pollatou A, Scheinost D. Cross Atlas Remapping via Optimal Transport (CAROT): Creating connectomes for different atlases when raw data is not available. Med Image Anal 2023; 88:102864. [PMID: 37352650 PMCID: PMC10526726 DOI: 10.1016/j.media.2023.102864] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Revised: 02/10/2023] [Accepted: 05/31/2023] [Indexed: 06/25/2023]
Abstract
Open-source, publicly available neuroimaging datasets - whether from large-scale data collection efforts or pooled from multiple smaller studies - offer unprecedented sample sizes and promote generalization efforts. Releasing data can democratize science, increase the replicability of findings, and lead to discoveries. Partly due to patient privacy, computational, and data storage concerns, researchers typically release preprocessed data with the voxelwise time series parcellated into a map of predefined regions, known as an atlas. However, releasing preprocessed data also limits the choices available to the end-user. This is especially true for connectomics, as connectomes created from different atlases are not directly comparable. Since there exist several atlases with no gold standards, it is unrealistic to have processed, open-source data available from all atlases. Together, these limitations directly inhibit the potential benefits of open-source neuroimaging data. To address these limitations, we introduce Cross Atlas Remapping via Optimal Transport (CAROT) to find a mapping between two atlases. This approach allows data processed from one atlas to be directly transformed into a connectome based on another atlas without the need for raw data access. To validate CAROT, we compare reconstructed connectomes against their original counterparts (i.e., connectomes generated directly from an atlas), demonstrate the utility of transformed connectomes in downstream analyses, and show how a connectome-based predictive model can generalize to publicly available data that was processed with different atlases. Overall, CAROT can reconstruct connectomes from an extensive set of atlases - without needing the raw data - allowing already processed connectomes to be easily reused in a wide range of analyses while eliminating redundant processing efforts. We share this tool as both source code and as a stand-alone web application (http://carotproject.com/).
Collapse
Affiliation(s)
| | - Amin Karbasi
- Computer Science Department, Yale University, New Haven, CT, USA; Department of Electrical Engineering, Yale University, New Haven, CT, USA; Department of Statistics & Data Science, Yale University, New Haven, CT, USA
| | - Qinghao Liang
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Matthew Rosenblatt
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Stephanie Noble
- Department of Radiology and Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Maya Foster
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Raimundo Rodriguez
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Brendan Adkinson
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Jean Ye
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Huili Sun
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Chris Camp
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Michael Farruggia
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Link Tejavibulya
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Wei Dai
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Rongtao Jiang
- Department of Radiology and Biomedical Engineering, Yale University, New Haven, CT, USA
| | - Angeliki Pollatou
- Developing Brain Institute, Children's National Hospital, Washington DC, USA
| | - Dustin Scheinost
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA; Department of Statistics & Data Science, Yale University, New Haven, CT, USA; Child Study Center, Yale School of Medicine, New Haven, CT, USA; Department of Radiology and Biomedical Engineering, Yale University, New Haven, CT, USA
| |
Collapse
|
7
|
McKay NS, Gordon BA, Hornbeck RC, Dincer A, Flores S, Keefe SJ, Joseph-Mathurin N, Jack CR, Koeppe R, Millar PR, Ances BM, Chen CD, Daniels A, Hobbs DA, Jackson K, Koudelis D, Massoumzadeh P, McCullough A, Nickels ML, Rahmani F, Swisher L, Wang Q, Allegri RF, Berman SB, Brickman AM, Brooks WS, Cash DM, Chhatwal JP, Day GS, Farlow MR, la Fougère C, Fox NC, Fulham M, Ghetti B, Graff-Radford N, Ikeuchi T, Klunk W, Lee JH, Levin J, Martins R, Masters CL, McConathy J, Mori H, Noble JM, Reischl G, Rowe C, Salloway S, Sanchez-Valle R, Schofield PR, Shimada H, Shoji M, Su Y, Suzuki K, Vöglein J, Yakushev I, Cruchaga C, Hassenstab J, Karch C, McDade E, Perrin RJ, Xiong C, Morris JC, Bateman RJ, Benzinger TLS. Positron emission tomography and magnetic resonance imaging methods and datasets within the Dominantly Inherited Alzheimer Network (DIAN). Nat Neurosci 2023; 26:1449-1460. [PMID: 37429916 PMCID: PMC10400428 DOI: 10.1038/s41593-023-01359-8] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Accepted: 05/15/2023] [Indexed: 07/12/2023]
Abstract
The Dominantly Inherited Alzheimer Network (DIAN) is an international collaboration studying autosomal dominant Alzheimer disease (ADAD). ADAD arises from mutations occurring in three genes. Offspring from ADAD families have a 50% chance of inheriting their familial mutation, so non-carrier siblings can be recruited for comparisons in case-control studies. The age of onset in ADAD is highly predictable within families, allowing researchers to estimate an individual's point in the disease trajectory. These characteristics allow candidate AD biomarker measurements to be reliably mapped during the preclinical phase. Although ADAD represents a small proportion of AD cases, understanding neuroimaging-based changes that occur during the preclinical period may provide insight into early disease stages of 'sporadic' AD also. Additionally, this study provides rich data for research in healthy aging through inclusion of the non-carrier controls. Here we introduce the neuroimaging dataset collected and describe how this resource can be used by a range of researchers.
Collapse
Affiliation(s)
| | | | | | - Aylin Dincer
- Washington University in St. Louis, St. Louis, MO, USA
| | - Shaney Flores
- Washington University in St. Louis, St. Louis, MO, USA
| | - Sarah J Keefe
- Washington University in St. Louis, St. Louis, MO, USA
| | | | | | | | | | - Beau M Ances
- Washington University in St. Louis, St. Louis, MO, USA
| | | | | | - Diana A Hobbs
- Washington University in St. Louis, St. Louis, MO, USA
| | | | | | | | | | | | | | - Laura Swisher
- Washington University in St. Louis, St. Louis, MO, USA
| | - Qing Wang
- Washington University in St. Louis, St. Louis, MO, USA
| | | | | | - Adam M Brickman
- Columbia University Irving Medical Center, New York, NY, USA
| | - William S Brooks
- Neuroscience Research Australia, Sydney, New South Wales, Australia
| | - David M Cash
- UK Dementia Research Institute at University College London, London, UK
- University College London, London, UK
| | - Jasmeer P Chhatwal
- Massachusetts General and Brigham & Women's Hospitals, Harvard Medical School, Boston, MA, USA
| | | | | | - Christian la Fougère
- Department of Radiology, University of Tübingen, Tübingen, Germany
- German Center for Neurodegenerative Diseases (DZNE), Tübingen, Germany
| | - Nick C Fox
- UK Dementia Research Institute at University College London, London, UK
- University College London, London, UK
| | - Michael Fulham
- Royal Prince Alfred Hospital, Sydney, New South Wales, Australia
| | | | | | | | | | | | - Johannes Levin
- German Center for Neurodegenerative Diseases (DZNE), Munich, Germany
| | - Ralph Martins
- Edith Cowan University, Joondalup, Western Australia, Australia
| | | | | | | | - James M Noble
- Columbia University Irving Medical Center, New York, NY, USA
| | - Gerald Reischl
- Department of Radiology, University of Tübingen, Tübingen, Germany
| | | | | | - Raquel Sanchez-Valle
- Alzheimer's Disease and Other Cognitive Disorders Unit, Neurology Service, Hospital Clínic de Barcelona, IDIBAPS, University of Barcelona, Barcelona, Spain
| | - Peter R Schofield
- Neuroscience Research Australia, Sydney, New South Wales, Australia
- School of Biomedical Sciences, University of New South Wales, Sydney, New South Wales, Australia
| | | | | | - Yi Su
- Banner Alzheimer's Institute, Phoenix, AZ, USA
| | | | - Jonathan Vöglein
- German Center for Neurodegenerative Diseases (DZNE), Munich, Germany
- Department of Neurology, Ludwig-Maximilians-Universität München, München, Germany
| | - Igor Yakushev
- School of Medicine, Technical University of Munich, Munich, Germany
| | | | | | - Celeste Karch
- Washington University in St. Louis, St. Louis, MO, USA
| | - Eric McDade
- Washington University in St. Louis, St. Louis, MO, USA
| | | | | | - John C Morris
- Washington University in St. Louis, St. Louis, MO, USA
| | | | | |
Collapse
|
8
|
Machlin L, McLaughlin KA. Pre-pandemic brain structure and function and adolescent psychopathology during the COVID-19 pandemic. Curr Opin Psychol 2023; 52:101647. [PMID: 37429074 PMCID: PMC10414753 DOI: 10.1016/j.copsyc.2023.101647] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2023] [Revised: 06/14/2023] [Accepted: 06/17/2023] [Indexed: 07/12/2023]
Abstract
The COVID-19 pandemic has presented unprecedented challenges for youths and families, dramatically increasing exposure to stressors and stress-related psychopathology. Increasing work has leveraged pre-pandemic neuroimaging data to predict adolescent psychopathology and stress responses during the pandemic, with a particular focus on internalizing symptoms. We review this recent literature on pre-pandemic brain structure and function and adolescent internalizing psychopathology during the pandemic. At present, existing studies have not consistently identified specific alterations in brain structure and function that predict anxiety or depressive symptoms during the pandemic. In contrast, exposure to stress and adversity before and during the pandemic as well as access to peer and family support have emerged as consistent and reliable predictors of youth mental health during the pandemic.
Collapse
Affiliation(s)
- Laura Machlin
- Department of Psychology, Harvard University, United States.
| | | |
Collapse
|
9
|
Rosenblatt M, Rodriguez RX, Westwater ML, Dai W, Horien C, Greene AS, Constable RT, Noble S, Scheinost D. Connectome-based machine learning models are vulnerable to subtle data manipulations. PATTERNS (NEW YORK, N.Y.) 2023; 4:100756. [PMID: 37521052 PMCID: PMC10382940 DOI: 10.1016/j.patter.2023.100756] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/06/2023] [Revised: 03/10/2023] [Accepted: 04/24/2023] [Indexed: 08/01/2023]
Abstract
Neuroimaging-based predictive models continue to improve in performance, yet a widely overlooked aspect of these models is "trustworthiness," or robustness to data manipulations. High trustworthiness is imperative for researchers to have confidence in their findings and interpretations. In this work, we used functional connectomes to explore how minor data manipulations influence machine learning predictions. These manipulations included a method to falsely enhance prediction performance and adversarial noise attacks designed to degrade performance. Although these data manipulations drastically changed model performance, the original and manipulated data were extremely similar (r = 0.99) and did not affect other downstream analysis. Essentially, connectome data could be inconspicuously modified to achieve any desired prediction performance. Overall, our enhancement attacks and evaluation of existing adversarial noise attacks in connectome-based models highlight the need for counter-measures that improve the trustworthiness to preserve the integrity of academic research and any potential translational applications.
Collapse
Affiliation(s)
- Matthew Rosenblatt
- Department of Biomedical Engineering, Yale School of Engineering and Applied Science, New Haven, CT 06510, USA
| | - Raimundo X. Rodriguez
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
| | - Margaret L. Westwater
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
| | - Wei Dai
- Department of Biostatistics, Yale School of Public Health, New Haven, CT 06510, USA
| | - Corey Horien
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
| | - Abigail S. Greene
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
| | - R. Todd Constable
- Department of Biomedical Engineering, Yale School of Engineering and Applied Science, New Haven, CT 06510, USA
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
- Department of Neurosurgery, Yale School of Medicine, New Haven, CT 06510, USA
| | - Stephanie Noble
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
| | - Dustin Scheinost
- Department of Biomedical Engineering, Yale School of Engineering and Applied Science, New Haven, CT 06510, USA
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
- Department of Statistics & Data Science, Yale University, New Haven, CT 06510, USA
- Child Study Center, Yale School of Medicine, New Haven, CT 06510, USA
- Wu Tsai Institute, Yale University, New Haven, CT 06510, USA
| |
Collapse
|
10
|
Hu F, Chen AA, Horng H, Bashyam V, Davatzikos C, Alexander-Bloch A, Li M, Shou H, Satterthwaite TD, Yu M, Shinohara RT. Image harmonization: A review of statistical and deep learning methods for removing batch effects and evaluation metrics for effective harmonization. Neuroimage 2023; 274:120125. [PMID: 37084926 PMCID: PMC10257347 DOI: 10.1016/j.neuroimage.2023.120125] [Citation(s) in RCA: 18] [Impact Index Per Article: 18.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Revised: 04/12/2023] [Accepted: 04/19/2023] [Indexed: 04/23/2023] Open
Abstract
Magnetic resonance imaging and computed tomography from multiple batches (e.g. sites, scanners, datasets, etc.) are increasingly used alongside complex downstream analyses to obtain new insights into the human brain. However, significant confounding due to batch-related technical variation, called batch effects, is present in this data; direct application of downstream analyses to the data may lead to biased results. Image harmonization methods seek to remove these batch effects and enable increased generalizability and reproducibility of downstream results. In this review, we describe and categorize current approaches in statistical and deep learning harmonization methods. We also describe current evaluation metrics used to assess harmonization methods and provide a standardized framework to evaluate newly-proposed methods for effective harmonization and preservation of biological information. Finally, we provide recommendations to end-users to advocate for more effective use of current methods and to methodologists to direct future efforts and accelerate development of the field.
Collapse
Affiliation(s)
- Fengling Hu
- Penn Statistics in Imaging and Visualization Endeavor (PennSIVE), Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, 423 Guardian Dr, Philadelphia, PA 19104, United States.
| | - Andrew A Chen
- Penn Statistics in Imaging and Visualization Endeavor (PennSIVE), Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, 423 Guardian Dr, Philadelphia, PA 19104, United States
| | - Hannah Horng
- Penn Statistics in Imaging and Visualization Endeavor (PennSIVE), Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, 423 Guardian Dr, Philadelphia, PA 19104, United States
| | - Vishnu Bashyam
- Center for Biomedical Image Computing and Analytics (CBICA), Perelman School of Medicine, United States
| | - Christos Davatzikos
- Center for Biomedical Image Computing and Analytics (CBICA), Perelman School of Medicine, United States
| | - Aaron Alexander-Bloch
- Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, United States; Penn-CHOP Lifespan Brain Institute, United States; Department of Child and Adolescent Psychiatry and Behavioral Science, Children's Hospital of Philadelphia, United States
| | - Mingyao Li
- Statistical Center for Single-Cell and Spatial Genomics, Perelman School of Medicine, University of Pennsylvania, United States
| | - Haochang Shou
- Penn Statistics in Imaging and Visualization Endeavor (PennSIVE), Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, 423 Guardian Dr, Philadelphia, PA 19104, United States; Center for Biomedical Image Computing and Analytics (CBICA), Perelman School of Medicine, United States
| | - Theodore D Satterthwaite
- Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, United States; Penn-CHOP Lifespan Brain Institute, United States; The Penn Lifespan Informatics and Neuroimaging Center, Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, United States
| | - Meichen Yu
- Indiana Alzheimer's Disease Research Center, Indiana University School of Medicine, United States
| | - Russell T Shinohara
- Penn Statistics in Imaging and Visualization Endeavor (PennSIVE), Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, 423 Guardian Dr, Philadelphia, PA 19104, United States; Center for Biomedical Image Computing and Analytics (CBICA), Perelman School of Medicine, United States
| |
Collapse
|
11
|
Haddad E, Pizzagalli F, Zhu AH, Bhatt RR, Islam T, Ba Gari I, Dixon D, Thomopoulos SI, Thompson PM, Jahanshad N. Multisite test-retest reliability and compatibility of brain metrics derived from FreeSurfer versions 7.1, 6.0, and 5.3. Hum Brain Mapp 2023; 44:1515-1532. [PMID: 36437735 PMCID: PMC9921222 DOI: 10.1002/hbm.26147] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Revised: 10/19/2022] [Accepted: 10/19/2022] [Indexed: 11/29/2022] Open
Abstract
Automatic neuroimaging processing tools provide convenient and systematic methods for extracting features from brain magnetic resonance imaging scans. One tool, FreeSurfer, provides an easy-to-use pipeline to extract cortical and subcortical morphometric measures. There have been over 25 stable releases of FreeSurfer, with different versions used across published works. The reliability and compatibility of regional morphometric metrics derived from the most recent version releases have yet to be empirically assessed. Here, we used test-retest data from three public data sets to determine within-version reliability and between-version compatibility across 42 regional outputs from FreeSurfer versions 7.1, 6.0, and 5.3. Cortical thickness from v7.1 was less compatible with that of older versions, particularly along the cingulate gyrus, where the lowest version compatibility was observed (intraclass correlation coefficient 0.37-0.61). Surface area of the temporal pole, frontal pole, and medial orbitofrontal cortex, also showed low to moderate version compatibility. We confirm low compatibility between v6.0 and v5.3 of pallidum and putamen volumes, while those from v7.1 were compatible with v6.0. Replication in an independent sample showed largely similar results for measures of surface area and subcortical volumes, but had lower overall regional thickness reliability and compatibility. Batch effect correction may adjust for some inter-version effects when most sites are run with one version, but results vary when more sites are run with different versions. Age associations in a quality controlled independent sample (N = 106) revealed version differences in results of downstream statistical analysis. We provide a reference to highlight the regional metrics that may yield recent version-related inconsistencies in published findings. An interactive viewer is provided at http://data.brainescience.org/Freesurfer_Reliability/.
Collapse
Affiliation(s)
- Elizabeth Haddad
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| | - Fabrizio Pizzagalli
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA.,Department of Neurosciences, University of Turin, Turin, Italy
| | - Alyssa H Zhu
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| | - Ravi R Bhatt
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| | - Tasfiya Islam
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| | - Iyad Ba Gari
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| | - Daniel Dixon
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| | - Sophia I Thomopoulos
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| | - Paul M Thompson
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| | - Neda Jahanshad
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, California, USA
| |
Collapse
|
12
|
Thomas AW, Ré C, Poldrack RA. Interpreting mental state decoding with deep learning models. Trends Cogn Sci 2022; 26:972-986. [PMID: 36223760 DOI: 10.1016/j.tics.2022.07.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2021] [Revised: 07/15/2022] [Accepted: 07/18/2022] [Indexed: 01/12/2023]
Abstract
In mental state decoding, researchers aim to identify the set of mental states (e.g., experiencing happiness or fear) that can be reliably identified from the activity patterns of a brain region (or network). Deep learning (DL) models are highly promising for mental state decoding because of their unmatched ability to learn versatile representations of complex data. However, their widespread application in mental state decoding is hindered by their lack of interpretability, difficulties in applying them to small datasets, and in ensuring their reproducibility and robustness. We recommend approaching these challenges by leveraging recent advances in explainable artificial intelligence (XAI) and transfer learning, and also provide recommendations on how to improve the reproducibility and robustness of DL models in mental state decoding.
Collapse
Affiliation(s)
- Armin W Thomas
- Stanford Data Science, Stanford University, Stanford, CA, USA; Department of Psychology, Stanford University, Stanford, CA, USA.
| | - Christopher Ré
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Russell A Poldrack
- Stanford Data Science, Stanford University, Stanford, CA, USA; Department of Psychology, Stanford University, Stanford, CA, USA
| |
Collapse
|
13
|
Horien C, Floris DL, Greene AS, Noble S, Rolison M, Tejavibulya L, O'Connor D, McPartland JC, Scheinost D, Chawarska K, Lake EMR, Constable RT. Functional Connectome-Based Predictive Modeling in Autism. Biol Psychiatry 2022; 92:626-642. [PMID: 35690495 PMCID: PMC10948028 DOI: 10.1016/j.biopsych.2022.04.008] [Citation(s) in RCA: 15] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/26/2021] [Revised: 04/14/2022] [Accepted: 04/17/2022] [Indexed: 01/08/2023]
Abstract
Autism is a heterogeneous neurodevelopmental condition, and functional magnetic resonance imaging-based studies have helped advance our understanding of its effects on brain network activity. We review how predictive modeling, using measures of functional connectivity and symptoms, has helped reveal key insights into this condition. We discuss how different prediction frameworks can further our understanding of the brain-based features that underlie complex autism symptomatology and consider how predictive models may be used in clinical settings. Throughout, we highlight aspects of study interpretation, such as data decay and sampling biases, that require consideration within the context of this condition. We close by suggesting exciting future directions for predictive modeling in autism.
Collapse
Affiliation(s)
- Corey Horien
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, Connecticut; MD-PhD Program, Yale School of Medicine, New Haven, Connecticut.
| | - Dorothea L Floris
- Methods of Plasticity Research, Department of Psychology, University of Zürich, Zurich, Switzerland; Donders Center for Brain, Cognition and Behavior, Radboud University Nijmegen, Nijmegen, The Netherlands
| | - Abigail S Greene
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, Connecticut; MD-PhD Program, Yale School of Medicine, New Haven, Connecticut
| | - Stephanie Noble
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut
| | - Max Rolison
- Yale Child Study Center, New Haven, Connecticut
| | - Link Tejavibulya
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, Connecticut
| | - David O'Connor
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut
| | - James C McPartland
- Department of Psychology, Yale University, New Haven, Connecticut; Yale Child Study Center, New Haven, Connecticut
| | - Dustin Scheinost
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, Connecticut; Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut; Department of Biomedical Engineering, Yale University, New Haven, Connecticut; Department of Statistics and Data Science, Yale University, New Haven, Connecticut; Yale Child Study Center, New Haven, Connecticut
| | - Katarzyna Chawarska
- Department of Pediatrics, Yale School of Medicine, New Haven, Connecticut; Department of Statistics and Data Science, Yale University, New Haven, Connecticut; Yale Child Study Center, New Haven, Connecticut
| | - Evelyn M R Lake
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut
| | - R Todd Constable
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, Connecticut; Department of Neurosurgery, Yale School of Medicine, New Haven, Connecticut; Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut; Department of Biomedical Engineering, Yale University, New Haven, Connecticut.
| |
Collapse
|
14
|
Farruggia MC, Pellegrino R, Scheinost D. Functional Connectivity of the Chemosenses: A Review. Front Syst Neurosci 2022; 16:865929. [PMID: 35813269 PMCID: PMC9257046 DOI: 10.3389/fnsys.2022.865929] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2022] [Accepted: 05/05/2022] [Indexed: 01/01/2023] Open
Abstract
Functional connectivity approaches have long been used in cognitive neuroscience to establish pathways of communication between and among brain regions. However, the use of these analyses to better understand how the brain processes chemosensory information remains nascent. In this review, we conduct a literature search of all functional connectivity papers of olfaction, gustation, and chemesthesis, with 103 articles discovered in total. These publications largely use approaches of seed-based functional connectivity and psychophysiological interactions, as well as effective connectivity approaches such as Granger Causality, Dynamic Causal Modeling, and Structural Equation Modeling. Regardless of modality, studies largely focus on elucidating neural correlates of stimulus qualities such as identity, pleasantness, and intensity, with task-based paradigms most frequently implemented. We call for further "model free" or data-driven approaches in predictive modeling to craft brain-behavior relationships that are free from a priori hypotheses and not solely based on potentially irreproducible literature. Moreover, we note a relative dearth of resting-state literature, which could be used to better understand chemosensory networks with less influence from motion artifacts induced via gustatory or olfactory paradigms. Finally, we note a lack of genomics data, which could clarify individual and heritable differences in chemosensory perception.
Collapse
Affiliation(s)
- Michael C. Farruggia
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, United States,*Correspondence: Michael C. Farruggia,
| | | | - Dustin Scheinost
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, United States,Child Study Center, Yale School of Medicine, New Haven, CT, United States,Department of Biomedical Engineering, Yale School of Engineering and Applied Science, New Haven, CT, United States,Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, United States,Wu Tsai Institute, Yale University, New Haven, CT, United States
| |
Collapse
|
15
|
Yeung AWK, More S, Wu J, Eickhoff SB. Reporting details of neuroimaging studies on individual traits prediction: A literature survey. Neuroimage 2022; 256:119275. [PMID: 35513295 DOI: 10.1016/j.neuroimage.2022.119275] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Revised: 04/27/2022] [Accepted: 04/29/2022] [Indexed: 11/17/2022] Open
Abstract
Using machine-learning tools to predict individual phenotypes from neuroimaging data is one of the most promising and hence dynamic fields in systems neuroscience. Here, we perform a literature survey of the rapidly work on phenotype prediction in healthy subjects or general population to sketch out the current state and ongoing developments in terms of data, analysis methods and reporting. Excluding papers on age-prediction and clinical applications, which form a distinct literature, we identified a total 108 papers published since 2007. In these, memory, fluid intelligence and attention were most common phenotypes to be predicted, which resonates with the observation that roughly a quarter of the papers used data from the Human Connectome Project, even though another half recruited their own cohort. Sample size (in terms of training and external test sets) and prediction accuracy (from internal and external validation respectively) did not show significant temporal trends. Prediction accuracy was negatively correlated with sample size of the training set, but not the external test set. While known to be optimistic, leave-one-out cross-validation (LOO CV) was the prevalent strategy for model validation (n = 48). Meanwhile, 27 studies used external validation with external test set. Both numbers showed no significant temporal trends. The most popular learning algorithm was connectome-based predictive modeling introduced by the Yale team. Other common learning algorithms were linear regression, relevance vector regression (RVR), support vector regression (SVR), least absolute shrinkage and selection operator (LASSO), and elastic net. Meanwhile, the amount of data from self-recruiting studies (but not studies using open, shared dataset) was positively correlated with internal validation prediction accuracy. At the same time, self-recruiting studies also reported a significantly higher internal validation prediction accuracy than those using open, shared datasets. Data type and participant age did not significantly influence prediction accuracy. Confound control also did not influence prediction accuracy after adjusted for other factors. To conclude, most of the current literature is probably quite optimistic with internal validation using LOO CV. More efforts should be made to encourage the use of external validation with external test sets to further improve generalizability of the models.
Collapse
Affiliation(s)
- Andy Wai Kan Yeung
- Oral and Maxillofacial Radiology, Applied Oral Sciences and Community Dental Care, Faculty of Dentistry, The University of Hong Kong, Hong Kong, China.
| | - Shammi More
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Centre Jülich, Jülich, Germany; Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Jianxiao Wu
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Centre Jülich, Jülich, Germany; Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Simon B Eickhoff
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Centre Jülich, Jülich, Germany; Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany.
| |
Collapse
|
16
|
Williams B, Roesch E, Christakou A. Systematic validation of an automated thalamic parcellation technique using anatomical data at 3T. Neuroimage 2022; 258:119340. [DOI: 10.1016/j.neuroimage.2022.119340] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2021] [Revised: 05/20/2022] [Accepted: 05/28/2022] [Indexed: 11/24/2022] Open
|
17
|
Horien C, Lee K, Westwater ML, Noble S, Tejavibulya L, Kayani T, Constable RT, Scheinost D. A protocol for working with open-source neuroimaging datasets. STAR Protoc 2022; 3:101077. [PMID: 35036958 PMCID: PMC8749295 DOI: 10.1016/j.xpro.2021.101077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023] Open
Abstract
Large, publicly available neuroimaging datasets are becoming increasingly common, but their use presents challenges because of insufficient knowledge of the tool options for data processing and proper data organization. Here, we describe a protocol to lessen these barriers. We describe the steps for the search and download of the open-source dataset. We detail the steps for proper data management and practical guidelines for data analysis. Finally, we give instructions for data and result sharing on public repositories and preprint services. For complete details on the use and execution of this profile, please refer to Horien et al. (2021).
Collapse
Affiliation(s)
- Corey Horien
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
- MD-PhD Program, Yale School of Medicine, New Haven, CT 06510, USA
| | - Kangjoo Lee
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
| | - Margaret L. Westwater
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
| | - Stephanie Noble
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
| | - Link Tejavibulya
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
| | - Teimur Kayani
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
| | - R. Todd Constable
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
- Department of Neurosurgery, Yale School of Medicine, New Haven, CT 06510, USA
| | - Dustin Scheinost
- Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven, CT 06510, USA
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, CT 06510, USA
- Yale Child Study Center, New Haven, CT 06510, USA
- Department of Statistics and Data Science, Yale University, New Haven, CT 06520, USA
| |
Collapse
|
18
|
Wagner AS, Waite LK, Wierzba M, Hoffstaedter F, Waite AQ, Poldrack B, Eickhoff SB, Hanke M. FAIRly big: A framework for computationally reproducible processing of large-scale data. Sci Data 2022; 9:80. [PMID: 35277501 PMCID: PMC8917149 DOI: 10.1038/s41597-022-01163-2] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2021] [Accepted: 02/11/2022] [Indexed: 11/30/2022] Open
Abstract
Large-scale datasets present unique opportunities to perform scientific investigations with unprecedented breadth. However, they also pose considerable challenges for the findability, accessibility, interoperability, and reusability (FAIR) of research outcomes due to infrastructure limitations, data usage constraints, or software license restrictions. Here we introduce a DataLad-based, domain-agnostic framework suitable for reproducible data processing in compliance with open science mandates. The framework attempts to minimize platform idiosyncrasies and performance-related complexities. It affords the capture of machine-actionable computational provenance records that can be used to retrace and verify the origins of research outcomes, as well as be re-executed independent of the original computing infrastructure. We demonstrate the framework's performance using two showcases: one highlighting data sharing and transparency (using the studyforrest.org dataset) and another highlighting scalability (using the largest public brain imaging dataset available: the UK Biobank dataset).
Collapse
Affiliation(s)
- Adina S Wagner
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany.
| | - Laura K Waite
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany
| | - Małgorzata Wierzba
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany
- Laboratory of Brain Imaging, Nencki Institute of Experimental Biology, Polish Academy of Sciences, Warsaw, Poland
| | - Felix Hoffstaedter
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany
| | - Alexander Q Waite
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany
| | - Benjamin Poldrack
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany
| | - Simon B Eickhoff
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Michael Hanke
- Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Center Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| |
Collapse
|
19
|
Pringle C, Kilday JP, Kamaly-Asl I, Stivaros SM. The role of artificial intelligence in paediatric neuroradiology. Pediatr Radiol 2022; 52:2159-2172. [PMID: 35347371 PMCID: PMC9537195 DOI: 10.1007/s00247-022-05322-w] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/07/2021] [Revised: 08/22/2021] [Accepted: 02/11/2022] [Indexed: 01/17/2023]
Abstract
Imaging plays a fundamental role in the managing childhood neurologic, neurosurgical and neuro-oncological disease. Employing multi-parametric MRI techniques, such as spectroscopy and diffusion- and perfusion-weighted imaging, to the radiophenotyping of neuroradiologic conditions is becoming increasingly prevalent, particularly with radiogenomic analyses correlating imaging characteristics with molecular biomarkers of disease. However, integration into routine clinical practice remains elusive. With modern multi-parametric MRI now providing additional data beyond anatomy, informing on histology, biology and physiology, such metric-rich information can present as information overload to the treating radiologist and, as such, information relevant to an individual case can become lost. Artificial intelligence techniques are capable of modelling the vast radiologic, biological and clinical datasets that accompany childhood neurologic disease, such that this information can become incorporated in upfront prognostic modelling systems, with artificial intelligence techniques providing a plausible approach to this solution. This review examines machine learning approaches than can be used to underpin such artificial intelligence applications, with exemplars for each machine learning approach from the world literature. Then, within the specific use case of paediatric neuro-oncology, we examine the potential future contribution for such artificial intelligence machine learning techniques to offer solutions for patient care in the form of decision support systems, potentially enabling personalised medicine within this domain of paediatric radiologic practice.
Collapse
Affiliation(s)
- Catherine Pringle
- Children’s Brain Tumour Research Network (CBTRN), Royal Manchester Children’s Hospital, Manchester, UK ,Division of Informatics, Imaging, and Data Sciences, School of Health Sciences, Faculty of Biology, Medicine, and Health, University of Manchester, Manchester, UK
| | - John-Paul Kilday
- Children’s Brain Tumour Research Network (CBTRN), Royal Manchester Children’s Hospital, Manchester, UK ,The Centre for Paediatric, Teenage and Young Adult Cancer, Institute of Cancer Sciences, University of Manchester, Manchester, UK
| | - Ian Kamaly-Asl
- Children’s Brain Tumour Research Network (CBTRN), Royal Manchester Children’s Hospital, Manchester, UK ,The Centre for Paediatric, Teenage and Young Adult Cancer, Institute of Cancer Sciences, University of Manchester, Manchester, UK
| | - Stavros Michael Stivaros
- Division of Informatics, Imaging, and Data Sciences, School of Health Sciences, Faculty of Biology, Medicine, and Health, University of Manchester, Manchester, UK. .,Department of Paediatric Radiology, Royal Manchester Children's Hospital, Central Manchester University Hospitals NHS Foundation Trust, Oxford Road, Manchester, M13 9WL, UK. .,The Geoffrey Jefferson Brain Research Centre, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
| |
Collapse
|
20
|
Korom M, Camacho MC, Filippi CA, Licandro R, Moore LA, Dufford A, Zöllei L, Graham AM, Spann M, Howell B, Shultz S, Scheinost D. Dear reviewers: Responses to common reviewer critiques about infant neuroimaging studies. Dev Cogn Neurosci 2021; 53:101055. [PMID: 34974250 PMCID: PMC8733260 DOI: 10.1016/j.dcn.2021.101055] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2021] [Revised: 11/28/2021] [Accepted: 12/26/2021] [Indexed: 01/07/2023] Open
Abstract
The field of adult neuroimaging relies on well-established principles in research design, imaging sequences, processing pipelines, as well as safety and data collection protocols. The field of infant magnetic resonance imaging, by comparison, is a young field with tremendous scientific potential but continuously evolving standards. The present article aims to initiate a constructive dialog between researchers who grapple with the challenges and inherent limitations of a nascent field and reviewers who evaluate their work. We address 20 questions that researchers commonly receive from research ethics boards, grant, and manuscript reviewers related to infant neuroimaging data collection, safety protocols, study planning, imaging sequences, decisions related to software and hardware, and data processing and sharing, while acknowledging both the accomplishments of the field and areas of much needed future advancements. This article reflects the cumulative knowledge of experts in the FIT’NG community and can act as a resource for both researchers and reviewers alike seeking a deeper understanding of the standards and tradeoffs involved in infant neuroimaging. The field of infant MRI is young with evolving standards. We address 20 questions that researchers commonly receive reviewers. These come from research ethics boards, grant, and manuscript reviewers. This article reflects the cumulative knowledge of experts in the FIT’NG community.
Collapse
Affiliation(s)
- Marta Korom
- Department of Psychological and Brain Sciences, University of Delaware, Newark, DE, USA.
| | - M Catalina Camacho
- Division of Biology and Biomedical Sciences (Neurosciences), Washington University School of Medicine, St. Louis, MO, USA.
| | - Courtney A Filippi
- Emotion and Development Branch, National Institute of Mental Health, Bethesda, MD, USA
| | - Roxane Licandro
- Institute of Visual Computing and Human-Centered Technology, Computer Vision Lab, TU Wien, Vienna, Austria; Department of Biomedical Imaging and Image-guided Therapy, Computational Imaging Research, Medical University of Vienna, Vienna, Austria
| | - Lucille A Moore
- Department of Psychiatry, Oregon Health and Science University, Portland, OR, USA
| | - Alexander Dufford
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
| | - Lilla Zöllei
- A.A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Boston, MA, USA
| | - Alice M Graham
- Department of Psychiatry, Oregon Health and Science University, Portland, OR, USA
| | - Marisa Spann
- Department of Psychiatry, Columbia University Irving Medical Center, New York, NY, USA
| | - Brittany Howell
- Fralin Biomedical Research Institute at Virginia Tech Carilion, Department of Human Development and Family Science, Virginia Polytechnic Institute and State University, Roanoke, VA, USA
| | | | - Sarah Shultz
- Division of Autism & Department of Pediatrics, Emory University School of Medicine, Atlanta, GA, USA; Marcus Autism Center, Children's Healthcare of Atlanta, Atlanta, GA, USA.
| | - Dustin Scheinost
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA.
| |
Collapse
|
21
|
Zamani Esfahlani F, Jo Y, Puxeddu MG, Merritt H, Tanner JC, Greenwell S, Patel R, Faskowitz J, Betzel RF. Modularity maximization as a flexible and generic framework for brain network exploratory analysis. Neuroimage 2021; 244:118607. [PMID: 34607022 DOI: 10.1016/j.neuroimage.2021.118607] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2021] [Revised: 09/03/2021] [Accepted: 09/20/2021] [Indexed: 11/28/2022] Open
Abstract
The modular structure of brain networks supports specialized information processing, complex dynamics, and cost-efficient spatial embedding. Inter-individual variation in modular structure has been linked to differences in performance, disease, and development. There exist many data-driven methods for detecting and comparing modular structure, the most popular of which is modularity maximization. Although modularity maximization is a general framework that can be modified and reparamaterized to address domain-specific research questions, its application to neuroscientific datasets has, thus far, been narrow. Here, we highlight several strategies in which the "out-of-the-box" version of modularity maximization can be extended to address questions specific to neuroscience. First, we present approaches for detecting "space-independent" modules and for applying modularity maximization to signed matrices. Next, we show that the modularity maximization frame is well-suited for detecting task- and condition-specific modules. Finally, we highlight the role of multi-layer models in detecting and tracking modules across time, tasks, subjects, and modalities. In summary, modularity maximization is a flexible and general framework that can be adapted to detect modular structure resulting from a wide range of hypotheses. This article highlights multiple frontiers for future research and applications.
Collapse
Affiliation(s)
- Farnaz Zamani Esfahlani
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, United States
| | - Youngheun Jo
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, United States
| | - Maria Grazia Puxeddu
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, United States; Department of Computer, Control and Management Engineering "Antonio Ruberti", Sapienza University of Rome, Rome 00185, Italy; IRCCS Fondazione Santa Lucia, Rome 00179, Italy
| | - Haily Merritt
- Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN 47405, United States; Cognitive Science Program, Indiana University, Bloomington, IN 47405, United States
| | - Jacob C Tanner
- Luddy School of Informatics, Computing, and Engineering, Indiana University, Bloomington, IN 47405, United States; Cognitive Science Program, Indiana University, Bloomington, IN 47405, United States
| | - Sarah Greenwell
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, United States
| | - Riya Patel
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, United States
| | - Joshua Faskowitz
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, United States; Program in Neuroscience, Indiana University, Bloomington, IN 47405, United States
| | - Richard F Betzel
- Department of Psychological and Brain Sciences, Indiana University, Bloomington, IN 47405, United States; Cognitive Science Program, Indiana University, Bloomington, IN 47405, United States; Program in Neuroscience, Indiana University, Bloomington, IN 47405, United States; Network Science Institute, Indiana University, Bloomington, IN 47405, United States.
| |
Collapse
|
22
|
Dufford AJ, Spann M, Scheinost D. How prenatal exposures shape the infant brain: Insights from infant neuroimaging studies. Neurosci Biobehav Rev 2021; 131:47-58. [PMID: 34536461 DOI: 10.1016/j.neubiorev.2021.09.017] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2021] [Revised: 07/30/2021] [Accepted: 09/12/2021] [Indexed: 10/20/2022]
Abstract
Brain development during the prenatal period is rapid and unparalleled by any other time during development. Biological systems undergoing rapid development are at higher risk for disorganizing influences. Therefore, certain prenatal exposures impact brain development, increasing risk for negative neurodevelopmental outcome. While prenatal exposures have been associated with cognitive and behavioral outcomes later in life, the underlying macroscopic brain pathways remain unclear. Here, we review magnetic resonance imaging (MRI) studies investigating the association between prenatal exposures and infant brain development focusing on prenatal exposures via maternal physical health factors, maternal mental health factors, and maternal drug and medication use. Further, we discuss the need for studies to consider multiple prenatal exposures in parallel and suggest future directions for this body of research.
Collapse
Affiliation(s)
| | - Marisa Spann
- Columbia University Irving Medical Center, 622 West 168th Street, New York, NY, 10032, USA
| | - Dustin Scheinost
- Child Study Center, Yale School of Medicine, New Haven, CT, USA; Department of Radiology and Biomedical Imaging, Yale School of Medicine, USA; Department of Statistics and Data Science, Yale University, New Haven, CT, USA; Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| |
Collapse
|
23
|
Large, open datasets for human connectomics research: Considerations for reproducible and responsible data use. Neuroimage 2021; 244:118579. [PMID: 34536537 DOI: 10.1016/j.neuroimage.2021.118579] [Citation(s) in RCA: 24] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2021] [Revised: 08/27/2021] [Accepted: 09/14/2021] [Indexed: 12/19/2022] Open
Abstract
Large, open datasets have emerged as important resources in the field of human connectomics. In this review, the evolution of data sharing involving magnetic resonance imaging is described. A summary of the challenges and progress in conducting reproducible data analyses is provided, including description of recent progress made in the development of community guidelines and recommendations, software and data management tools, and initiatives to enhance training and education. Finally, this review concludes with a discussion of ethical conduct relevant to analyses of large, open datasets and a researcher's responsibility to prevent further stigmatization of historically marginalized racial and ethnic groups. Moving forward, future work should include an enhanced emphasis on the social determinants of health, which may further contextualize findings among diverse population-based samples. Leveraging the progress to date and guided by interdisciplinary collaborations, the future of connectomics promises to be an impressive era of innovative research, yielding a more inclusive understanding of brain structure and function.
Collapse
|
24
|
Virtual Connectomic Datasets in Alzheimer's Disease and Aging Using Whole-Brain Network Dynamics Modelling. eNeuro 2021; 8:ENEURO.0475-20.2021. [PMID: 34045210 PMCID: PMC8260273 DOI: 10.1523/eneuro.0475-20.2021] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2020] [Revised: 03/08/2021] [Accepted: 04/12/2021] [Indexed: 12/18/2022] Open
Abstract
Large neuroimaging datasets, including information about structural connectivity (SC) and functional connectivity (FC), play an increasingly important role in clinical research, where they guide the design of algorithms for automated stratification, diagnosis or prediction. A major obstacle is, however, the problem of missing features [e.g., lack of concurrent DTI SC and resting-state functional magnetic resonance imaging (rsfMRI) FC measurements for many of the subjects]. We propose here to address the missing connectivity features problem by introducing strategies based on computational whole-brain network modeling. Using two datasets, the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset and a healthy aging dataset, for proof-of-concept, we demonstrate the feasibility of virtual data completion (i.e., inferring “virtual FC” from empirical SC or “virtual SC” from empirical FC), by using self-consistent simulations of linear and nonlinear brain network models. Furthermore, by performing machine learning classification (to separate age classes or control from patient subjects), we show that algorithms trained on virtual connectomes achieve discrimination performance comparable to when trained on actual empirical data; similarly, algorithms trained on virtual connectomes can be used to successfully classify novel empirical connectomes. Completion algorithms can be combined and reiterated to generate realistic surrogate connectivity matrices in arbitrarily large number, opening the way to the generation of virtual connectomic datasets with network connectivity information comparable to the one of the original data.
Collapse
|
25
|
Barron DS, Baker JT, Budde KS, Bzdok D, Eickhoff SB, Friston KJ, Fox PT, Geha P, Heisig S, Holmes A, Onnela JP, Powers A, Silbersweig D, Krystal JH. Decision Models and Technology Can Help Psychiatry Develop Biomarkers. Front Psychiatry 2021; 12:706655. [PMID: 34566711 PMCID: PMC8458705 DOI: 10.3389/fpsyt.2021.706655] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/07/2021] [Accepted: 08/02/2021] [Indexed: 12/02/2022] Open
Abstract
Why is psychiatry unable to define clinically useful biomarkers? We explore this question from the vantage of data and decision science and consider biomarkers as a form of phenotypic data that resolves a well-defined clinical decision. We introduce a framework that systematizes different forms of phenotypic data and further introduce the concept of decision model to describe the strategies a clinician uses to seek out, combine, and act on clinical data. Though many medical specialties rely on quantitative clinical data and operationalized decision models, we observe that, in psychiatry, clinical data are gathered and used in idiosyncratic decision models that exist solely in the clinician's mind and therefore are outside empirical evaluation. This, we argue, is a fundamental reason why psychiatry is unable to define clinically useful biomarkers: because psychiatry does not currently quantify clinical data, decision models cannot be operationalized and, in the absence of an operationalized decision model, it is impossible to define how a biomarker might be of use. Here, psychiatry might benefit from digital technologies that have recently emerged specifically to quantify clinically relevant facets of human behavior. We propose that digital tools might help psychiatry in two ways: first, by quantifying data already present in the standard clinical interaction and by allowing decision models to be operationalized and evaluated; second, by testing whether new forms of data might have value within an operationalized decision model. We reference successes from other medical specialties to illustrate how quantitative data and operationalized decision models improve patient care.
Collapse
Affiliation(s)
- Daniel S Barron
- Department of Psychiatry, Harvard Medical School, Brigham and Women's Hospital, Boston, MA, United States.,Department of Anesthesiology and Pain Medicine, Harvard Medical School, Brigham and Women's Hospital, Boston, MA, United States.,Department of Psychiatry, Yale University, New Haven, CT, United States.,Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, WA, United States
| | - Justin T Baker
- Department of Psychiatry, Harvard Medical School, McLean Hospital, Belmont, MA, United States
| | - Kristin S Budde
- Department of Psychiatry, Harvard Medical School, Brigham and Women's Hospital, Boston, MA, United States.,Department of Psychiatry, Yale University, New Haven, CT, United States.,Department of Psychiatry, University of Washington, Seattle, WA, United States
| | - Danilo Bzdok
- Department of Biomedical Engineering, Faculty of Medicine, McConnell Brain Imaging Centre (BIC), Montreal Neurological Institute (MNI), McGill University, Montreal, QC, Canada.,Mila-Quebec Artificial Intelligence Institute, Montreal, QC, Canada
| | - Simon B Eickhoff
- Medical Faculty, Institute of Systems Neuroscience, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Karl J Friston
- The Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, London, United Kingdom
| | - Peter T Fox
- Research Imaging Institute, University of Texas Health, San Antonio, TX, United States
| | - Paul Geha
- Departments of Psychiatry, University of Rochester Medical Center, Rochester, NY, United States
| | - Stephen Heisig
- T.J. Watson IBM Research Laboratory, Yorktown Heights, NY, United States.,Department of Neurology, Icahn School of Medicine, New York, NY, United States
| | - Avram Holmes
- Department of Psychiatry, Yale University, New Haven, CT, United States.,Department of Psychology, Yale University, New Haven, CT, United States
| | - Jukka-Pekka Onnela
- Department of Biostatistics, T. H. Chan School of Public Health, Harvard University, Boston, MA, United States
| | - Albert Powers
- Department of Psychiatry, Yale University, New Haven, CT, United States
| | - David Silbersweig
- Department of Psychiatry, Harvard Medical School, Brigham and Women's Hospital, Boston, MA, United States
| | - John H Krystal
- Department of Psychiatry, Yale University, New Haven, CT, United States
| |
Collapse
|