1
|
Hearne LJ, Yeo BTT, Webb L, Zalesky A, Fitzgerald PB, Murphy OW, Tian Y, Breakspear M, Hall CV, Choi S, Kim M, Kwon JS, Cocchi L. Distinct cognitive and functional connectivity features from healthy cohorts can identify clinical obsessive-compulsive disorder. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.09.02.24312960. [PMID: 39281735 PMCID: PMC11398446 DOI: 10.1101/2024.09.02.24312960] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 09/18/2024]
Abstract
Improving diagnostic accuracy of obsessive-compulsive disorder (OCD) using models of brain imaging data is a key goal of the field, but this objective is challenging due to the limited size and phenotypic depth of clinical datasets. Leveraging the phenotypic diversity in large non-clinical datasets such as the UK Biobank (UKBB), offers a potential solution to this problem. Nevertheless, it remains unclear whether classification models trained on non-clinical populations will generalise to individuals with clinical OCD. This question is also relevant for the conceptualisation of OCD; specifically, whether the symptomology of OCD exists on a continuum from normal to pathological. Here, we examined a recently published "meta-matching" model trained on functional connectivity data from five large normative datasets (N=45,507) to predict cognitive, health and demographic variables. Specifically, we tested whether this model could classify OCD status in three independent clinical datasets (N=345). We found that the model could identify out-of-sample OCD individuals. Notably, the most predictive functional connectivity features mapped onto known cortico-striatal abnormalities in OCD and correlated with genetic brain expression maps previously implicated in the disorder. Further, the meta-matching model relied upon estimates of cognitive functions, such as cognitive flexibility and inhibition, to successfully predict OCD. These findings suggest that variability in non-clinical brain and behavioural features can discriminate clinical OCD status. These results support a dimensional and transdiagnostic conceptualisation of the brain and behavioural basis of OCD, with implications for research approaches and treatment targets.
Collapse
Affiliation(s)
- Luke J Hearne
- QIMR Berghofer Medical Research Institute, Brisbane, QLD, Australia
| | - B T Thomas Yeo
- Centre for Sleep & Cognition & Centre for Translational Magnetic Resonance Research, Yong Loo Lin School of Medicine, Singapore, National University of Singapore, Singapore, Singapore
- Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Singapore
- N.1 Institute for Health, National University of Singapore, Singapore, Singapore
- Integrative Sciences and Engineering Programme (ISEP), National University of Singapore, Singapore, Singapore
- Department of Medicine, Human Potential Translational Research Programme & Institute for Digital Medicine (WisDM), Yong Loo Lin School of Medicine, National University of Singapore, Singapore
- Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, USA
| | - Lachlan Webb
- QIMR Berghofer Medical Research Institute, Brisbane, QLD, Australia
| | - Andrew Zalesky
- Melbourne Neuropsychiatry Centre, Department of Psychiatry, The University of Melbourne & Melbourne Health, Melbourne, Australia
| | - Paul B Fitzgerald
- School of Medicine and Psychology, Australian National University, Canberra, Australia
| | - Oscar W Murphy
- Central Clinical School, Monash University, Clayton, Australia
- Bionics Institute, East Melbourne, Australia
| | - Ye Tian
- Melbourne Neuropsychiatry Centre, Department of Psychiatry, The University of Melbourne & Melbourne Health, Melbourne, Australia
| | - Michael Breakspear
- School of Psychological Sciences, College of Engineering Science and Environment, University of Newcastle, Callaghan, Australia
- School of Medicine and Public Health, College of Health and Medicine, University of Newcastle, Callaghan, Australia
- Program of Neuromodulation, Hunter Medical Research Institute, New Lambton, Australia
| | - Caitlin V Hall
- QIMR Berghofer Medical Research Institute, Brisbane, QLD, Australia
| | - Sunah Choi
- Department of Brain and Cognitive Sciences, Seoul National University College of Natural Sciences, Republic of Korea
| | - Minah Kim
- Department of Neuropsychiatry, Seoul National University Hospital, Republic of Korea
- Department of Psychiatry, Seoul National University College of Medicine, Republic of Korea
| | - Jun Soo Kwon
- Department of Brain and Cognitive Sciences, Seoul National University College of Natural Sciences, Republic of Korea
- Department of Neuropsychiatry, Seoul National University Hospital, Republic of Korea
- Department of Psychiatry, Seoul National University College of Medicine, Republic of Korea
| | - Luca Cocchi
- QIMR Berghofer Medical Research Institute, Brisbane, QLD, Australia
- Faculty of Medicine, School of Biomedical Sciences, University of Queensland, Brisbane, Australia
| |
Collapse
|
2
|
Kincses B, Forkmann K, Schlitt F, Jan Pawlik R, Schmidt K, Timmann D, Elsenbruch S, Wiech K, Bingel U, Spisak T. An externally validated resting-state brain connectivity signature of pain-related learning. Commun Biol 2024; 7:875. [PMID: 39020002 PMCID: PMC11255216 DOI: 10.1038/s42003-024-06574-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Accepted: 07/10/2024] [Indexed: 07/19/2024] Open
Abstract
Pain can be conceptualized as a precision signal for reinforcement learning in the brain and alterations in these processes are a hallmark of chronic pain conditions. Investigating individual differences in pain-related learning therefore holds important clinical and translational relevance. Here, we developed and externally validated a novel resting-state brain connectivity-based predictive model of pain-related learning. The pre-registered external validation indicates that the proposed model explains 8-12% of the inter-individual variance in pain-related learning. Model predictions are driven by connections of the amygdala, posterior insula, sensorimotor, frontoparietal, and cerebellar regions, outlining a network commonly described in aversive learning and pain. We propose the resulting model as a robust and highly accessible biomarker candidate for clinical and translational pain research, with promising implications for personalized treatment approaches and with a high potential to advance our understanding of the neural mechanisms of pain-related learning.
Collapse
Affiliation(s)
- Balint Kincses
- Department of Neurology, Center for Translational Neuro- and Behavioral Sciences, University Medicine Essen, Essen, Germany.
- Institute for Diagnostic and Interventional Radiology and Neuroradiology, University Medicine Essen, Essen, Germany.
| | - Katarina Forkmann
- Department of Neurology, Center for Translational Neuro- and Behavioral Sciences, University Medicine Essen, Essen, Germany
| | - Frederik Schlitt
- Department of Neurology, Center for Translational Neuro- and Behavioral Sciences, University Medicine Essen, Essen, Germany
| | - Robert Jan Pawlik
- Department of Medical Psychology and Medical Sociology, Faculty of Medicine, Ruhr University Bochum, Bochum, Germany
| | - Katharina Schmidt
- Department of Neurology, Center for Translational Neuro- and Behavioral Sciences, University Medicine Essen, Essen, Germany
| | - Dagmar Timmann
- Department of Neurology, Center for Translational Neuro- and Behavioral Sciences, University Medicine Essen, Essen, Germany
| | - Sigrid Elsenbruch
- Department of Neurology, Center for Translational Neuro- and Behavioral Sciences, University Medicine Essen, Essen, Germany
- Department of Medical Psychology and Medical Sociology, Faculty of Medicine, Ruhr University Bochum, Bochum, Germany
| | - Katja Wiech
- Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Ulrike Bingel
- Department of Neurology, Center for Translational Neuro- and Behavioral Sciences, University Medicine Essen, Essen, Germany
| | - Tamas Spisak
- Department of Neurology, Center for Translational Neuro- and Behavioral Sciences, University Medicine Essen, Essen, Germany
- Institute for Diagnostic and Interventional Radiology and Neuroradiology, University Medicine Essen, Essen, Germany
| |
Collapse
|
3
|
Rosenblatt M, Tejavibulya L, Jiang R, Noble S, Scheinost D. Data leakage inflates prediction performance in connectome-based machine learning models. Nat Commun 2024; 15:1829. [PMID: 38418819 PMCID: PMC10901797 DOI: 10.1038/s41467-024-46150-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Accepted: 02/15/2024] [Indexed: 03/02/2024] Open
Abstract
Predictive modeling is a central technique in neuroimaging to identify brain-behavior relationships and test their generalizability to unseen data. However, data leakage undermines the validity of predictive models by breaching the separation between training and test data. Leakage is always an incorrect practice but still pervasive in machine learning. Understanding its effects on neuroimaging predictive models can inform how leakage affects existing literature. Here, we investigate the effects of five forms of leakage-involving feature selection, covariate correction, and dependence between subjects-on functional and structural connectome-based machine learning models across four datasets and three phenotypes. Leakage via feature selection and repeated subjects drastically inflates prediction performance, whereas other forms of leakage have minor effects. Furthermore, small datasets exacerbate the effects of leakage. Overall, our results illustrate the variable effects of leakage and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.
Collapse
Affiliation(s)
- Matthew Rosenblatt
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA.
| | - Link Tejavibulya
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
| | - Rongtao Jiang
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
| | - Stephanie Noble
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
- Department of Bioengineering, Northeastern University, Boston, MA, USA
- Department of Psychology, Northeastern University, Boston, MA, USA
| | - Dustin Scheinost
- Department of Biomedical Engineering, Yale University, New Haven, CT, USA
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, USA
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA
- Child Study Center, Yale School of Medicine, New Haven, CT, USA
- Department of Statistics & Data Science, Yale University, New Haven, CT, USA
| |
Collapse
|
4
|
Badrulhisham F, Pogatzki-Zahn E, Segelcke D, Spisak T, Vollert J. Machine learning and artificial intelligence in neuroscience: A primer for researchers. Brain Behav Immun 2024; 115:470-479. [PMID: 37972877 DOI: 10.1016/j.bbi.2023.11.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Revised: 10/16/2023] [Accepted: 11/08/2023] [Indexed: 11/19/2023] Open
Abstract
Artificial intelligence (AI) is often used to describe the automation of complex tasks that we would attribute intelligence to. Machine learning (ML) is commonly understood as a set of methods used to develop an AI. Both have seen a recent boom in usage, both in scientific and commercial fields. For the scientific community, ML can solve bottle necks created by complex, multi-dimensional data generated, for example, by functional brain imaging or *omics approaches. ML can here identify patterns that could not have been found using traditional statistic approaches. However, ML comes with serious limitations that need to be kept in mind: their tendency to optimise solutions for the input data means it is of crucial importance to externally validate any findings before considering them more than a hypothesis. Their black-box nature implies that their decisions usually cannot be understood, which renders their use in medical decision making problematic and can lead to ethical issues. Here, we present an introduction for the curious to the field of ML/AI. We explain the principles as commonly used methods as well as recent methodological advancements before we discuss risks and what we see as future directions of the field. Finally, we show practical examples of neuroscience to illustrate the use and limitations of ML.
Collapse
Affiliation(s)
| | - Esther Pogatzki-Zahn
- Department of Anaesthesiology, Intensive Care and Pain Medicine, University Hospital Muenster, Muenster, Germany
| | - Daniel Segelcke
- Department of Anaesthesiology, Intensive Care and Pain Medicine, University Hospital Muenster, Muenster, Germany
| | - Tamas Spisak
- Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Medicine Essen, Essen, Germany; Center for Translational Neuro- and Behavioral Sciences, Department of Neurology, University Medicine Essen, Essen, Germany
| | - Jan Vollert
- Department of Clinical and Biomedical Sciences, Faculty of Health and Life Sciences, University of Exeter, Exeter, United Kingdom; Pain Research, Department of Surgery and Cancer, Imperial College London, London, United Kingdom.
| |
Collapse
|
5
|
Rosenblatt M, Tejavibulya L, Jiang R, Noble S, Scheinost D. The effects of data leakage on connectome-based machine learning models. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.06.09.544383. [PMID: 38234740 PMCID: PMC10793416 DOI: 10.1101/2023.06.09.544383] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/19/2024]
Abstract
Predictive modeling has now become a central technique in neuroimaging to identify complex brain-behavior relationships and test their generalizability to unseen data. However, data leakage, which unintentionally breaches the separation between data used to train and test the model, undermines the validity of predictive models. Previous literature suggests that leakage is generally pervasive in machine learning, but few studies have empirically evaluated the effects of leakage in neuroimaging data. Although leakage is always an incorrect practice, understanding the effects of leakage on neuroimaging predictive models provides insight into the extent to which leakage may affect the literature. Here, we investigated the effects of leakage on machine learning models in two common neuroimaging modalities, functional and structural connectomes. Using over 400 different pipelines spanning four large datasets and three phenotypes, we evaluated five forms of leakage fitting into three broad categories: feature selection, covariate correction, and lack of independence between subjects. As expected, leakage via feature selection and repeated subjects drastically inflated prediction performance. Notably, other forms of leakage had only minor effects (e.g., leaky site correction) or even decreased prediction performance (e.g., leaky covariate regression). In some cases, leakage affected not only prediction performance, but also model coefficients, and thus neurobiological interpretations. Finally, we found that predictive models using small datasets were more sensitive to leakage. Overall, our results illustrate the variable effects of leakage on prediction pipelines and underscore the importance of avoiding data leakage to improve the validity and reproducibility of predictive modeling.
Collapse
Affiliation(s)
| | - Link Tejavibulya
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT
| | - Rongtao Jiang
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT
| | - Stephanie Noble
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT
- Department of Bioengineering, Northeastern University, Boston, MA
- Department of Psychology, Northeastern University, Boston, MA
| | - Dustin Scheinost
- Department of Biomedical Engineering, Yale University, New Haven, CT
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT
- Department of Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT
- Child Study Center, Yale School of Medicine, New Haven, CT
- Department of Statistics & Data Science, Yale University, New Haven, CT
| |
Collapse
|
6
|
Scheinost D, Pollatou A, Dufford AJ, Jiang R, Farruggia MC, Rosenblatt M, Peterson H, Rodriguez RX, Dadashkarimi J, Liang Q, Dai W, Foster ML, Camp CC, Tejavibulya L, Adkinson BD, Sun H, Ye J, Cheng Q, Spann MN, Rolison M, Noble S, Westwater ML. Machine Learning and Prediction in Fetal, Infant, and Toddler Neuroimaging: A Review and Primer. Biol Psychiatry 2023; 93:893-904. [PMID: 36759257 PMCID: PMC10259670 DOI: 10.1016/j.biopsych.2022.10.014] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/17/2022] [Revised: 09/10/2022] [Accepted: 10/07/2022] [Indexed: 12/01/2022]
Abstract
Predictive models in neuroimaging are increasingly designed with the intent to improve risk stratification and support interventional efforts in psychiatry. Many of these models have been developed in samples of children school-aged or older. Nevertheless, despite growing evidence that altered brain maturation during the fetal, infant, and toddler (FIT) period modulates risk for poor mental health outcomes in childhood, these models are rarely implemented in FIT samples. Applications of predictive modeling in children of these ages provide an opportunity to develop powerful tools for improved characterization of the neural mechanisms underlying development. To facilitate the broader use of predictive models in FIT neuroimaging, we present a brief primer and systematic review on the methods used in current predictive modeling FIT studies. Reflecting on current practices in more than 100 studies conducted over the past decade, we provide an overview of topics, modalities, and methods commonly used in the field and under-researched areas. We then outline ethical and future considerations for neuroimaging researchers interested in predicting health outcomes in early life, including researchers who may be relatively new to either advanced machine learning methods or using FIT data. Altogether, the last decade of FIT research in machine learning has provided a foundation for accelerating the prediction of early-life trajectories across the full spectrum of illness and health.
Collapse
Affiliation(s)
- Dustin Scheinost
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut; Department of Biomedical Engineering, Yale University, New Haven, Connecticut; Department of Statistics and Data Science, Yale University, New Haven, Connecticut; Child Study Center, Yale School of Medicine, New Haven, Connecticut; Interdepartmental Neuroscience Program, Yale University, New Haven, Connecticut.
| | - Angeliki Pollatou
- Department of Psychiatry, Columbia University Irving Medical Center, New York, New York
| | - Alexander J Dufford
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut
| | - Rongtao Jiang
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut
| | - Michael C Farruggia
- Interdepartmental Neuroscience Program, Yale University, New Haven, Connecticut
| | - Matthew Rosenblatt
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut
| | - Hannah Peterson
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut
| | | | | | - Qinghao Liang
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut
| | - Wei Dai
- Department of Biostatistics, Yale School of Public Health, New Haven, Connecticut
| | - Maya L Foster
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut
| | - Chris C Camp
- Interdepartmental Neuroscience Program, Yale University, New Haven, Connecticut
| | - Link Tejavibulya
- Interdepartmental Neuroscience Program, Yale University, New Haven, Connecticut
| | - Brendan D Adkinson
- Interdepartmental Neuroscience Program, Yale University, New Haven, Connecticut
| | - Huili Sun
- Department of Biomedical Engineering, Yale University, New Haven, Connecticut
| | - Jean Ye
- Interdepartmental Neuroscience Program, Yale University, New Haven, Connecticut
| | - Qi Cheng
- Departments of Neuroscience and Psychology, Smith College, Northampton, Massachusetts
| | - Marisa N Spann
- Department of Psychiatry, Columbia University Irving Medical Center, New York, New York
| | - Max Rolison
- Child Study Center, Yale School of Medicine, New Haven, Connecticut
| | - Stephanie Noble
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut
| | - Margaret L Westwater
- Department of Radiology and Biomedical Imaging, Yale School of Medicine, New Haven, Connecticut
| |
Collapse
|
7
|
Kotikalapudi R, Moser DA, Dricu M, Spisak T, Aue T. Predictive modeling of optimism bias using gray matter cortical thickness. Sci Rep 2023; 13:302. [PMID: 36609577 PMCID: PMC9822990 DOI: 10.1038/s41598-022-26550-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Accepted: 12/15/2022] [Indexed: 01/07/2023] Open
Abstract
People have been shown to be optimistically biased when their future outcome expectancies are assessed. In fact, we display optimism bias (OB) toward our own success when compared to a rival individual's (personal OB [POB]). Similarly, success expectancies for social groups we like reliably exceed those we mention for a rival group (social OB [SOB]). Recent findings suggest the existence of neural underpinnings for OB. Mostly using structural/functional MRI, these findings rely on voxel-based mass-univariate analyses. While these results remain associative in nature, an open question abides whether MRI information can accurately predict OB. In this study, we hence used predictive modelling to forecast the two OBs. The biases were quantified using a validated soccer paradigm, where personal (self versus rival) and social (in-group versus out-group) forms of OB were extracted at the participant level. Later, using gray matter cortical thickness, we predicted POB and SOB via machine-learning. Our model explained 17% variance (R2 = 0.17) in individual variability for POB (but not SOB). Key predictors involved the rostral-caudal anterior cingulate cortex, pars orbitalis and entorhinal cortex-areas that have been associated with OB before. We need such predictive models on a larger scale, to help us better understand positive psychology and individual well-being.
Collapse
Affiliation(s)
- Raviteja Kotikalapudi
- Institute of Psychology, University of Bern, Fabrikstrasse 8, 3012, Bern, Switzerland. .,Department of Neurology, University Hospital Essen, Hufelandstrasse 55, 45147, Essen, Germany.
| | - Dominik A. Moser
- grid.5734.50000 0001 0726 5157Institute of Psychology, University of Bern, Fabrikstrasse 8, 3012 Bern, Switzerland
| | - Mihai Dricu
- grid.5734.50000 0001 0726 5157Institute of Psychology, University of Bern, Fabrikstrasse 8, 3012 Bern, Switzerland
| | - Tamas Spisak
- grid.410718.b0000 0001 0262 7331Department of Neurology, University Hospital Essen, Hufelandstrasse 55, 45147 Essen, Germany
| | - Tatjana Aue
- Institute of Psychology, University of Bern, Fabrikstrasse 8, 3012, Bern, Switzerland.
| |
Collapse
|
8
|
Hamdan S, Love BC, von Polier GG, Weis S, Schwender H, Eickhoff SB, Patil KR. Confound-leakage: confound removal in machine learning leads to leakage. Gigascience 2022; 12:giad071. [PMID: 37776368 PMCID: PMC10541796 DOI: 10.1093/gigascience/giad071] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2023] [Revised: 06/01/2023] [Accepted: 08/17/2023] [Indexed: 10/02/2023] Open
Abstract
BACKGROUND Machine learning (ML) approaches are a crucial component of modern data analysis in many fields, including epidemiology and medicine. Nonlinear ML methods often achieve accurate predictions, for instance, in personalized medicine, as they are capable of modeling complex relationships between features and the target. Problematically, ML models and their predictions can be biased by confounding information present in the features. To remove this spurious signal, researchers often employ featurewise linear confound regression (CR). While this is considered a standard approach for dealing with confounding, possible pitfalls of using CR in ML pipelines are not fully understood. RESULTS We provide new evidence that, contrary to general expectations, linear confound regression can increase the risk of confounding when combined with nonlinear ML approaches. Using a simple framework that uses the target as a confound, we show that information leaked via CR can increase null or moderate effects to near-perfect prediction. By shuffling the features, we provide evidence that this increase is indeed due to confound-leakage and not due to revealing of information. We then demonstrate the danger of confound-leakage in a real-world clinical application where the accuracy of predicting attention-deficit/hyperactivity disorder is overestimated using speech-derived features when using depression as a confound. CONCLUSIONS Mishandling or even amplifying confounding effects when building ML models due to confound-leakage, as shown, can lead to untrustworthy, biased, and unfair predictions. Our expose of the confound-leakage pitfall and provided guidelines for dealing with it can help create more robust and trustworthy ML models.
Collapse
Affiliation(s)
- Sami Hamdan
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Forschungszentrum Jülich, 52428 Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich-Heine University Düsseldorf, 40225 Düsseldorf, Germany
| | - Bradley C Love
- Department of Experimental Psychology, University College London, WC1H 0AP London, UK
- The Alan Turing Institute, London NW1 2DB, UK
- European Lab for Learning & Intelligent Systems (ELLIS), WC1E 6BT, London, UK
| | - Georg G von Polier
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Forschungszentrum Jülich, 52428 Jülich, Germany
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, University Hospital Frankfurt, 60528 Frankfurt, Germany
- Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, RWTH Aachen University, 52074 Aachen, Germany
| | - Susanne Weis
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Forschungszentrum Jülich, 52428 Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich-Heine University Düsseldorf, 40225 Düsseldorf, Germany
| | - Holger Schwender
- Institute of Mathematics, Heinrich-Heine University Düsseldorf, 40225 Düsseldorf, Germany
| | - Simon B Eickhoff
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Forschungszentrum Jülich, 52428 Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich-Heine University Düsseldorf, 40225 Düsseldorf, Germany
| | - Kaustubh R Patil
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Forschungszentrum Jülich, 52428 Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich-Heine University Düsseldorf, 40225 Düsseldorf, Germany
| |
Collapse
|
9
|
Spisak T. Statistical quantification of confounding bias in machine learning models. Gigascience 2022; 11:giac082. [PMID: 36017878 PMCID: PMC9412867 DOI: 10.1093/gigascience/giac082] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2022] [Revised: 07/07/2022] [Accepted: 07/28/2022] [Indexed: 11/12/2022] Open
Abstract
BACKGROUND The lack of nonparametric statistical tests for confounding bias significantly hampers the development of robust, valid, and generalizable predictive models in many fields of research. Here I propose the partial confounder test, which, for a given confounder variable, probes the null hypotheses of the model being unconfounded. RESULTS The test provides a strict control for type I errors and high statistical power, even for nonnormally and nonlinearly dependent predictions, often seen in machine learning. Applying the proposed test on models trained on large-scale functional brain connectivity data (N= 1,865) (i) reveals previously unreported confounders and (ii) shows that state-of-the-art confound mitigation approaches may fail preventing confounder bias in several cases. CONCLUSIONS The proposed test (implemented in the package mlconfound; https://mlconfound.readthedocs.io) can aid the assessment and improvement of the generalizability and validity of predictive models and, thereby, fosters the development of clinically useful machine learning biomarkers.
Collapse
Affiliation(s)
- Tamas Spisak
- Center for Translational Neuro- and Behavioral Sciences, Institute for Diagnostic and Interventional Radiology and Neuroradiology, Center University Hospital Essen, Essen, D-45147, Germany
| |
Collapse
|