16
|
Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, Kirchler M, Iwanir R, Mumford JA, Adcock RA, Avesani P, Baczkowski BM, Bajracharya A, Bakst L, Ball S, Barilari M, Bault N, Beaton D, Beitner J, Benoit RG, Berkers RMWJ, Bhanji JP, Biswal BB, Bobadilla-Suarez S, Bortolini T, Bottenhorn KL, Bowring A, Braem S, Brooks HR, Brudner EG, Calderon CB, Camilleri JA, Castrellon JJ, Cecchetti L, Cieslik EC, Cole ZJ, Collignon O, Cox RW, Cunningham WA, Czoschke S, Dadi K, Davis CP, Luca AD, Delgado MR, Demetriou L, Dennison JB, Di X, Dickie EW, Dobryakova E, Donnat CL, Dukart J, Duncan NW, Durnez J, Eed A, Eickhoff SB, Erhart A, Fontanesi L, Fricke GM, Fu S, Galván A, Gau R, Genon S, Glatard T, Glerean E, Goeman JJ, Golowin SAE, González-García C, Gorgolewski KJ, Grady CL, Green MA, Guassi Moreira JF, Guest O, Hakimi S, Hamilton JP, Hancock R, Handjaras G, Harry BB, Hawco C, Herholz P, Herman G, Heunis S, Hoffstaedter F, Hogeveen J, Holmes S, Hu CP, Huettel SA, Hughes ME, Iacovella V, Iordan AD, Isager PM, Isik AI, Jahn A, Johnson MR, Johnstone T, Joseph MJE, Juliano AC, Kable JW, Kassinopoulos M, Koba C, Kong XZ, Koscik TR, Kucukboyaci NE, Kuhl BA, Kupek S, Laird AR, Lamm C, Langner R, Lauharatanahirun N, Lee H, Lee S, Leemans A, Leo A, Lesage E, Li F, Li MYC, Lim PC, Lintz EN, Liphardt SW, Losecaat Vermeer AB, Love BC, Mack ML, Malpica N, Marins T, Maumet C, McDonald K, McGuire JT, Melero H, Méndez Leal AS, Meyer B, Meyer KN, Mihai G, Mitsis GD, Moll J, Nielson DM, Nilsonne G, Notter MP, Olivetti E, Onicas AI, Papale P, Patil KR, Peelle JE, Pérez A, Pischedda D, Poline JB, Prystauka Y, Ray S, Reuter-Lorenz PA, Reynolds RC, Ricciardi E, Rieck JR, Rodriguez-Thompson AM, Romyn A, Salo T, Samanez-Larkin GR, Sanz-Morales E, Schlichting ML, Schultz DH, Shen Q, Sheridan MA, Silvers JA, Skagerlund K, Smith A, Smith DV, Sokol-Hessner P, Steinkamp SR, Tashjian SM, Thirion B, Thorp JN, Tinghög G, Tisdall L, Tompson SH, Toro-Serey C, Torre Tresols JJ, Tozzi L, Truong V, Turella L, van 't Veer AE, Verguts T, Vettel JM, Vijayarajah S, Vo K, Wall MB, Weeda WD, Weis S, White DJ, Wisniewski D, Xifra-Porxas A, Yearling EA, Yoon S, Yuan R, Yuen KSL, Zhang L, Zhang X, Zosky JE, Nichols TE, Poldrack RA, Schonberg T. Variability in the analysis of a single neuroimaging dataset by many teams. Nature 2020; 582:84-88. [PMID: 32483374 PMCID: PMC7771346 DOI: 10.1038/s41586-020-2314-9] [Citation(s) in RCA: 423] [Impact Index Per Article: 105.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2019] [Accepted: 04/07/2020] [Indexed: 01/13/2023]
Abstract
Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.
Collapse
Affiliation(s)
- Rotem Botvinik-Nezer
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
- Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Felix Holzmeister
- Department of Banking and Finance, University of Innsbruck, Innsbruck, Austria
| | - Colin F Camerer
- HSS and CNS, California Institute of Technology, Pasadena, CA, USA
| | - Anna Dreber
- Department of Economics, Stockholm School of Economics, Stockholm, Sweden
- Department of Economics, University of Innsbruck, Innsbruck, Austria
| | - Juergen Huber
- Department of Banking and Finance, University of Innsbruck, Innsbruck, Austria
| | - Magnus Johannesson
- Department of Economics, Stockholm School of Economics, Stockholm, Sweden
| | - Michael Kirchler
- Department of Banking and Finance, University of Innsbruck, Innsbruck, Austria
| | - Roni Iwanir
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
- Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel
| | - Jeanette A Mumford
- Center for Healthy Minds, University of Wisconsin-Madison, Madison, WI, USA
| | - R Alison Adcock
- Center for Cognitive Neuroscience, Duke University, Durham, NC, USA
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, NC, USA
| | - Paolo Avesani
- Neuroinformatics Laboratory, Fondazione Bruno Kessler, Trento, Italy
- Center for Mind/Brain Sciences - CIMeC, University of Trento, Rovereto, Italy
| | - Blazej M Baczkowski
- Department of Neurology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Aahana Bajracharya
- Department of Otolaryngology, Washington University in St. Louis, St. Louis, MO, USA
| | - Leah Bakst
- Department of Psychological and Brain Sciences, Boston University, Boston, MA, USA
- Center for Systems Neuroscience, Boston University, Boston, MA, USA
| | - Sheryl Ball
- Department of Economics, Virginia Tech, Blacksburg, VA, USA
- School of Neuroscience, Virginia Tech, Blacksburg, VA, USA
| | - Marco Barilari
- Crossmodal Perception and Plasticity Laboratory, Institutes for Research in Psychology (IPSY) and Neurosciences (IoNS), UCLouvain, Louvain-la-Neuve, Belgium
| | - Nadège Bault
- School of Psychology, University of Plymouth, Plymouth, UK
| | - Derek Beaton
- Rotman Research Institute, Baycrest Health Sciences Centre, Toronto, Ontario, Canada
| | - Julia Beitner
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- Department of Psychology, Goethe University, Frankfurt am Main, Germany
| | - Roland G Benoit
- Max Planck Research Group: Adaptive Memory, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Ruud M W J Berkers
- Max Planck Research Group: Adaptive Memory, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Jamil P Bhanji
- Department of Psychology, Rutgers University-Newark, Newark, NJ, USA
| | - Bharat B Biswal
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, USA
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | | | - Tiago Bortolini
- D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil
| | | | - Alexander Bowring
- Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK
| | - Senne Braem
- Department of Experimental Psychology, Ghent University, Ghent, Belgium
- Department of Psychology, Vrije Universiteit Brussel, Brussels, Belgium
| | - Hayley R Brooks
- Department of Psychology, University of Denver, Denver, CO, USA
| | - Emily G Brudner
- Department of Psychology, Rutgers University-Newark, Newark, NJ, USA
| | | | - Julia A Camilleri
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Jaime J Castrellon
- Center for Cognitive Neuroscience, Duke University, Durham, NC, USA
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA
| | - Luca Cecchetti
- MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Edna C Cieslik
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Zachary J Cole
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA
| | - Olivier Collignon
- Center for Mind/Brain Sciences - CIMeC, University of Trento, Rovereto, Italy
- Crossmodal Perception and Plasticity Laboratory, Institutes for Research in Psychology (IPSY) and Neurosciences (IoNS), UCLouvain, Louvain-la-Neuve, Belgium
| | - Robert W Cox
- National Institute of Mental Health (NIMH), National Institutes of Health, Bethesda, MD, USA
| | | | - Stefan Czoschke
- Institute of Medical Psychology, Goethe University, Frankfurt am Main, Germany
| | | | - Charles P Davis
- Department of Psychological Sciences, University of Connecticut, Storrs, CT, USA
- Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA
- Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA
| | - Alberto De Luca
- PROVIDI Lab, Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands
| | | | - Lysia Demetriou
- Section of Endocrinology and Investigative Medicine, Faculty of Medicine, Imperial College London, London, UK
- Nuffield Department of Women's and Reproductive Health, University of Oxford, Oxford, UK
| | | | - Xin Di
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, USA
- School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Erin W Dickie
- Krembil Centre for Neuroinformatics, Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada
- Department of Psychiatry, University of Toronto, Toronto, Ontario, Canada
| | - Ekaterina Dobryakova
- Center for Traumatic Brain Injury Research, Kessler Foundation, East Hanover, NJ, USA
| | - Claire L Donnat
- Department of Statistics, Stanford University, Stanford, CA, USA
| | - Juergen Dukart
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Niall W Duncan
- Graduate Institute of Mind, Brain and Consciousness, Taipei Medical University, Taipei, Taiwan
- Brain and Consciousness Research Centre, TMU-ShuangHo Hospital, New Taipei City, Taiwan
| | - Joke Durnez
- Department of Psychology and Stanford Center for Reproducible Neuroscience, Stanford University, Stanford, CA, USA
| | - Amr Eed
- Instituto de Neurociencias, CSIC-UMH, Alicante, Spain
| | - Simon B Eickhoff
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Andrew Erhart
- Department of Psychology, University of Denver, Denver, CO, USA
| | - Laura Fontanesi
- Faculty of Psychology, University of Basel, Basel, Switzerland
| | - G Matthew Fricke
- Computer Science Department, University of New Mexico, Albuquerque, NM, USA
| | - Shiguang Fu
- School of Management, Zhejiang University of Technology, Hangzhou, China
- Institute of Neuromanagement, Zhejiang University of Technology, Hangzhou, China
| | - Adriana Galván
- Department of Psychology, University of California Los Angeles, Los Angeles, CA, USA
| | - Remi Gau
- Crossmodal Perception and Plasticity Laboratory, Institutes for Research in Psychology (IPSY) and Neurosciences (IoNS), UCLouvain, Louvain-la-Neuve, Belgium
| | - Sarah Genon
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Tristan Glatard
- Department of Computer Science and Software Engineering, Concordia University, Montreal, Quebec, Canada
| | - Enrico Glerean
- Department of Neuroscience and Biomedical Engineering, Aalto University, Espoo, Finland
| | - Jelle J Goeman
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands
| | - Sergej A E Golowin
- Graduate Institute of Mind, Brain and Consciousness, Taipei Medical University, Taipei, Taiwan
| | | | | | - Cheryl L Grady
- Rotman Research Institute, Baycrest Health Sciences Centre, Toronto, Ontario, Canada
| | - Mikella A Green
- Center for Cognitive Neuroscience, Duke University, Durham, NC, USA
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA
| | - João F Guassi Moreira
- Department of Psychology, University of California Los Angeles, Los Angeles, CA, USA
| | - Olivia Guest
- Department of Experimental Psychology, University College London, London, UK
- Research Centre on Interactive Media, Smart Systems and Emerging Technologies - RISE, Nicosia, Cyprus
| | - Shabnam Hakimi
- Center for Cognitive Neuroscience, Duke University, Durham, NC, USA
| | - J Paul Hamilton
- Center for Social and Affective Neuroscience, Department of Biomedical and Clinical Sciences, Linköping University, Linköping, Sweden
| | - Roeland Hancock
- Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA
- Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA
| | - Giacomo Handjaras
- MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Bronson B Harry
- The MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Sydney, New South Wales, Australia
| | - Colin Hawco
- Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada
| | - Peer Herholz
- McConnell Brain Imaging Centre, The Neuro (Montreal Neurological Institute-Hospital), Faculty of Medicine, McGill University, Montreal, Quebec, Canada
| | - Gabrielle Herman
- Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada
| | - Stephan Heunis
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, The Netherlands
- Department of Research and Development, Epilepsy Centre Kempenhaeghe, Heeze, The Netherlands
| | - Felix Hoffstaedter
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Jeremy Hogeveen
- Department of Psychology, University of New Mexico, Albuquerque, NM, USA
- Psychology Clinical Neuroscience Center, University of New Mexico, Albuquerque, NM, USA
| | - Susan Holmes
- Department of Statistics, Stanford University, Stanford, CA, USA
| | - Chuan-Peng Hu
- Leibniz-Institut für Resilienzforschung (LIR), Mainz, Germany
| | - Scott A Huettel
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA
| | - Matthew E Hughes
- School of Health Sciences, Swinburne University of Technology, Hawthorn, Victoria, Australia
| | - Vittorio Iacovella
- Center for Mind/Brain Sciences - CIMeC, University of Trento, Rovereto, Italy
| | | | - Peder M Isager
- Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands
| | - Ayse I Isik
- Department of Neuroscience, Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany
| | - Andrew Jahn
- fMRI Laboratory, University of Michigan, Ann Arbor, MI, USA
| | - Matthew R Johnson
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA
- Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, USA
| | - Tom Johnstone
- School of Health Sciences, Swinburne University of Technology, Hawthorn, Victoria, Australia
| | - Michael J E Joseph
- Campbell Family Mental Health Research Institute, Centre for Addiction and Mental Health, Toronto, Ontario, Canada
| | - Anthony C Juliano
- Center for Neuropsychology and Neuroscience Research, Kessler Foundation, East Hanover, NJ, USA
| | - Joseph W Kable
- Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA
- MindCORE, University of Pennsylvania, Philadelphia, PA, USA
| | - Michalis Kassinopoulos
- Graduate Program in Biological and Biomedical Engineering, McGill University, Montreal, Quebec, Canada
| | - Cemal Koba
- MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Xiang-Zhen Kong
- Language and Genetics Department, Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
| | - Timothy R Koscik
- Department of Psychiatry, University of Iowa Carver College of Medicine, Iowa City, IA, USA
| | - Nuri Erkut Kucukboyaci
- Center for Traumatic Brain Injury Research, Kessler Foundation, East Hanover, NJ, USA
- Department of Physical Medicine and Rehabilitation, Rutgers New Jersey Medical School, Newark, NJ, USA
| | - Brice A Kuhl
- Department of Psychology, University of Oregon, Eugene, OR, USA
| | - Sebastian Kupek
- Faculty of Economics and Statistics, University of Innsbruck, Innsbruck, Austria
| | - Angela R Laird
- Department of Physics, Florida International University, Miami, Florida, USA
| | - Claus Lamm
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
- Vienna Cognitive Science Hub, University of Vienna, Vienna, Austria
| | - Robert Langner
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Nina Lauharatanahirun
- US CCDC Army Research Laboratory, Human Research and Engineering Directorate, Aberdeen Proving Ground, MD, USA
- Annenberg School for Communication, University of Pennsylvania, Philadelphia, PA, USA
| | - Hongmi Lee
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Sangil Lee
- Department of Psychology, University of Pennsylvania, Philadelphia, PA, USA
| | - Alexander Leemans
- PROVIDI Lab, Image Sciences Institute, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Andrea Leo
- MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Elise Lesage
- Department of Experimental Psychology, Ghent University, Ghent, Belgium
| | - Flora Li
- Fralin Biomedical Research Institute, Roanoke, VA, USA
- Economics Experimental Lab, Nanjing Audit University, Nanjing, China
| | - Monica Y C Li
- Department of Psychological Sciences, University of Connecticut, Storrs, CT, USA
- Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA
- Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA
- Haskins Laboratories, New Haven, CT, USA
| | - Phui Cheng Lim
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA
- Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, USA
| | - Evan N Lintz
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA
| | | | - Annabel B Losecaat Vermeer
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
| | - Bradley C Love
- Department of Experimental Psychology, University College London, London, UK
- The Alan Turing Institute, London, UK
| | - Michael L Mack
- Department of Psychology, University of Toronto, Toronto, Ontario, Canada
| | - Norberto Malpica
- Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos, Madrid, Spain
| | - Theo Marins
- D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil
| | - Camille Maumet
- Inria, Univ Rennes, CNRS, Inserm, IRISA UMR 6074, Empenn ERL U 1228, Rennes, France
| | - Kelsey McDonald
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA
| | - Joseph T McGuire
- Department of Psychological and Brain Sciences, Boston University, Boston, MA, USA
- Center for Systems Neuroscience, Boston University, Boston, MA, USA
| | - Helena Melero
- Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos, Madrid, Spain
- Departamento de Psicobiología, División de Psicología, CES Cardenal Cisneros, Madrid, Spain
- Northeastern University Biomedical Imaging Center, Northeastern University, Boston, MA, USA
| | - Adriana S Méndez Leal
- Department of Psychology, University of California Los Angeles, Los Angeles, CA, USA
| | - Benjamin Meyer
- Leibniz-Institut für Resilienzforschung (LIR), Mainz, Germany
- Neuroimaging Center (NIC), Focus Program Translational Neurosciences (FTN), Johannes Gutenberg University Medical Center Mainz, Mainz, Germany
| | - Kristin N Meyer
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Glad Mihai
- Max Planck Research Group: Neural Mechanisms of Human Communication, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany
| | - Georgios D Mitsis
- Department of Bioengineering, McGill University, Montreal, Quebec, Canada
| | - Jorge Moll
- D'Or Institute for Research and Education (IDOR), Rio de Janeiro, Brazil
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Dylan M Nielson
- Data Science and Sharing Team, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA
| | - Gustav Nilsonne
- Department of Clinical Neuroscience, Karolinska Institutet, Stockholm, Sweden
- Department of Psychology, Stockholm University, Stockholm, Sweden
| | - Michael P Notter
- The Laboratory for Investigative Neurophysiology (The LINE), Department of Radiology, University Hospital Center and University of Lausanne, Lausanne, Switzerland
| | - Emanuele Olivetti
- Neuroinformatics Laboratory, Fondazione Bruno Kessler, Trento, Italy
- Center for Mind/Brain Sciences - CIMeC, University of Trento, Rovereto, Italy
| | - Adrian I Onicas
- MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Paolo Papale
- MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Amsterdam, The Netherlands
| | - Kaustubh R Patil
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Jonathan E Peelle
- Department of Otolaryngology, Washington University in St. Louis, St. Louis, MO, USA
| | - Alexandre Pérez
- McConnell Brain Imaging Centre, The Neuro (Montreal Neurological Institute-Hospital), Faculty of Medicine, McGill University, Montreal, Quebec, Canada
| | - Doris Pischedda
- Bernstein Center for Computational Neuroscience and Berlin Center for Advanced Neuroimaging and Clinic for Neurology, Charité Universitätsmedizin, corporate member of Freie Universität Berlin, Humboldt Universität zu Berlin, and Berlin Institute of Health, Berlin, Germany
- Cluster of Excellence Science of Intelligence, Technische Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany
- NeuroMI - Milan Center for Neuroscience, Milan, Italy
| | - Jean-Baptiste Poline
- McConnell Brain Imaging Centre, The Neuro (Montreal Neurological Institute-Hospital), Faculty of Medicine, McGill University, Montreal, Quebec, Canada
- Henry H. Wheeler, Jr. Brain Imaging Center, Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA
| | - Yanina Prystauka
- Department of Psychological Sciences, University of Connecticut, Storrs, CT, USA
- Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA
- Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA
| | - Shruti Ray
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ, USA
| | | | - Richard C Reynolds
- Scientific and Statistical Computing Core, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA
| | - Emiliano Ricciardi
- MoMiLab Research Unit, IMT School for Advanced Studies Lucca, Lucca, Italy
| | - Jenny R Rieck
- Rotman Research Institute, Baycrest Health Sciences Centre, Toronto, Ontario, Canada
| | - Anais M Rodriguez-Thompson
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Anthony Romyn
- Department of Psychology, University of Toronto, Toronto, Ontario, Canada
| | - Taylor Salo
- Department of Psychology, Florida International University, Miami, FL, USA
| | - Gregory R Samanez-Larkin
- Center for Cognitive Neuroscience, Duke University, Durham, NC, USA
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA
| | - Emilio Sanz-Morales
- Laboratorio de Análisis de Imagen Médica y Biometría (LAIMBIO), Universidad Rey Juan Carlos, Madrid, Spain
| | | | - Douglas H Schultz
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA
- Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, USA
| | - Qiang Shen
- School of Management, Zhejiang University of Technology, Hangzhou, China
- Institute of Neuromanagement, Zhejiang University of Technology, Hangzhou, China
| | - Margaret A Sheridan
- Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Jennifer A Silvers
- Department of Psychology, University of California Los Angeles, Los Angeles, CA, USA
| | - Kenny Skagerlund
- Department of Behavioural Sciences and Learning, Linköping University, Linköping, Sweden
- Center for Social and Affective Neuroscience, Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden
| | - Alec Smith
- Department of Economics, Virginia Tech, Blacksburg, VA, USA
- School of Neuroscience, Virginia Tech, Blacksburg, VA, USA
| | - David V Smith
- Department of Psychology, Temple University, Philadelphia, PA, USA
| | | | - Simon R Steinkamp
- Institute of Neuroscience and Medicine, Cognitive Neuroscience (INM-3), Research Centre Jülich, Jülich, Germany
| | - Sarah M Tashjian
- Department of Psychology, University of California Los Angeles, Los Angeles, CA, USA
| | | | - John N Thorp
- Department of Psychology, Columbia University, New York, NY, USA
| | - Gustav Tinghög
- Department of Management and Engineering, Linköping University, Linköping, Sweden
- Department of Health, Medicine and Caring Sciences, Linköping University, Linköping, Sweden
| | - Loreen Tisdall
- Department of Psychology, Stanford University, Stanford, CA, USA
- Center for Cognitive and Decision Sciences, University of Basel, Basel, Switzerland
| | - Steven H Tompson
- US CCDC Army Research Laboratory, Human Research and Engineering Directorate, Aberdeen Proving Ground, MD, USA
| | - Claudio Toro-Serey
- Department of Psychological and Brain Sciences, Boston University, Boston, MA, USA
- Center for Systems Neuroscience, Boston University, Boston, MA, USA
| | | | - Leonardo Tozzi
- Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Vuong Truong
- Graduate Institute of Mind, Brain and Consciousness, Taipei Medical University, Taipei, Taiwan
- Brain and Consciousness Research Centre, TMU-ShuangHo Hospital, New Taipei City, Taiwan
| | - Luca Turella
- Center for Mind/Brain Sciences - CIMeC, University of Trento, Rovereto, Italy
| | - Anna E van 't Veer
- Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, The Netherlands
| | - Tom Verguts
- Department of Experimental Psychology, Ghent University, Ghent, Belgium
| | - Jean M Vettel
- US Combat Capabilities Development Command Army Research Laboratory, Aberdeen, MD, USA
- University of California Santa Barbara, Santa Barbara, CA, USA
- University of Pennsylvania, Philadelphia, PA, USA
| | - Sagana Vijayarajah
- Department of Psychology, University of Toronto, Toronto, Ontario, Canada
| | - Khoi Vo
- Center for Cognitive Neuroscience, Duke University, Durham, NC, USA
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA
| | - Matthew B Wall
- Invicro, London, UK
- Faculty of Medicine, Imperial College London, London, UK
- Clinical Psychopharmacology Unit, University College London, London, UK
| | - Wouter D Weeda
- Methodology and Statistics Unit, Institute of Psychology, Leiden University, Leiden, The Netherlands
| | - Susanne Weis
- Institute of Neuroscience and Medicine, Brain and Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - David J White
- Centre for Human Psychopharmacology, Swinburne University, Hawthorn, Victoria, Australia
| | - David Wisniewski
- Department of Experimental Psychology, Ghent University, Ghent, Belgium
| | - Alba Xifra-Porxas
- Graduate Program in Biological and Biomedical Engineering, McGill University, Montreal, Quebec, Canada
| | - Emily A Yearling
- Department of Psychological Sciences, University of Connecticut, Storrs, CT, USA
- Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA
- Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA
| | - Sangsuk Yoon
- Department of Management and Marketing, School of Business, University of Dayton, Dayton, OH, USA
| | - Rui Yuan
- Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Kenneth S L Yuen
- Leibniz-Institut für Resilienzforschung (LIR), Mainz, Germany
- Neuroimaging Center (NIC), Focus Program Translational Neurosciences (FTN), Johannes Gutenberg University Medical Center Mainz, Mainz, Germany
| | - Lei Zhang
- Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria
| | - Xu Zhang
- Brain Imaging Research Center, University of Connecticut, Storrs, CT, USA
- Connecticut Institute for the Brain and Cognitive Sciences, University of Connecticut, Storrs, CT, USA
- Biomedical Engineering Department, University of Connecticut, Storrs, CT, USA
| | - Joshua E Zosky
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE, USA
- Center for Brain, Biology and Behavior, University of Nebraska-Lincoln, Lincoln, NE, USA
| | - Thomas E Nichols
- Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK.
| | | | - Tom Schonberg
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel.
- Department of Neurobiology, The George S. Wise Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel.
| |
Collapse
|