1
|
Botvinik-Nezer R, Wager TD. Reproducibility in Neuroimaging Analysis: Challenges and Solutions. BIOLOGICAL PSYCHIATRY. COGNITIVE NEUROSCIENCE AND NEUROIMAGING 2023; 8:780-788. [PMID: 36906444 DOI: 10.1016/j.bpsc.2022.12.006] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 11/27/2022] [Accepted: 12/11/2022] [Indexed: 12/23/2022]
Abstract
Recent years have marked a renaissance in efforts to increase research reproducibility in psychology, neuroscience, and related fields. Reproducibility is the cornerstone of a solid foundation of fundamental research-one that will support new theories built on valid findings and technological innovation that works. The increased focus on reproducibility has made the barriers to it increasingly apparent, along with the development of new tools and practices to overcome these barriers. Here, we review challenges, solutions, and emerging best practices with a particular emphasis on neuroimaging studies. We distinguish 3 main types of reproducibility, discussing each in turn. Analytical reproducibility is the ability to reproduce findings using the same data and methods. Replicability is the ability to find an effect in new datasets, using the same or similar methods. Finally, robustness to analytical variability refers to the ability to identify a finding consistently across variation in methods. The incorporation of these tools and practices will result in more reproducible, replicable, and robust psychological and brain research and a stronger scientific foundation across fields of inquiry.
Collapse
Affiliation(s)
- Rotem Botvinik-Nezer
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire.
| | - Tor D Wager
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire
| |
Collapse
|
2
|
Reer A, Wiebe A, Wang X, Rieger JW. FAIR human neuroscientific data sharing to advance AI driven research and applications: Legal frameworks and missing metadata standards. Front Genet 2023; 14:1086802. [PMID: 37007976 PMCID: PMC10065194 DOI: 10.3389/fgene.2023.1086802] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2022] [Accepted: 02/21/2023] [Indexed: 03/16/2023] Open
Abstract
Modern AI supported research holds many promises for basic and applied science. However, the application of AI methods is often limited because most labs cannot, on their own, acquire large and diverse datasets, which are best for training these methods. Data sharing and open science initiatives promise some relief to the problem, but only if the data are provided in a usable way. The FAIR principles state very general requirements for useful data sharing: they should be findable, accessible, interoperable, and reusable. This article will focus on two challenges to implement the FAIR framework for human neuroscience data. On the one hand, human data can fall under special legal protection. The legal frameworks regulating how and what data can be openly shared differ greatly across countries which can complicate data sharing or even discourage researchers from doing so. Moreover, openly accessible data require standardization of data and metadata organization and annotation in order to become interpretable and useful. This article briefly introduces open neuroscience initiatives that support the implementation of the FAIR principles. It then reviews legal frameworks, their consequences for accessibility of human neuroscientific data and some ethical implications. We hope this comparison of legal jurisdictions helps to elucidate that some alleged obstacles for data sharing only require an adaptation of procedures but help to protect the privacy of our most generous donors to research … our study participants. Finally, it elaborates on the problem of missing standards for metadata annotation and introduces initiatives that aim at developing tools to make neuroscientific data acquisition and analysis pipelines FAIR by design. While the paper focuses on making human neuroscience data useful for data-intensive AI the general considerations hold for other fields where large amounts of openly available human data would be helpful.
Collapse
Affiliation(s)
- Aaron Reer
- Applied Neurocognitive Psychology Lab, Institute for Medicine and Healthcare, Department of Psychology, Oldenburg University, Oldenburg, Germany
- *Correspondence: Aaron Reer,
| | - Andreas Wiebe
- Chair for Intellectual Property and Information Law, Göttingen University, Göttingen, Germany
| | - Xu Wang
- Chair for Intellectual Property and Information Law, Göttingen University, Göttingen, Germany
| | - Jochem W. Rieger
- Applied Neurocognitive Psychology Lab, Institute for Medicine and Healthcare, Department of Psychology, Oldenburg University, Oldenburg, Germany
| |
Collapse
|
4
|
Niso G, Botvinik-Nezer R, Appelhoff S, De La Vega A, Esteban O, Etzel JA, Finc K, Ganz M, Gau R, Halchenko YO, Herholz P, Karakuzu A, Keator DB, Markiewicz CJ, Maumet C, Pernet CR, Pestilli F, Queder N, Schmitt T, Sójka W, Wagner AS, Whitaker KJ, Rieger JW. Open and reproducible neuroimaging: From study inception to publication. Neuroimage 2022; 263:119623. [PMID: 36100172 PMCID: PMC10008521 DOI: 10.1016/j.neuroimage.2022.119623] [Citation(s) in RCA: 29] [Impact Index Per Article: 14.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Revised: 08/17/2022] [Accepted: 09/09/2022] [Indexed: 10/31/2022] Open
Abstract
Empirical observations of how labs conduct research indicate that the adoption rate of open practices for transparent, reproducible, and collaborative science remains in its infancy. This is at odds with the overwhelming evidence for the necessity of these practices and their benefits for individual researchers, scientific progress, and society in general. To date, information required for implementing open science practices throughout the different steps of a research project is scattered among many different sources. Even experienced researchers in the topic find it hard to navigate the ecosystem of tools and to make sustainable choices. Here, we provide an integrated overview of community-developed resources that can support collaborative, open, reproducible, replicable, robust and generalizable neuroimaging throughout the entire research cycle from inception to publication and across different neuroimaging modalities. We review tools and practices supporting study inception and planning, data acquisition, research data management, data processing and analysis, and research dissemination. An online version of this resource can be found at https://oreoni.github.io. We believe it will prove helpful for researchers and institutions to make a successful and sustainable move towards open and reproducible science and to eventually take an active role in its future development.
Collapse
Affiliation(s)
- Guiomar Niso
- Psychological & Brain Sciences, Indiana University, Bloomington, IN, USA; Universidad Politecnica de Madrid, Madrid and CIBER-BBN, Spain; Instituto Cajal, CSIC, Madrid, Spain.
| | - Rotem Botvinik-Nezer
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA.
| | - Stefan Appelhoff
- Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany
| | | | - Oscar Esteban
- Dept. of Radiology, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland; Department of Psychology, Stanford University, Stanford, CA, USA
| | - Joset A Etzel
- Department of Psychological and Brain Sciences, Washington University in St. Louis, St. Louis, MO, USA
| | - Karolina Finc
- Centre for Modern Interdisciplinary Technologies, Nicolaus Copernicus University, Toruń, Poland
| | - Melanie Ganz
- Neurobiology Research Unit, Rigshospitalet, Copenhagen, Denmark; Department of Computer Science, University of Copenhagen, Copenhagen, Denmark
| | - Rémi Gau
- Institute of Psychology, Université catholique de Louvain, Louvain la Neuve, Belgium
| | - Yaroslav O Halchenko
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Peer Herholz
- Montreal Neurological Institute-Hospital, McGill University, Montréal, Quebec, Canada
| | - Agah Karakuzu
- Biomedical Engineering Institute, Polytechnique Montréal, Montréal, Quebec, Canada; Montréal Heart Institute, Montréal, Quebec, Canada
| | - David B Keator
- Department of Psychiatry and Human Behavior, University of California, Irvine, CA, USA
| | | | - Camille Maumet
- Inria, Univ Rennes, CNRS, Inserm - IRISA UMR 6074, Empenn ERL U 1228, Rennes, France
| | - Cyril R Pernet
- Neurobiology Research Unit, Rigshospitalet, Copenhagen, Denmark
| | - Franco Pestilli
- Psychological & Brain Sciences, Indiana University, Bloomington, IN, USA; Department of Psychology, The University of Texas at Austin, Austin, TX, USA
| | - Nazek Queder
- Montreal Neurological Institute-Hospital, McGill University, Montréal, Quebec, Canada; Department of Neurobiology and Behavior, University of California, Irvine, CA, USA
| | - Tina Schmitt
- Neuroimaging Unit, Carl-von-Ossietzky Universität, Oldenburg, Germany
| | - Weronika Sójka
- Faculty of Philosophy and Social Sciences, Nicolaus Copernicus University, Toruń, Poland
| | - Adina S Wagner
- Institute for Neuroscience and Medicine, Research Centre Juelich, Germany
| | | | - Jochem W Rieger
- Neuroimaging Unit, Carl-von-Ossietzky Universität, Oldenburg, Germany; Department of Psychology, Carl-von-Ossietzky Universität, Oldenburg, Germany.
| |
Collapse
|
5
|
Drude N, Martinez-Gamboa L, Haven T, Holman C, Holst M, Kniffert S, McCann S, Rackoll T, Schulz R, Weschke S. Finding the best fit for improving reproducibility: reflections from the QUEST Center for Responsible Research. BMC Res Notes 2022; 15:270. [PMID: 35922820 PMCID: PMC9351171 DOI: 10.1186/s13104-022-06108-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Accepted: 06/14/2022] [Indexed: 11/10/2022] Open
Abstract
Increasing the reproducibility and trustworthiness of biomedical research requires engaging stakeholders from all levels in an institutional setting. The QUEST Center for Responsible Research aims to develop and implement new approaches to improve the culture and practice of research, tailored to the needs of these stakeholders. Members of the QUEST Center organised a brainstorm to reflect on the challenges and new opportunities encountered in implementing different projects through QUEST and share the lessons that working groups have learned over the first five years. The authors informally surveyed and interviewed working groups where relevant and highlight common themes that have influenced the success of many projects, including top-down and bottom-up engagement, managing expectations, the availability of expertise, ensuring sustainability, and considering incentives. The commentary authors conclude by encouraging the research community to view initiatives that promote reproducibility not as a one-size-fits-all undertaking, but rather as an opportunity to unite stakeholders and customise drivers of cultural change.
Collapse
Affiliation(s)
- Natascha Drude
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Lorena Martinez-Gamboa
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Tamarinde Haven
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany.
| | - Constance Holman
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Martin Holst
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany.,Institute of Ethics, History and Philosophy of Medicine, Medizinische Hochschule Hannover, Carl-Neuberg-Str. 1, 30625, Hannover, Germany
| | - Silke Kniffert
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Sarah McCann
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Torsten Rackoll
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Robert Schulz
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Sarah Weschke
- QUEST Center for Responsible Research, Berlin Institute of Health at Charité - Universitätsmedizin Berlin, Charitéplatz 1, 10117, Berlin, Germany
| |
Collapse
|
6
|
Abstract
The risk of accidental or deliberate misuse of biological research is increasing as biotechnology advances. As open science becomes widespread, we must consider its impact on those risks and develop solutions that ensure security while facilitating scientific progress. Here, we examine the interaction between open science practices and biosecurity and biosafety to identify risks and opportunities for risk mitigation. Increasing the availability of computational tools, datasets, and protocols could increase risks from research with misuse potential. For instance, in the context of viral engineering, open code, data, and materials may increase the risk of release of enhanced pathogens. For this dangerous subset of research, both open science and biosecurity goals may be achieved by using access-controlled repositories or application programming interfaces. While preprints accelerate dissemination of findings, their increased use could challenge strategies for risk mitigation at the publication stage. This highlights the importance of oversight earlier in the research lifecycle. Preregistration of research, a practice promoted by the open science community, provides an opportunity for achieving biosecurity risk assessment at the conception of research. Open science and biosecurity experts have an important role to play in enabling responsible research with maximal societal benefit.
Collapse
Affiliation(s)
- James Andrew Smith
- Botnar Research Centre and Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, United Kingdom
- National Institute for Health Research Oxford Biomedical Research Centre, John Radcliffe Hospital, Oxford, United Kingdom
| | - Jonas B. Sandbrink
- Nuffield Department of Medicine, University of Oxford, Oxford, United Kingdom
- Future of Humanity Institute, University of Oxford, Oxford, United Kingdom
| |
Collapse
|