1
|
Edfeldt K, Edwards AM, Engkvist O, Günther J, Hartley M, Hulcoop DG, Leach AR, Marsden BD, Menge A, Misquitta L, Müller S, Owen DR, Schütt KT, Skelton N, Steffen A, Tropsha A, Vernet E, Wang Y, Wellnitz J, Willson TM, Clevert DA, Haibe-Kains B, Schiavone LH, Schapira M. A data science roadmap for open science organizations engaged in early-stage drug discovery. Nat Commun 2024; 15:5640. [PMID: 38965235 PMCID: PMC11224410 DOI: 10.1038/s41467-024-49777-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2023] [Accepted: 06/12/2024] [Indexed: 07/06/2024] Open
Abstract
The Structural Genomics Consortium is an international open science research organization with a focus on accelerating early-stage drug discovery, namely hit discovery and optimization. We, as many others, believe that artificial intelligence (AI) is poised to be a main accelerator in the field. The question is then how to best benefit from recent advances in AI and how to generate, format and disseminate data to enable future breakthroughs in AI-guided drug discovery. We present here the recommendations of a working group composed of experts from both the public and private sectors. Robust data management requires precise ontologies and standardized vocabulary while a centralized database architecture across laboratories facilitates data integration into high-value datasets. Lab automation and opening electronic lab notebooks to data mining push the boundaries of data sharing and data modeling. Important considerations for building robust machine-learning models include transparent and reproducible data processing, choosing the most relevant data representation, defining the right training and test sets, and estimating prediction uncertainty. Beyond data-sharing, cloud-based computing can be harnessed to build and disseminate machine-learning models. Important vectors of acceleration for hit and chemical probe discovery will be (1) the real-time integration of experimental data generation and modeling workflows within design-make-test-analyze (DMTA) cycles openly, and at scale and (2) the adoption of a mindset where data scientists and experimentalists work as a unified team, and where data science is incorporated into the experimental design.
Collapse
Affiliation(s)
- Kristina Edfeldt
- Structural Genomics Consortium, Department of Medicine, Karolinska University Hospital and Karolinska Institutet, Stockholm, Sweden
| | - Aled M Edwards
- Structural Genomics Consortium, University of Toronto, Toronto, ON, Canada
| | - Ola Engkvist
- Discovery Sciences, R&D, AstraZeneca, Gothenburg, Sweden & Department of Computer Science and Engineering, Chalmers University of Technology, Gothenburg, Sweden
| | - Judith Günther
- Bayer AG Research and Development, Computational Molecular Design, Berlin, Germany
| | - Matthew Hartley
- European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Wellcome Genome Campus, Hinxton, UK
| | - David G Hulcoop
- Open Targets, Wellcome Genome Campus, Hinxton, Cambridgeshire, UK
- European Bioinformatics Institute (EMBL-EBI), Wellcome Genome Campus, Hinxton, Cambridge, UK
| | - Andrew R Leach
- European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Wellcome Genome Campus, Hinxton, UK
| | - Brian D Marsden
- Centre for Medicines Discovery, NDM, University of Oxford, Oxford, UK
| | - Amelie Menge
- Institute of Pharmaceutical Chemistry, Johann Wolfgang Goethe University, Frankfurt am Main, 60438, Germany & Structural Genomics Consortium (SGC), Buchmann Institute for Life Sciences, Johann Wolfgang Goethe University, Frankfurt am Main, Germany
| | - Leonie Misquitta
- National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
| | - Susanne Müller
- Institute of Pharmaceutical Chemistry, Johann Wolfgang Goethe University, Frankfurt am Main, 60438, Germany & Structural Genomics Consortium (SGC), Buchmann Institute for Life Sciences, Johann Wolfgang Goethe University, Frankfurt am Main, Germany
| | - Dafydd R Owen
- Pfizer Worldwide Research, Development & Medical, Cambridge, MA, USA
| | - Kristof T Schütt
- Pfizer, Worldwide Research, Development and Medical, Machine Learning & Computational Sciences, Berlin, Germany
| | - Nicholas Skelton
- Department of Discovery Chemistry, Genentech, Inc., South San Francisco, CA, USA
| | - Andreas Steffen
- Pfizer, Worldwide Research, Development and Medical, Machine Learning & Computational Sciences, Berlin, Germany
| | - Alexander Tropsha
- Laboratory for Molecular Modeling, Division of Chemical Biology and Medicinal Chemistry, UNC Eshelman School of Pharmacy, University of North Carolina, Chapel Hill, North Carolina, USA
| | - Erik Vernet
- Digital Science & Innovation, Novo Nordisk A/S, Maaloev, Denmark
| | - Yanli Wang
- National Library of Medicine, National Institutes of Health, Bethesda, MD, USA
| | - James Wellnitz
- Laboratory for Molecular Modeling, Division of Chemical Biology and Medicinal Chemistry, UNC Eshelman School of Pharmacy, University of North Carolina, Chapel Hill, North Carolina, USA
| | - Timothy M Willson
- Structural Genomics Consortium, UNC Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Djork-Arné Clevert
- Pfizer, Worldwide Research, Development and Medical, Machine Learning & Computational Sciences, Berlin, Germany.
| | - Benjamin Haibe-Kains
- Structural Genomics Consortium, University of Toronto, Toronto, ON, Canada.
- Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada.
- Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada.
- Vector Institute for Artificial Intelligence, Toronto, ON, Canada.
| | | | - Matthieu Schapira
- Structural Genomics Consortium, University of Toronto, Toronto, ON, Canada.
- Department of Pharmacology & Toxicology, University of Toronto, Toronto, ON, Canada.
| |
Collapse
|
2
|
Han H. Using measurement alignment in research on adolescence involving multiple groups: A brief tutorial with R. JOURNAL OF RESEARCH ON ADOLESCENCE : THE OFFICIAL JOURNAL OF THE SOCIETY FOR RESEARCH ON ADOLESCENCE 2024; 34:235-242. [PMID: 37837225 DOI: 10.1111/jora.12891] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 06/29/2023] [Accepted: 09/22/2023] [Indexed: 10/15/2023]
Abstract
Measurement alignment adjusts factor loadings and intercepts across different groups to achieve measurement invariance, which assumes the equal measurement model is validated across different groups. It should be achieved for validly conducting analysis and comparison in studies involving multiple groups, such as cross-cultural or cross-national studies. In this paper, I presented how to conduct measurement alignment with R. In addition to measurement alignment, I explained how to perform the Monte Carlo simulation to test the consistency and validity of alignment results and factor score calculation to facilitate further statistical analysis. A tutorial R code that implements all described procedures is freely shared via GitHub to inform readers who intend to use the alignment technique in their research projects.
Collapse
Affiliation(s)
- Hyemin Han
- Educational Psychology Program, University of Alabama, Tuscaloosa, Alabama, USA
| |
Collapse
|
3
|
Plante J, Langerwerf L, Klopper M, Rhon DI, Young JL. Evaluation of Transparency and Openness Guidelines in Physical Therapist Journals. Phys Ther 2024; 104:pzad133. [PMID: 37815940 DOI: 10.1093/ptj/pzad133] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Revised: 06/05/2023] [Accepted: 08/21/2023] [Indexed: 10/12/2023]
Abstract
OBJECTIVE The goals of this study were to evaluate the extent that physical therapist journals support open science research practices by adhering to the Transparency and Openness Promotion (TOP) guidelines and to assess the relationship between journal scores and their respective journal impact factor (JIF). METHODS Scimago, mapping studies, the National Library of Medicine, and journal author guidelines were searched to identify physical therapist journals for inclusion. Journals were graded on 10 standards (29 available total points) related to transparency with data, code, research materials, study design and analysis, preregistration of studies and statistical analyses, replication, and open science badges. The relationship between journal transparency and openness scores and their JIF was determined. RESULTS Thirty-five journals' author guidelines were assigned transparency and openness factor scores. The median score (interquartile range) across journals was 3.00 out of 29 (3.00) points (for all journals the scores ranged from 0 to 8). The 2 standards with the highest degree of implementation were design and analysis transparency (reporting guidelines) and study preregistration. No journals reported on code transparency, materials transparency, replication, and open science badges. TOP factor scores were a significant predictor of JIF scores. CONCLUSION There is low implementation of the TOP standards by physical therapist journals. TOP factor scores demonstrated predictive abilities for JIF scores. Policies from journals must improve to make open science practices the standard in research. Journals are in an influential position to guide practices that can improve the rigor of publication which, ultimately, enhances the evidence-based information used by physical therapists. IMPACT Transparent, open, and reproducible research will move the profession forward by improving the quality of research and increasing the confidence in results for implementation in clinical care.
Collapse
Affiliation(s)
- Jacqueline Plante
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
| | - Leigh Langerwerf
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
| | - Mareli Klopper
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
| | - Daniel I Rhon
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
- Department of Rehabilitation Medicine, School of Medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland, USA
| | - Jodi L Young
- Department of Physical Therapy, Doctor of Science in Physical Therapy Program, Bellin College, Green Bay, Wisconsin, USA
| |
Collapse
|
4
|
Larsen SA, Asbury K, Coventry WL, Hart SA, Little CW, Petrill SA. Measuring CHAOS? Evaluating the short-form Confusion, Hubbub And Order Scale. COLLABRA. PSYCHOLOGY 2023; 9:77837. [PMID: 38528944 PMCID: PMC10961925 DOI: 10.1525/collabra.77837] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/27/2024]
Abstract
The Confusion, Hubbub and Order Scale (CHAOS) - short form - is a survey tool intended to capture information about home environments. It is widely used in studies of child and adolescent development and psychopathology, particularly twin studies. The original long form of the scale comprised 15 items and was validated in a sample of infants in the 1980s. The short form of the scale was developed in the late 1990s and contains six items, including four from the original scale, and two new items. This short form has not been validated and is the focus of this study. We use five samples drawn from twin studies in Australia, the UK, and the USA, and examine measurement invariance of the CHAOS short-form. We first compare alternate confirmatory factor models for each group; we next test between-group configural, metric and scalar invariance; finally, we examine predictive validity of the scale under different conditions. We find evidence that a two-factor configuration of the six items is more appropriate than the commonly used one-factor model. Second, we find measurement non-invariance across groups at the metric invariance step, with items performing differently depending on the sample. We also find inconsistent results in tests of predictive validity using family-level socioeconomic status and academic achievement as criterion variables. The results caution the continued use of the short-form CHAOS in its current form and recommend future revisions and development of the scale for use in developmental research.
Collapse
Affiliation(s)
| | | | | | - Sara A. Hart
- Department of Psychology, Florida State University, USA
- Florida Center for Reading Research, Florida State University, USA
| | - Callie W. Little
- Florida Center for Reading Research, Florida State University, USA
| | | |
Collapse
|
5
|
Huff M, Bongartz EC. Low Research-Data Availability in Educational-Psychology Journals: No Indication of Effective Research-Data Policies. ADVANCES IN METHODS AND PRACTICES IN PSYCHOLOGICAL SCIENCE 2023. [DOI: 10.1177/25152459231156419] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/30/2023]
Abstract
Research-data availability contributes to the transparency of the research process and the credibility of educational-psychology research and science in general. Recently, there have been many initiatives to increase the availability and quality of research data. Many research institutions have adopted research-data policies. This increased awareness might have raised the sharing of research data in empirical articles. To test this idea, we coded 1,242 publications from six educational-psychology journals and the psychological journal Cognition (as a baseline) published in 2018 and 2020. Research-data availability was low (3.85% compared with 62.74% in Cognition) but has increased from 0.32% (2018) to 7.16% (2020). However, neither the data-transparency level of the journal nor the existence of an official research-data policy on the level of the corresponding author’s institution was related to research-data availability. We discuss the consequences of these findings for institutional research-data-management processes.
Collapse
|
6
|
Cook BG, Fleming JI, Hart SA, Lane KL, Therrien WJ, van Dijk W, Wilson SE. A How-To Guide for Open-Science Practices in Special Education Research. REMEDIAL AND SPECIAL EDUCATION : RASE 2022; 43:270-280. [PMID: 36052401 PMCID: PMC9426775 DOI: 10.1177/07419325211019100] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Open-science reforms, which aim to increase credibility and access of research, have the potential to benefit the research base in special education, as well as practice and policy informed by that research base. Awareness of open science is increasing among special education researchers. However, relatively few researchers in the field have experience using multiple open-science practices, and few practical guidelines or resources have been tailored to special education researchers to support their exploration and adoption of open science. In this paper, we described and provided guidelines and resources for applying five core open-science practices-preregistration, registered reports, data sharing, materials sharing, and open-access publishing-in special education research.
Collapse
|
8
|
Steinbrenner JR, McIntyre N, Rentschler LF, Pearson JN, Luelmo P, Jaramillo ME, Boyd BA, Wong C, Nowell SW, Odom SL, Hume KA. Patterns in reporting and participant inclusion related to race and ethnicity in autism intervention literature: Data from a large-scale systematic review of evidence-based practices. AUTISM : THE INTERNATIONAL JOURNAL OF RESEARCH AND PRACTICE 2022; 26:2026-2040. [PMID: 35068190 PMCID: PMC9596958 DOI: 10.1177/13623613211072593] [Citation(s) in RCA: 39] [Impact Index Per Article: 19.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
There are marked racial and ethnic disparities in diagnosis and services for individuals on the autism spectrum, yet race and ethnicity are underreported and underexamined in autism research. The current study examines the reporting of race and ethnicity and the inclusion of participants across racial and ethnic groups in studies included in a large-scale systematic review of autism intervention research (1990–2017). Trained research assistants reviewed 1013 articles and extracted data on the reporting of race and ethnicity data and the inclusion of participants from different racial and ethnic categories from each article. Only 25% of the articles reported any data on race and ethnicity and reporting over time has slowly increased across the 28 years of the review. Descriptive statistics suggest that race and ethnicity reporting varied by study design, intervention, and outcomes. In studies with reported data, White participants had the highest rate of participation (64.8%), with a large gap between the next highest rates of participation, which were among Hispanic/Latino (9.4%), Black (7.7%), and Asian (6.4%) participants. The lack of reporting and the limited inclusion of participants across minoritized racial and ethnic groups are concerning and suggest a need to examine practices in autism research from planning to dissemination.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | - Connie Wong
- California State University, Northridge, USA
| | | | | | - Kara A Hume
- University of North Carolina at Chapel Hill, USA
| |
Collapse
|
9
|
Fleming JI, Wilson SE, Hart SA, Therrien WJ, Cook BG. Open Accessibility in Education Research: Enhancing the Credibility, Equity, Impact, and Efficiency of Research. EDUCATIONAL PSYCHOLOGIST 2021; 56:110-121. [PMID: 35582472 PMCID: PMC9109832 DOI: 10.1080/00461520.2021.1897593] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/17/2023]
Abstract
Openness is a foundational principle in science. Making the tools and products of scientific research openly accessible advances core aims and values of education researchers, such as the credibility, equity, impact, and efficiency of research. The digital revolution has expanded opportunities for providing greater access to research. In this article, we examine three open-science practices-open data and code, open materials, and open access-that education researchers can use to increase accessibility to the tools and products of research in the field. For each open-science practice, we discuss what the practice is and how it works, its primary benefits, some important limitations and challenges, and two thorny issues.
Collapse
|