1
|
Brechtmann F, Bechtler T, Londhe S, Mertes C, Gagneur J. Evaluation of input data modality choices on functional gene embeddings. NAR Genom Bioinform 2023; 5:lqad095. [PMID: 37942285 PMCID: PMC10629286 DOI: 10.1093/nargab/lqad095] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2023] [Revised: 09/07/2023] [Accepted: 09/28/2023] [Indexed: 11/10/2023] Open
Abstract
Functional gene embeddings, numerical vectors capturing gene function, provide a promising way to integrate functional gene information into machine learning models. These embeddings are learnt by applying self-supervised machine-learning algorithms on various data types including quantitative omics measurements, protein-protein interaction networks and literature. However, downstream evaluations comparing alternative data modalities used to construct functional gene embeddings have been lacking. Here we benchmarked functional gene embeddings obtained from various data modalities for predicting disease-gene lists, cancer drivers, phenotype-gene associations and scores from genome-wide association studies. Off-the-shelf predictors trained on precomputed embeddings matched or outperformed dedicated state-of-the-art predictors, demonstrating their high utility. Embeddings based on literature and protein-protein interactions inferred from low-throughput experiments outperformed embeddings derived from genome-wide experimental data (transcriptomics, deletion screens and protein sequence) when predicting curated gene lists. In contrast, they did not perform better when predicting genome-wide association signals and were biased towards highly-studied genes. These results indicate that embeddings derived from literature and low-throughput experiments appear favourable in many existing benchmarks because they are biased towards well-studied genes and should therefore be considered with caution. Altogether, our study and precomputed embeddings will facilitate the development of machine-learning models in genetics and related fields.
Collapse
Affiliation(s)
- Felix Brechtmann
- TUM School of Computation, Information and Technology, Technical University of Munich, Garching, Germany
- Munich Center for Machine Learning, Munich, Germany
| | - Thibault Bechtler
- TUM School of Computation, Information and Technology, Technical University of Munich, Garching, Germany
| | - Shubhankar Londhe
- TUM School of Computation, Information and Technology, Technical University of Munich, Garching, Germany
| | - Christian Mertes
- TUM School of Computation, Information and Technology, Technical University of Munich, Garching, Germany
- Munich Data Science Institute, Technical University of Munich, Garching, Germany
- Institute of Human Genetics, School of Medicine, Technical University of Munich, Munich, Germany
| | - Julien Gagneur
- TUM School of Computation, Information and Technology, Technical University of Munich, Garching, Germany
- Institute of Human Genetics, School of Medicine, Technical University of Munich, Munich, Germany
- Computational Health Center, Helmholtz Center Munich, Neuherberg, Germany
| |
Collapse
|
2
|
Krafczyk MS, Shi A, Bhaskar A, Marinov D, Stodden V. Learning from reproducing computational results: introducing three principles and the Reproduction Package. PHILOSOPHICAL TRANSACTIONS. SERIES A, MATHEMATICAL, PHYSICAL, AND ENGINEERING SCIENCES 2021; 379:20200069. [PMID: 33775145 PMCID: PMC8059663 DOI: 10.1098/rsta.2020.0069] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Accepted: 01/20/2021] [Indexed: 06/12/2023]
Abstract
We carry out efforts to reproduce computational results for seven published articles and identify barriers to computational reproducibility. We then derive three principles to guide the practice and dissemination of reproducible computational research: (i) Provide transparency regarding how computational results are produced; (ii) When writing and releasing research software, aim for ease of (re-)executability; (iii) Make any code upon which the results rely as deterministic as possible. We then exemplify these three principles with 12 specific guidelines for their implementation in practice. We illustrate the three principles of reproducible research with a series of vignettes from our experimental reproducibility work. We define a novel Reproduction Package, a formalism that specifies a structured way to share computational research artifacts that implements the guidelines generated from our reproduction efforts to allow others to build, reproduce and extend computational science. We make our reproduction efforts in this paper publicly available as exemplar Reproduction Packages. This article is part of the theme issue 'Reliability and reproducibility in computational science: implementing verification, validation and uncertainty quantification in silico'.
Collapse
Affiliation(s)
- M. S. Krafczyk
- University of Illinois at Urbana-Champaign, Urbana, IL, USA
| | - A. Shi
- University of Illinois at Urbana-Champaign, Urbana, IL, USA
| | - A. Bhaskar
- University of Illinois at Urbana-Champaign, Urbana, IL, USA
| | - D. Marinov
- University of Illinois at Urbana-Champaign, Urbana, IL, USA
| | - V. Stodden
- University of Illinois at Urbana-Champaign, Urbana, IL, USA
| |
Collapse
|
3
|
Strickland B, Mercier H. Bias Neglect: A Blind Spot in the Evaluation of Scientific Results. Q J Exp Psychol (Hove) 2014; 67:570-80. [DOI: 10.1080/17470218.2013.821510] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
Abstract
Experimenter bias occurs when scientists' hypotheses influence their results, even if involuntarily. Meta-analyses have suggested that in some domains, such as psychology, up to a third of the studies could be unreliable due to such biases. A series of experiments demonstrates that while people are aware of the possibility that scientists can be more biased when the conclusions of their experiments fit their initial hypotheses, they robustly fail to appreciate that they should also be more sceptical of such results. This is true even when participants read descriptions of studies that have been shown to be biased. Moreover, participants take other sources of bias—such as financial incentives—into account, showing that this bias neglect may be specific to theory-driven hypothesis testing. In combination with a common style of scientific reporting, bias neglect could lead the public to accept premature conclusions.
Collapse
|
4
|
Park IU, Peacey MW, Munafò MR. Modelling the effects of subjective and objective decision making in scientific peer review. Nature 2013; 506:93-6. [PMID: 24305052 DOI: 10.1038/nature12786] [Citation(s) in RCA: 70] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2013] [Accepted: 10/16/2013] [Indexed: 11/09/2022]
Abstract
The objective of science is to advance knowledge, primarily in two interlinked ways: circulating ideas, and defending or criticizing the ideas of others. Peer review acts as the gatekeeper to these mechanisms. Given the increasing concern surrounding the reproducibility of much published research, it is critical to understand whether peer review is intrinsically susceptible to failure, or whether other extrinsic factors are responsible that distort scientists' decisions. Here we show that even when scientists are motivated to promote the truth, their behaviour may be influenced, and even dominated, by information gleaned from their peers' behaviour, rather than by their personal dispositions. This phenomenon, known as herding, subjects the scientific community to an inherent risk of converging on an incorrect answer and raises the possibility that, under certain conditions, science may not be self-correcting. We further demonstrate that exercising some subjectivity in reviewer decisions, which serves to curb the herding process, can be beneficial for the scientific community in processing available information to estimate truth more accurately. By examining the impact of different models of reviewer decisions on the dynamic process of publication, and thereby on eventual aggregation of knowledge, we provide a new perspective on the ongoing discussion of how the peer-review process may be improved.
Collapse
Affiliation(s)
- In-Uck Park
- 1] Department of Economics, University of Bristol, Bristol BS8 1TN, UK [2] Department of Economics, Sungkyunkwan University, Seoul 110-745, South Korea
| | - Mike W Peacey
- 1] Department of Economics, University of Bristol, Bristol BS8 1TN, UK [2] Department of Economics, University of Bath, Bath BA2 7AY, UK
| | - Marcus R Munafò
- 1] MRC Integrative Epidemiology Unit (IEU), University of Bristol, Bristol BS8 1BN, UK [2] UK Centre for Tobacco and Alcohol Studies, University of Bristol, Bristol BS8 1TU, UK [3] School of Experimental Psychology, University of Bristol, Bristol BS8 1TU, UK
| |
Collapse
|
5
|
Jager LR, Leek JT. Rejoinder: An estimate of the science-wise false discovery rate and application to the top medical literature. Biostatistics 2013. [DOI: 10.1093/biostatistics/kxt038] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/14/2022] Open
|
6
|
Ioannidis JP, Hozo I, Djulbegovic B. Optimal type I and type II error pairs when the available sample size is fixed. J Clin Epidemiol 2013; 66:903-910.e2. [DOI: 10.1016/j.jclinepi.2013.03.002] [Citation(s) in RCA: 22] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2012] [Revised: 02/13/2013] [Accepted: 03/12/2013] [Indexed: 11/29/2022]
|
7
|
Ioannidis JPA. This I believe in genetics: discovery can be a nuisance, replication is science, implementation matters. Front Genet 2013; 4:33. [PMID: 23505393 PMCID: PMC3596761 DOI: 10.3389/fgene.2013.00033] [Citation(s) in RCA: 23] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2013] [Accepted: 02/27/2013] [Indexed: 11/18/2022] Open
Affiliation(s)
- John P A Ioannidis
- Department of Medicine, Stanford Prevention Research Center, Stanford University School of Medicine Stanford, CA, USA ; Department of Health Research and Policy, Stanford Prevention Research Center, Stanford University School of Medicine Stanford, CA, USA ; Department of Statistics, Stanford University School of Humanities and Sciences Stanford, CA, USA
| |
Collapse
|
8
|
Eckhart AD, Beebe K, Milburn M. Metabolomics as a key integrator for "omic" advancement of personalized medicine and future therapies. Clin Transl Sci 2012; 5:285-8. [PMID: 22686208 DOI: 10.1111/j.1752-8062.2011.00388.x] [Citation(s) in RCA: 38] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022] Open
Abstract
Investigation into biological complexity, whether for a better understanding of disease or drug process, is a monumental task plaguing investigators. The lure of "omic" technologies for circumventing much of these challenges has led to widespread efforts and adoption. It is becoming clearer that a single "omic" approach (e.g., genomics) is often insufficient for completely defining the complexity in these biological systems. Hence, there is an increasing awareness that a "systems" approach will serve to increase resolution and confidence and provide a strong foundation for further hypothesis-driven investigation. Although certain metabolites are already considered clinically important, the profiling of metabolites via metabolomics (the profiling of metabolites to fully characterize metabolic pathways) is the most recent to mature of these "omic" technologies and has been only recently adopted as compared to genomic or proteomic approaches in systems inquiries. Recent reports suggest that this "omic" may well be a key data stream in systems investigations for endeavors in personalized medicine and biomarker identification, as it seems most closely relevant to the phenotype.
Collapse
Affiliation(s)
- Andrea D Eckhart
- Center for Translational Medicine, Thomas Jefferson University, Philadelphia, Pennsylvania, USA.
| | | | | |
Collapse
|
9
|
Yu X, Wallqvist A, Reifman J. Inferring high-confidence human protein-protein interactions. BMC Bioinformatics 2012; 13:79. [PMID: 22558947 PMCID: PMC3416704 DOI: 10.1186/1471-2105-13-79] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2011] [Accepted: 05/04/2012] [Indexed: 01/09/2023] Open
Abstract
Background As numerous experimental factors drive the acquisition, identification, and interpretation of protein-protein interactions (PPIs), aggregated assemblies of human PPI data invariably contain experiment-dependent noise. Ascertaining the reliability of PPIs collected from these diverse studies and scoring them to infer high-confidence networks is a non-trivial task. Moreover, a large number of PPIs share the same number of reported occurrences, making it impossible to distinguish the reliability of these PPIs and rank-order them. For example, for the data analyzed here, we found that the majority (>83%) of currently available human PPIs have been reported only once. Results In this work, we proposed an unsupervised statistical approach to score a set of diverse, experimentally identified PPIs from nine primary databases to create subsets of high-confidence human PPI networks. We evaluated this ranking method by comparing it with other methods and assessing their ability to retrieve protein associations from a number of diverse and independent reference sets. These reference sets contain known biological data that are either directly or indirectly linked to interactions between proteins. We quantified the average effect of using ranked protein interaction data to retrieve this information and showed that, when compared to randomly ranked interaction data sets, the proposed method created a larger enrichment (~134%) than either ranking based on the hypergeometric test (~109%) or occurrence ranking (~46%). Conclusions From our evaluations, it was clear that ranked interactions were always of value because higher-ranked PPIs had a higher likelihood of retrieving high-confidence experimental data. Reducing the noise inherent in aggregated experimental PPIs via our ranking scheme further increased the accuracy and enrichment of PPIs derived from a number of biologically relevant data sets. These results suggest that using our high-confidence protein interactions at different levels of confidence will help clarify the topological and biological properties associated with human protein networks.
Collapse
Affiliation(s)
- Xueping Yu
- Biotechnology High-Performance Computing Software Applications Institute, Telemedicine and Advanced Technology Research Center, U.S. Army Medical Research and Materiel Command, Ft. Detrick, MD 21702, USA
| | | | | |
Collapse
|
10
|
Ledgerwood A, Sherman JW. Short, Sweet, and Problematic? The Rise of the Short Report in Psychological Science. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2012; 7:60-6. [DOI: 10.1177/1745691611427304] [Citation(s) in RCA: 41] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
Our field has witnessed a rapid increase in the appeal and prevalence of the short report format over the last two decades. In this article, we discuss both the benefits and drawbacks of the trend toward shorter and faster publications. Although the short report format can help us cope with ever-increasing time constraints; ease the burden on hiring, promotion, and tenure committees; speed the publication of our findings; and promote the dissemination of research beyond the borders of our discipline, it can also exacerbate problems with publication bias and selective reporting, decrease theoretical integration within our science, and risk overemphasizing colorful effects relative to basic processes. In the face of these challenges, we believe it is essential to find ways to preserve the advantages of the short-and-fast approach while minimizing its disadvantages and while acknowledging the complementary and critical importance of longer articles in advancing the field.
Collapse
|
11
|
Affiliation(s)
- Samuel Arbesman
- Department of Health Care Policy, Harvard Medical School, Boston, Massachusetts, United States of America.
| | | |
Collapse
|
12
|
Weckwerth W. Unpredictability of metabolism--the key role of metabolomics science in combination with next-generation genome sequencing. Anal Bioanal Chem 2011; 400:1967-78. [PMID: 21556754 PMCID: PMC3098350 DOI: 10.1007/s00216-011-4948-9] [Citation(s) in RCA: 51] [Impact Index Per Article: 3.9] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2011] [Revised: 03/16/2011] [Accepted: 03/22/2011] [Indexed: 12/25/2022]
Abstract
Next-generation sequencing provides technologies which sequence whole prokaryotic and eukaryotic genomes in days, perform genome-wide association studies, chromatin immunoprecipitation followed by sequencing and RNA sequencing for transcriptome studies. An exponentially growing volume of sequence data can be anticipated, yet functional interpretation does not keep pace with the amount of data produced. In principle, these data contain all the secrets of living systems, the genotype-phenotype relationship. Firstly, it is possible to derive the structure and connectivity of the metabolic network from the genotype of an organism in the form of the stoichiometric matrix N. This is, however, static information. Strategies for genome-scale measurement, modelling and predicting of dynamic metabolic networks need to be applied. Consequently, metabolomics science--the quantitative measurement of metabolism in conjunction with metabolic modelling--is a key discipline for the functional interpretation of whole genomes and especially for testing the numerical predictions of metabolism based on genome-scale metabolic network models. In this context, a systematic equation is derived based on metabolomics covariance data and the genome-scale stoichiometric matrix which describes the genotype-phenotype relationship.
Collapse
Affiliation(s)
- Wolfram Weckwerth
- Department of Molecular Systems Biology, University of Vienna, Althanstrasse 14, 1090, Vienna, Austria.
| |
Collapse
|
13
|
Pfeiffer T, Almenberg J. Prediction markets and their potential role in biomedical research – A review. Biosystems 2010; 102:71-6. [DOI: 10.1016/j.biosystems.2010.09.005] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2010] [Revised: 08/25/2010] [Accepted: 09/06/2010] [Indexed: 11/26/2022]
|
14
|
Young SN. Bias in the research literature and conflict of interest: an issue for publishers, editors, reviewers and authors, and it is not just about the money. J Psychiatry Neurosci 2009; 34:412-7. [PMID: 19949717 PMCID: PMC2783432] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 03/01/2023] Open
Affiliation(s)
- Simon N. Young
- Correspondence to: Dr. S.N. Young, Department of Psychiatry, McGill University, 1033 Pine Ave. W, Montréal QC H3A 1A1; fax 514 398-4370;
| |
Collapse
|