An empirical assessment of validation practices for molecular classifiers

Role of diagnostic tests for sepsis in children: a review

Paediatric sepsis has a significant global impact and highly heterogeneous clinical presentation. The clinical pathway encompasses recognition, escalation and de-escalation. In each aspect, diagnostics have a fundamental influence over outcomes in children. Biomarkers can aid in creating a larger low-risk group of children from those in the clinical grey area who would otherwise receive antibiotics 'just in case'. Current biomarkers include C reactive protein and procalcitonin, which are limited in their clinical use to guide appropriate and rapid treatment. Biomarker discovery has focused on single biomarkers, which, so far, have not outperformed current biomarkers, as they fail to recognise the complexity of sepsis. The identification of multiple host biomarkers that may form a panel in a clinical test has the potential to recognise the complexity of sepsis and provide improved diagnostic performance. In this review, we discuss novel biomarkers and novel ways of using existing biomarkers in the assessment and management of sepsis along with the significant challenges in biomarker discovery at present. Validation of biomarkers is made less meaningful due to methodological heterogeneity, including variations in sepsis diagnosis, biomarker cut-off values and patient populations. Therefore, the utilisation of platform studies is necessary to improve the efficiency of biomarkers in clinical practice.

SERS microscopy as a tool for comprehensive biochemical characterization in complex samples

Surface enhanced Raman scattering (SERS) spectra of biomaterials such as cells or tissues can be used to obtain biochemical information from nanoscopic volumes in these heterogeneous samples. This tutorial review discusses the factors that determine the outcome of a SERS experiment in complex bioorganic samples. They are related to the SERS process itself, the possibility to selectively probe certain regions or constituents of a sample, and the retrieval of the vibrational information in order to identify molecules and their interaction. After introducing basic aspects of SERS experiments in the context of biocompatible environments, spectroscopy in typical microscopic settings is exemplified, including the possibilities to combine SERS with other linear and non-linear microscopic tools, and to exploit approaches that improve lateral and temporal resolution. In particular the great variation of data in a SERS experiment calls for robust data analysis tools. Approaches will be introduced that have been originally developed in the field of bioinformatics for the application to omics data and that show specific potential in the analysis of SERS data. They include the use of simulated data and machine learning tools that can yield chemical information beyond achieving spectral classification.

Blood-based biomarkers and novel technologies for the diagnosis of colorectal cancer and adenomas: a narrative review

Aim: Blood-based biomarkers have shown promise for diagnosing colorectal cancer (CRC) and adenomas (CRA). This review summarizes recent studies in this area. Methods: A literature search was undertaken for 01/01/2017-01/03/2023. Criteria included CRC, CRA, liquid-biopsy, blood-based tests and diagnosis. Results: 12,378 studies were reduced to 178 for data extraction. Sixty focused on proteomics, 53 on RNA species, 30 on cfDNA methylation, seven on antigens and autoantibodies and 28 on novel techniques. 169 case control and nine cohort studies. Number of participants ranged 100-54,297, mean age 58.26. CRC sensitivity and specificity ranged 9.10-100% and 20.40-100%, respectively. CRA sensitivity and specificity ranged 8.00-95.70% and 4.00-97.00%, respectively. Conclusion: Sensitive and specific blood-based tests exist for CRC and CRA. However, studies demonstrate heterogenous techniques and reporting quality. Further work should concentrate on validation and meta-analyzes.

Prediction of Deleterious Single Amino Acid Polymorphisms with a Consensus Holdout Sampler

Background

Single Amino Acid Polymorphisms (SAPs) or nonsynonymous Single Nucleotide Variants (nsSNVs) are the most common genetic variations. They result from missense mutations where a single base pair substitution changes the genetic code in such a way that the triplet of bases (codon) at a given position is coding a different amino acid. Since genetic mutations sometimes cause genetic diseases, it is important to comprehend and foresee which variations are harmful and which ones are neutral (not causing changes in the phenotype). This can be posed as a classification problem.

Methods

Computational methods using machine intelligence are gradually replacing repetitive and exceedingly overpriced mutagenic tests. By and large, uneven quality, deficiencies, and irregularities of nsSNVs datasets debase the convenience of artificial intelligence-based methods. Subsequently, strong and more exact approaches are needed to address these problems. In the present work paper, we show a consensus classifier built on the holdout sampler, which appears strong and precise and outflanks all other popular methods.

Results

We produced 100 holdouts to test the structures and diverse classification variables of diverse classifiers during the training phase. The finest performing holdouts were chosen to develop a consensus classifier and tested using a k-fold (1 ≤ k ≤5) cross-validation method. We also examined which protein properties have the biggest impact on the precise prediction of the effects of nsSNVs.

Conclusion

Our Consensus Holdout Sampler outflanks other popular algorithms, and gives excellent results, highly accurate with low standard deviation. The advantage of our method emerges from using a tree of holdouts, where diverse LM/AI-based programs are sampled in diverse ways.

Computational Tools to Assist in Analyzing Effects of the SERPINA1 Gene Variation on Alpha-1 Antitrypsin (AAT)

In the rapidly advancing field of bioinformatics, the development and application of computational tools to predict the effects of single nucleotide variants (SNVs) are shedding light on the molecular mechanisms underlying disorders. Also, they hold promise for guiding therapeutic interventions and personalized medicine strategies in the future. A comprehensive understanding of the impact of SNVs in the SERPINA1 gene on alpha-1 antitrypsin (AAT) protein structure and function requires integrating bioinformatic approaches. Here, we provide a guide for clinicians to navigate through the field of computational analyses which can be applied to describe a novel genetic variant. Predicting the clinical significance of SERPINA1 variation allows clinicians to tailor treatment options for individuals with alpha-1 antitrypsin deficiency (AATD) and related conditions, ultimately improving the patient's outcome and quality of life. This paper explores the various bioinformatic methodologies and cutting-edge approaches dedicated to the assessment of molecular variants of genes and their product proteins using SERPINA1 and AAT as an example.

Predictive Performance of Cell-Free Nucleic Acid-Based Multi-Cancer Early Detection Tests: A Systematic Review

BACKGROUND

Cancer-screening tests that can detect multiple cancer types, or multi-cancer early detection (MCED) tests, have emerged recently as a potential new tool in decreasing cancer morbidity and mortality. Most MCED assays are based on detecting cell-free tumor DNA (CF-DNA) in the blood. MCEDs offer the potential for screening for cancer organ sites with high mortality, both with and without recommended screening. However, their clinical utility has not been established. Before clinical utility can be established, the clinical validity of MCEDs, i.e., their ability to predict cancer status, must be demonstrated. In this study we performed a systematic review of the predictive ability for cancer of cell-free-nucleic acid-based MCED tests.

CONTENT

We searched PubMed for relevant publications from January 2017 to February 2023, using MeSH terms related to multi-cancer detection, circulating DNA, and related concepts. Of 1811 publications assessed, 61 were reviewed in depth and 20 are included in this review. For almost all studies, the cancer cases were assessed at time of diagnosis. Most studies reported specificity (generally 95% or higher) and overall sensitivity (73% median). The median number of cancer types assessed per assay was 5. Many studies also reported sensitivity by stage and/or cancer type. Sensitivity generally increased with stage.

SUMMARY

To date, relatively few published studies have assessed the clinical validity of MCED tests. Most used cancer cases assessed at diagnosis, with generally high specificity and variable sensitivity depending on cancer type and stage. The next steps should be testing in the intended-use population, i.e., asymptomatic persons.

Multilingual RECIST classification of radiology reports using supervised learning

Objectives

The objective of this study is the exploration of Artificial Intelligence and Natural Language Processing techniques to support the automatic assignment of the four Response Evaluation Criteria in Solid Tumors (RECIST) scales based on radiology reports. We also aim at evaluating how languages and institutional specificities of Swiss teaching hospitals are likely to affect the quality of the classification in French and German languages.

Methods

In our approach, 7 machine learning methods were evaluated to establish a strong baseline. Then, robust models were built, fine-tuned according to the language (French and German), and compared with the expert annotation.

Results

The best strategies yield average F1-scores of 90% and 86% respectively for the 2-classes (Progressive/Non-progressive) and the 4-classes (Progressive Disease, Stable Disease, Partial Response, Complete Response) RECIST classification tasks.

Conclusions

These results are competitive with the manual labeling as measured by Matthew's correlation coefficient and Cohen's Kappa (79% and 76%). On this basis, we confirm the capacity of specific models to generalize on new unseen data and we assess the impact of using Pre-trained Language Models (PLMs) on the accuracy of the classifiers.

Development and validation of a dynamic 48-hour in-hospital mortality risk stratification for COVID-19 in a UK teaching hospital: a retrospective cohort study

OBJECTIVES

To develop a disease stratification model for COVID-19 that updates according to changes in a patient's condition while in hospital to facilitate patient management and resource allocation.

DESIGN

In this retrospective cohort study, we adopted a landmarking approach to dynamic prediction of all-cause in-hospital mortality over the next 48 hours. We accounted for informative predictor missingness and selected predictors using penalised regression.

SETTING

All data used in this study were obtained from a single UK teaching hospital.

PARTICIPANTS

We developed the model using 473 consecutive patients with COVID-19 presenting to a UK hospital between 1 March 2020 and 12 September 2020; and temporally validated using data on 1119 patients presenting between 13 September 2020 and 17 March 2021.

PRIMARY AND SECONDARY OUTCOME MEASURES

The primary outcome is all-cause in-hospital mortality within 48 hours of the prediction time. We accounted for the competing risks of discharge from hospital alive and transfer to a tertiary intensive care unit for extracorporeal membrane oxygenation.

RESULTS

Our final model includes age, Clinical Frailty Scale score, heart rate, respiratory rate, oxygen saturation/fractional inspired oxygen ratio, white cell count, presence of acidosis (pH <7.35) and interleukin-6. Internal validation achieved an area under the receiver operating characteristic (AUROC) of 0.90 (95% CI 0.87 to 0.93) and temporal validation gave an AUROC of 0.86 (95% CI 0.83 to 0.88).

CONCLUSIONS

Our model incorporates both static risk factors (eg, age) and evolving clinical and laboratory data, to provide a dynamic risk prediction model that adapts to both sudden and gradual changes in an individual patient's clinical condition. On successful external validation, the model has the potential to be a powerful clinical risk assessment tool.

TRIAL REGISTRATION

The study is registered as 'researchregistry5464' on the Research Registry (www.researchregistry.com).

Clinical prediction models in psychiatry: a systematic review of two decades of progress and challenges

Recent years have seen the rapid proliferation of clinical prediction models aiming to support risk stratification and individualized care within psychiatry. Despite growing interest, attempts to synthesize current evidence in the nascent field of precision psychiatry have remained scarce. This systematic review therefore sought to summarize progress towards clinical implementation of prediction modeling for psychiatric outcomes. We searched MEDLINE, PubMed, Embase, and PsychINFO databases from inception to September 30, 2020, for English-language articles that developed and/or validated multivariable models to predict (at an individual level) onset, course, or treatment response for non-organic psychiatric disorders (PROSPERO: CRD42020216530). Individual prediction models were evaluated based on three key criteria: (i) mitigation of bias and overfitting; (ii) generalizability, and (iii) clinical utility. The Prediction model Risk Of Bias ASsessment Tool (PROBAST) was used to formally appraise each study's risk of bias. 228 studies detailing 308 prediction models were ultimately eligible for inclusion. 94.5% of developed prediction models were deemed to be at high risk of bias, largely due to inadequate or inappropriate analytic decisions. Insufficient internal validation efforts (within the development sample) were also observed, while only one-fifth of models underwent external validation in an independent sample. Finally, our search identified just one published model whose potential utility in clinical practice was formally assessed. Our findings illustrated significant growth in precision psychiatry with promising progress towards real-world application. Nevertheless, these efforts have been inhibited by a preponderance of bias and overfitting, while the generalizability and clinical utility of many published models has yet to be formally established. Through improved methodological rigor during initial development, robust evaluations of reproducibility via independent validation, and evidence-based implementation frameworks, future research has the potential to generate risk prediction tools capable of enhancing clinical decision-making in psychiatric care.

Generalizing predictions to unseen sequencing profiles via deep generative models

Predictive models trained on sequencing profiles often fail to achieve expected performance when externally validated on unseen profiles. While many factors such as batch effects, small data sets, and technical errors contribute to the gap between source and unseen data distributions, it is a challenging problem to generalize the predictive models across studies without any prior knowledge of the unseen data distribution. Here, this study proposes DeepBioGen, a sequencing profile augmentation procedure that characterizes visual patterns of sequencing profiles, generates realistic profiles based on a deep generative model capturing the patterns, and generalizes the subsequent classifiers. DeepBioGen outperforms other methods in terms of enhancing the generalizability of the prediction models on unseen data. The generalized classifiers surpass the state-of-the-art method, evaluated on RNA sequencing tumor expression profiles for anti-PD1 therapy response prediction and WGS human gut microbiome profiles for type 2 diabetes diagnosis.

PREDICTION OF HEREDITARY CANCERS USING NEURAL NETWORKS

Family history is a major risk factor for many types of cancer. Mendelian risk prediction models translate family histories into cancer risk predictions, based on knowledge of cancer susceptibility genes. These models are widely used in clinical practice to help identify high-risk individuals. Mendelian models leverage the entire family history, but they rely on many assumptions about cancer susceptibility genes that are either unrealistic or challenging to validate, due to low mutation prevalence. Training more flexible models, such as neural networks, on large databases of pedigrees can potentially lead to accuracy gains. In this paper we develop a framework to apply neural networks to family history data and investigate their ability to learn inherited susceptibility to cancer. While there is an extensive literature on neural networks and their state-of-the-art performance in many tasks, there is little work applying them to family history data. We propose adaptations of fully-connected neural networks and convolutional neural networks to pedigrees. In data simulated under Mendelian inheritance, we demonstrate that our proposed neural network models are able to achieve nearly optimal prediction performance. Moreover, when the observed family history includes misreported cancer diagnoses, neural networks are able to outperform the Mendelian BRCAPRO model embedding the correct inheritance laws. Using a large dataset of over 200,000 family histories, the Risk Service cohort, we train prediction models for future risk of breast cancer. We validate the models using data from the Cancer Genetics Network.

Predicting genotoxicity of viral vectors for stem cell gene therapy using gene expression-based machine learning

Hematopoietic stem cell gene therapy is emerging as a promising therapeutic strategy for many diseases of the blood and immune system. However, several individuals who underwent gene therapy in different trials developed hematological malignancies caused by insertional mutagenesis. Preclinical assessment of vector safety remains challenging because there are few reliable assays to screen for potential insertional mutagenesis effects in vitro. Here we demonstrate that genotoxic vectors induce a unique gene expression signature linked to stemness and oncogenesis in transduced murine hematopoietic stem and progenitor cells. Based on this finding, we developed the surrogate assay for genotoxicity assessment (SAGA). SAGA classifies integrating retroviral vectors using machine learning to detect this gene expression signature during the course of in vitro immortalization. On a set of benchmark vectors with known genotoxic potential, SAGA achieved an accuracy of 90.9%. SAGA is more robust and sensitive and faster than previous assays and reliably predicts a mutagenic risk for vectors that led to leukemic severe adverse events in clinical trials. Our work provides a fast and robust tool for preclinical risk assessment of gene therapy vectors, potentially paving the way for safer gene therapy trials.

Prognostic Models Predicting Mortality in Preterm Infants: Systematic Review and Meta-analysis

CONTEXT

Prediction models can be a valuable tool in performing risk assessment of mortality in preterm infants.

OBJECTIVE

Summarizing prognostic models for predicting mortality in very preterm infants and assessing their quality.

DATA SOURCES

Medline was searched for all articles (up to June 2020).

STUDY SELECTION

All developed or externally validated prognostic models for mortality prediction in liveborn infants born <32 weeks' gestation and/or <1500 g birth weight were included.

DATA EXTRACTION

Data were extracted by 2 independent authors. Risk of bias (ROB) and applicability assessment was performed by 2 independent authors using Prediction model Risk of Bias Assessment Tool.

RESULTS

One hundred forty-two models from 35 studies reporting on model development and 112 models from 33 studies reporting on external validation were included. ROB assessment revealed high ROB in the majority of the models, most often because of inadequate (reporting of) analysis. Internal and external validation was lacking in 41% and 96% of these models. Meta-analyses revealed an average C-statistic of 0.88 (95% confidence interval [CI]: 0.83-0.91) for the Clinical Risk Index for Babies score, 0.87 (95% CI: 0.81-0.92) for the Clinical Risk Index for Babies II score, and 0.86 (95% CI: 0.78-0.92) for the Score for Neonatal Acute Physiology Perinatal Extension II score.

LIMITATIONS

Occasionally, an external validation study was included, but not the development study, because studies developed in the presurfactant era or general NICU population were excluded.

CONCLUSIONS

Instead of developing additional mortality prediction models for preterm infants, the emphasis should be shifted toward external validation and consecutive adaption of the existing prediction models.

The impact of different sources of heterogeneity on loss of accuracy from genomic prediction models

Cross-study validation (CSV) of prediction models is an alternative to traditional cross-validation (CV) in domains where multiple comparable datasets are available. Although many studies have noted potential sources of heterogeneity in genomic studies, to our knowledge none have systematically investigated their intertwined impacts on prediction accuracy across studies. We employ a hybrid parametric/non-parametric bootstrap method to realistically simulate publicly available compendia of microarray, RNA-seq, and whole metagenome shotgun microbiome studies of health outcomes. Three types of heterogeneity between studies are manipulated and studied: (i) imbalances in the prevalence of clinical and pathological covariates, (ii) differences in gene covariance that could be caused by batch, platform, or tumor purity effects, and (iii) differences in the "true" model that associates gene expression and clinical factors to outcome. We assess model accuracy, while altering these factors. Lower accuracy is seen in CSV than in CV. Surprisingly, heterogeneity in known clinical covariates and differences in gene covariance structure have very limited contributions in the loss of accuracy when validating in new studies. However, forcing identical generative models greatly reduces the within/across study difference. These results, observed consistently for multiple disease outcomes and omics platforms, suggest that the most easily identifiable sources of study heterogeneity are not necessarily the primary ones that undermine the ability to accurately replicate the accuracy of omics prediction models in new studies. Unidentified heterogeneity, such as could arise from unmeasured confounding, may be more important.

Gene Networks Constructed Through Simulated Treatment Learning can Predict Proteasome Inhibitor Benefit in Multiple Myeloma

PURPOSE

Proteasome inhibitors are widely used in treating multiple myeloma, but can cause serious side effects and response varies among patients. It is, therefore, important to gain more insight into which patients will benefit from proteasome inhibitors.

EXPERIMENTAL DESIGN

We introduce simulated treatment learned signatures (STLsig), a machine learning method to identify predictive gene expression signatures. STLsig uses genetically similar patients who have received an alternative treatment to model which patients will benefit more from proteasome inhibitors than from an alternative treatment. STLsig constructs gene networks by linking genes that are synergistic in their ability to predict benefit.

RESULTS

In a dataset of 910 patients with multiple myeloma, STLsig identified two gene networks that together can predict benefit to the proteasome inhibitor, bortezomib. In class "benefit," we found an HR of 0.47 (P = 0.04) in favor of bortezomib, while in class "no benefit," the HR was 0.91 (P = 0.68). Importantly, we observed a similar performance (HR class benefit, 0.46; P = 0.04) in an independent patient cohort. Moreover, this signature also predicts benefit for the proteasome inhibitor, carfilzomib, indicating it is not specific to bortezomib. No equivalent signature can be found when the genes in the signature are excluded from the analysis, indicating that they are essential. Multiple genes in the signature are linked to working mechanisms of proteasome inhibitors or multiple myeloma disease progression.

CONCLUSIONS

STLsig can identify gene signatures that could aid in treatment decisions for patients with multiple myeloma and provide insight into the biological mechanism behind treatment benefit.

Large-scale benchmark study of survival prediction methods using multi-omics data

Multi-omics data, that is, datasets containing different types of high-dimensional molecular variables, are increasingly often generated for the investigation of various diseases. Nevertheless, questions remain regarding the usefulness of multi-omics data for the prediction of disease outcomes such as survival time. It is also unclear which methods are most appropriate to derive such prediction models. We aim to give some answers to these questions through a large-scale benchmark study using real data. Different prediction methods from machine learning and statistics were applied on 18 multi-omics cancer datasets (35 to 1000 observations, up to 100 000 variables) from the database 'The Cancer Genome Atlas' (TCGA). The considered outcome was the (censored) survival time. Eleven methods based on boosting, penalized regression and random forest were compared, comprising both methods that do and that do not take the group structure of the omics variables into account. The Kaplan-Meier estimate and a Cox model using only clinical variables were used as reference methods. The methods were compared using several repetitions of 5-fold cross-validation. Uno's C-index and the integrated Brier score served as performance metrics. The results indicate that methods taking into account the multi-omics structure have a slightly better prediction performance. Taking this structure into account can protect the predictive information in low-dimensional groups-especially clinical variables-from not being exploited during prediction. Moreover, only the block forest method outperformed the Cox model on average, and only slightly. This indicates, as a by-product of our study, that in the considered TCGA studies the utility of multi-omics data for prediction purposes was limited. Contact:moritz.herrmann@stat.uni-muenchen.de, +49 89 2180 3198 Supplementary information: Supplementary data are available at Briefings in Bioinformatics online. All analyses are reproducible using R code freely available on Github.

A decision-theoretic approach to the evaluation of machine learning algorithms in computational drug discovery

Motivation

Artificial intelligence, trained via machine learning (e.g. neural nets, random forests) or computational statistical algorithms (e.g. support vector machines, ridge regression), holds much promise for the improvement of small-molecule drug discovery. However, small-molecule structure-activity data are high dimensional with low signal-to-noise ratios and proper validation of predictive methods is difficult. It is poorly understood which, if any, of the currently available machine learning algorithms will best predict new candidate drugs.

Results

The quantile-activity bootstrap is proposed as a new model validation framework using quantile splits on the activity distribution function to construct training and testing sets. In addition, we propose two novel rank-based loss functions which penalize only the out-of-sample predicted ranks of high-activity molecules. The combination of these methods was used to assess the performance of neural nets, random forests, support vector machines (regression) and ridge regression applied to 25 diverse high-quality structure-activity datasets publicly available on ChEMBL. Model validation based on random partitioning of available data favours models that overfit and ‘memorize’ the training set, namely random forests and deep neural nets. Partitioning based on quantiles of the activity distribution correctly penalizes extrapolation of models onto structurally different molecules outside of the training data. Simpler, traditional statistical methods such as ridge regression can outperform state-of-the-art machine learning methods in this setting. In addition, our new rank-based loss functions give considerably different results from mean squared error highlighting the necessity to define model optimality with respect to the decision task at hand.

Availability and implementation

All software and data are available as Jupyter notebooks found at https://github.com/owatson/QuantileBootstrap.

Supplementary information

Supplementary data are available at Bioinformatics online.

Variable selection and validation in multivariate modelling

MOTIVATION

Validation of variable selection and predictive performance is crucial in construction of robust multivariate models that generalize well, minimize overfitting and facilitate interpretation of results. Inappropriate variable selection leads instead to selection bias, thereby increasing the risk of model overfitting and false positive discoveries. Although several algorithms exist to identify a minimal set of most informative variables (i.e. the minimal-optimal problem), few can select all variables related to the research question (i.e. the all-relevant problem). Robust algorithms combining identification of both minimal-optimal and all-relevant variables with proper cross-validation are urgently needed.

RESULTS

We developed the MUVR algorithm to improve predictive performance and minimize overfitting and false positives in multivariate analysis. In the MUVR algorithm, minimal variable selection is achieved by performing recursive variable elimination in a repeated double cross-validation (rdCV) procedure. The algorithm supports partial least squares and random forest modelling, and simultaneously identifies minimal-optimal and all-relevant variable sets for regression, classification and multilevel analyses. Using three authentic omics datasets, MUVR yielded parsimonious models with minimal overfitting and improved model performance compared with state-of-the-art rdCV. Moreover, MUVR showed advantages over other variable selection algorithms, i.e. Boruta and VSURF, including simultaneous variable selection and validation scheme and wider applicability.

AVAILABILITY AND IMPLEMENTATION

Algorithms, data, scripts and tutorial are open source and available as an R package ('MUVR') at https://gitlab.com/CarlBrunius/MUVR.git.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

A data-driven interactome of synergistic genes improves network-based cancer outcome prediction

Robustly predicting outcome for cancer patients from gene expression is an important challenge on the road to better personalized treatment. Network-based outcome predictors (NOPs), which considers the cellular wiring diagram in the classification, hold much promise to improve performance, stability and interpretability of identified marker genes. Problematically, reports on the efficacy of NOPs are conflicting and for instance suggest that utilizing random networks performs on par to networks that describe biologically relevant interactions. In this paper we turn the prediction problem around: instead of using a given biological network in the NOP, we aim to identify the network of genes that truly improves outcome prediction. To this end, we propose SyNet, a gene network constructed ab initio from synergistic gene pairs derived from survival-labelled gene expression data. To obtain SyNet, we evaluate synergy for all 69 million pairwise combinations of genes resulting in a network that is specific to the dataset and phenotype under study and can be used to in a NOP model. We evaluated SyNet and 11 other networks on a compendium dataset of >4000 survival-labelled breast cancer samples. For this purpose, we used cross-study validation which more closely emulates real world application of these outcome predictors. We find that SyNet is the only network that truly improves performance, stability and interpretability in several existing NOPs. We show that SyNet overlaps significantly with existing gene networks, and can be confidently predicted (~85% AUC) from graph-topological descriptions of these networks, in particular the breast tissue-specific network. Due to its data-driven nature, SyNet is not biased to well-studied genes and thus facilitates post-hoc interpretation. We find that SyNet is highly enriched for known breast cancer genes and genes related to e.g. histological grade and tamoxifen resistance, suggestive of a role in determining breast cancer outcome.

PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.

Challenges and emergent solutions for LC-MS/MS based untargeted metabolomics in diseases

In the past decade, advances in liquid chromatography-mass spectrometry (LC-MS) have revolutionized untargeted metabolomics analyses. By mining metabolomes more deeply, researchers are now primed to uncover key metabolites and their associations with diseases. The employment of untargeted metabolomics has led to new biomarker discoveries and a better mechanistic understanding of diseases with applications in precision medicine. However, many major pertinent challenges remain. First, compound identification has been poor, and left an overwhelming number of unidentified peaks. Second, partial, incomplete metabolomes persist due to factors such as limitations in mass spectrometry data acquisition speeds, wide-range of metabolites concentrations, and cellular/tissue/temporal-specific expression changes that confound our understanding of metabolite perturbations. Third, to contextualize metabolites in pathways and biology is difficult because many metabolites partake in multiple pathways, have yet to be described species specificity, or possess unannotated or more-complex functions that are not easily characterized through metabolomics analyses. From a translational perspective, information related to novel metabolite biomarkers, metabolic pathways, and drug targets might be sparser than they should be. Thankfully, significant progress has been made and novel solutions are emerging, achieved through sustained academic and industrial community efforts in terms of hardware, computational, and experimental approaches. Given the rapidly growing utility of metabolomics, this review will offer new perspectives, increase awareness of the major challenges in LC-MS metabolomics that will significantly benefit the metabolomics community and also the broader the biomedical community metabolomics aspire to serve.

Big Data in Public Health: Terminology, Machine Learning, and Privacy

The digital world is generating data at a staggering and still increasing rate. While these "big data" have unlocked novel opportunities to understand public health, they hold still greater potential for research and practice. This review explores several key issues that have arisen around big data. First, we propose a taxonomy of sources of big data to clarify terminology and identify threads common across some subtypes of big data. Next, we consider common public health research and practice uses for big data, including surveillance, hypothesis-generating research, and causal inference, while exploring the role that machine learning may play in each use. We then consider the ethical implications of the big data revolution with particular emphasis on maintaining appropriate care for privacy in a world in which technology is rapidly changing social norms regarding the need for (and even the meaning of) privacy. Finally, we make suggestions regarding structuring teams and training to succeed in working with big data in research and practice.

Platform-independent gene expression signature differentiates sessile serrated adenomas/polyps and hyperplastic polyps of the colon

Background

Sessile serrated adenomas/polyps are distinguished from hyperplastic colonic polyps subjectively by their endoscopic appearance and histological morphology. However, hyperplastic and sessile serrated polyps can have overlapping morphological features resulting in sessile serrated polyps diagnosed as hyperplastic. While sessile serrated polyps can progress into colon cancer, hyperplastic polyps have virtually no risk for colon cancer. Objective measures, differentiating these types of polyps would improve cancer prevention and treatment outcome.

Methods

RNA-seq training data set and Affimetrix, Illumina testing data sets were obtained from Gene Expression Omnibus (GEO). RNA-seq single-end reads were filtered with FastX toolkit. Read mapping to the human genome, gene abundance estimation, and differential expression analysis were performed with Tophat-Cufflinks pipeline. Background correction, normalization, and probe summarization steps for Affimetrix arrays were performed using the robust multi-array method (RMA). For Illumina arrays, log₂-scale expression data was obtained from GEO. Pathway analysis was implemented using Bioconductor package GSAR. To build a platform-independent molecular classifier that accurately differentiates sessile serrated and hyperplastic polyps we developed a new feature selection step. We also developed a simple procedure to classify new samples as either sessile serrated or hyperplastic with a class probability assigned to the decision, estimated using Cantelli’s inequality.

Results

The classifier trained on RNA-seq data and tested on two independent microarray data sets resulted in zero and three errors. The classifier was further tested using quantitative real-time PCR expression levels of 45 blinded independent formalin-fixed paraffin-embedded specimens and was highly accurate. Pathway analyses have shown that sessile serrated polyps are distinguished from hyperplastic polyps and normal controls by: up-regulation of pathways implicated in proliferation, inflammation, cell-cell adhesion and down-regulation of serine threonine kinase signaling pathway; differential co-expression of pathways regulating cell division, protein trafficking and kinase activities.

Conclusions

Most of the differentially expressed pathways are known as hallmarks of cancer and likely to explain why sessile serrated polyps are more prone to neoplastic transformation than hyperplastic. The new molecular classifier includes 13 genes and may facilitate objective differentiation between two polyps.

Electronic supplementary material

The online version of this article (10.1186/s12920-017-0317-7) contains supplementary material, which is available to authorized users.

Improving cross-study prediction through addon batch effect adjustment or addon normalization

Motivation

To date most medical tests derived by applying classification methods to high-dimensional molecular data are hardly used in clinical practice. This is partly because the prediction error resulting when applying them to external data is usually much higher than internal error as evaluated through within-study validation procedures. We suggest the use of addon normalization and addon batch effect removal techniques in this context to reduce systematic differences between external data and the original dataset with the aim to improve prediction performance.

Results

We evaluate the impact of addon normalization and seven batch effect removal methods on cross-study prediction performance for several common classifiers using a large collection of microarray gene expression datasets, showing that some of these techniques reduce prediction error.

Availability and Implementation

All investigated addon methods are implemented in our R package bapred.

Contact

hornung@ibe.med.uni-muenchen.de.

Supplementary information

Supplementary data are available at Bioinformatics online.

Waste, Leaks, and Failures in the Biomarker Pipeline

BACKGROUND

The large, expanding literature on biomarkers is characterized by almost ubiquitous significant results, with claims about the potential importance, but few of these discovered biomarkers are used in routine clinical care.

CONTENT

The pipeline of biomarker development includes several specific stages: discovery, validation, clinical translation, evaluation, implementation (and, in the case of nonutility, deimplementation). Each of these stages can be plagued by problems that cause failures of the overall pipeline. Some problems are nonspecific challenges for all biomedical investigation, while others are specific to the peculiarities of biomarker research. Discovery suffers from poor methods and incomplete and selective reporting. External independent validation is limited. Selection for clinical translation is often shaped by nonrational choices. Evaluation is sparse and the clinical utility of many biomarkers remains unknown. The regulatory environment for biomarkers remains weak and guidelines can reach biased or divergent recommendations. Removing inefficient or even harmful biomarkers that have been entrenched in clinical care can meet with major resistance.

SUMMARY

The current biomarker pipeline is too prone to failures. Consideration of clinical needs should become a starting point for the development of biomarkers. Improvements can include the use of more stringent methodology, better reporting, larger collaborative studies, careful external independent validation, preregistration, rigorous systematic reviews and umbrella reviews, pivotal randomized trials, and implementation and deimplementation studies. Incentives should be aligned toward delivering useful biomarkers.

Characteristics and Validation Techniques for PCA-Based Gene-Expression Signatures

Background. Many gene-expression signatures exist for describing the biological state of profiled tumors. Principal Component Analysis (PCA) can be used to summarize a gene signature into a single score. Our hypothesis is that gene signatures can be validated when applied to new datasets, using inherent properties of PCA. Results. This validation is based on four key concepts. Coherence: elements of a gene signature should be correlated beyond chance. Uniqueness: the general direction of the data being examined can drive most of the observed signal. Robustness: if a gene signature is designed to measure a single biological effect, then this signal should be sufficiently strong and distinct compared to other signals within the signature. Transferability: the derived PCA gene signature score should describe the same biology in the target dataset as it does in the training dataset. Conclusions. The proposed validation procedure ensures that PCA-based gene signatures perform as expected when applied to datasets other than those that the signatures were trained upon. Complex signatures, describing multiple independent biological components, are also easily identified.

Molecular Diagnostics of Ageing and Tackling Age-related Disease

As average life expectancy increases there is a greater focus on health-span and, in particular, how to treat or prevent chronic age-associated diseases. Therapies which were able to control 'biological age' with the aim of postponing chronic and costly diseases of old age require an entirely new approach to drug development. Molecular technologies and machine-learning methods have already yielded diagnostics that help guide cancer treatment and cardiovascular procedures. Discovery of valid and clinically informative diagnostics of human biological age (combined with disease-specific biomarkers) has the potential to alter current drug-discovery strategies, aid clinical trial recruitment and maximize healthy ageing. I will review some basic principles that govern the development of 'ageing' diagnostics, how such assays could be used during the drug-discovery or development process. Important logistical and statistical considerations are illustrated by reviewing recent biomarker activity in the field of Alzheimer's disease, as dementia represents the most pressing of priorities for the pharmaceutical industry, as well as the chronic disease in humans most associated with age.

'Cytology-on-a-chip' based sensors for monitoring of potentially malignant oral lesions

Despite significant advances in surgical procedures and treatment, long-term prognosis for patients with oral cancer remains poor, with survival rates among the lowest of major cancers. Better methods are desperately needed to identify potential malignancies early when treatments are more effective.

OBJECTIVE

To develop robust classification models from cytology-on-a-chip measurements that mirror diagnostic performance of gold standard approach involving tissue biopsy.

MATERIALS AND METHODS

Measurements were recorded from 714 prospectively recruited patients with suspicious lesions across 6 diagnostic categories (each confirmed by tissue biopsy -histopathology) using a powerful new 'cytology-on-a-chip' approach capable of executing high content analysis at a single cell level. Over 200 cellular features related to biomarker expression, nuclear parameters and cellular morphology were recorded per cell. By cataloging an average of 2000 cells per patient, these efforts resulted in nearly 13 million indexed objects.

RESULTS

Binary "low-risk"/"high-risk" models yielded AUC values of 0.88 and 0.84 for training and validation models, respectively, with an accompanying difference in sensitivity+specificity of 6.2%. In terms of accuracy, this model accurately predicted the correct diagnosis approximately 70% of the time, compared to the 69% initial agreement rate of the pool of expert pathologists. Key parameters identified in these models included cell circularity, Ki67 and EGFR expression, nuclear-cytoplasmic ratio, nuclear area, and cell area.

CONCLUSIONS

This chip-based approach yields objective data that can be leveraged for diagnosis and management of patients with PMOL as well as uncovering new molecular-level insights behind cytological differences across the OED spectrum.

Emerging applications of metabolomics in drug discovery and precision medicine

Metabolomics is an emerging 'omics' science involving the comprehensive characterization of metabolites and metabolism in biological systems. Recent advances in metabolomics technologies are leading to a growing number of mainstream biomedical applications. In particular, metabolomics is increasingly being used to diagnose disease, understand disease mechanisms, identify novel drug targets, customize drug treatments and monitor therapeutic outcomes. This Review discusses some of the latest technological advances in metabolomics, focusing on the application of metabolomics towards uncovering the underlying causes of complex diseases (such as atherosclerosis, cancer and diabetes), the growing role of metabolomics in drug discovery and its potential effect on precision medicine.

COPD subtypes identified by network-based clustering of blood gene expression

One of the most common smoking-related diseases, chronic obstructive pulmonary disease (COPD), results from a dysregulated, multi-tissue inflammatory response to cigarette smoke. We hypothesized that systemic inflammatory signals in genome-wide blood gene expression can identify clinically important COPD-related disease subtypes, and we leveraged pre-existing gene interaction networks to guide unsupervised clustering of blood microarray expression data. Using network-informed non-negative matrix factorization, we analyzed genome-wide blood gene expression from 229 former smokers in the ECLIPSE Study, and we identified novel, clinically relevant molecular subtypes of COPD. These network-informed clusters were more stable and more strongly associated with measures of lung structure and function than clusters derived from a network-naïve approach, and they were associated with subtype-specific enrichment for inflammatory and protein catabolic pathways. These clusters were successfully reproduced in an independent sample of 135 smokers from the COPDGene Study.

Analytical strategies for studying stem cell metabolism

Owing to their capacity for self-renewal and pluripotency, stem cells possess untold potential for revolutionizing the field of regenerative medicine through the development of novel therapeutic strategies for treating cancer, diabetes, cardiovascular and neurodegenerative diseases. Central to developing these strategies is improving our understanding of biological mechanisms responsible for governing stem cell fate and self-renewal. Increasing attention is being given to the significance of metabolism, through the production of energy and generation of small molecules, as a critical regulator of stem cell functioning. Rapid advances in the field of metabolomics now allow for in-depth profiling of stem cells both in vitro and in vivo, providing a systems perspective on key metabolic and molecular pathways which influence stem cell biology. Understanding the analytical platforms and techniques that are currently used to study stem cell metabolomics, as well as how new insights can be derived from this knowledge, will accelerate new research in the field and improve future efforts to expand our understanding of the interplay between metabolism and stem cell biology.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

A five-gene hedgehog signature developed as a patient preselection tool for hedgehog inhibitor therapy in medulloblastoma

PURPOSE

Distinct molecular subgroups of medulloblastoma, including hedgehog (Hh) pathway-activated disease, have been reported. We identified and clinically validated a five-gene Hh signature assay that can be used to preselect patients with Hh pathway-activated medulloblastoma.

EXPERIMENTAL DESIGN

Gene characteristics of the Hh medulloblastoma subgroup were identified through published bioinformatic analyses. Thirty-two genes shown to be differentially expressed in fresh-frozen and formalin-fixed paraffin-embedded tumor samples and reproducibly analyzed by RT-PCR were measured in matched samples. These data formed the basis for building a multi-gene logistic regression model derived through elastic net methods from which the five-gene Hh signature emerged after multiple iterations. On the basis of signature gene expression levels, the model computed a propensity score to determine Hh activation using a threshold set a priori. The association between Hh activation status and tumor response to the Hh pathway inhibitor sonidegib (LDE225) was analyzed.

RESULTS

Five differentially expressed genes in medulloblastoma (GLI1, SPHK1, SHROOM2, PDLIM3, and OTX2) were found to associate with Hh pathway activation status. In an independent validation study, Hh activation status of 25 medulloblastoma samples showed 100% concordance between the five-gene signature and Affymetrix profiling. Further, in medulloblastoma samples from 50 patients treated with sonidegib, all 6 patients who responded were found to have Hh-activated tumors. Three patients with Hh-activated tumors had stable or progressive disease. No patients with Hh-nonactivated tumors responded.

CONCLUSIONS

This five-gene Hh signature can robustly identify Hh-activated medulloblastoma and may be used to preselect patients who might benefit from sonidegib treatment.

Added predictive value of omics data: specific issues related to validation illustrated by two case studies

Background

In the last years, the importance of independent validation of the prediction ability of a new gene signature has been largely recognized. Recently, with the development of gene signatures which integrate rather than replace the clinical predictors in the prediction rule, the focus has been moved to the validation of the added predictive value of a gene signature, i.e. to the verification that the inclusion of the new gene signature in a prediction model is able to improve its prediction ability.

Methods

The high-dimensional nature of the data from which a new signature is derived raises challenging issues and necessitates the modification of classical methods to adapt them to this framework. Here we show how to validate the added predictive value of a signature derived from high-dimensional data and critically discuss the impact of the choice of methods on the results.

Results

The analysis of the added predictive value of two gene signatures developed in two recent studies on the survival of leukemia patients allows us to illustrate and empirically compare different validation techniques in the high-dimensional framework.

Conclusions

The issues related to the high-dimensional nature of the omics predictors space affect the validation process. An analysis procedure based on repeated cross-validation is suggested.

Cross-study validation for the assessment of prediction algorithms

Motivation: Numerous competing algorithms for prediction in high-dimensional settings have been developed in the statistical and machine-learning literature. Learning algorithms and the prediction models they generate are typically evaluated on the basis of cross-validation error estimates in a few exemplary datasets. However, in most applications, the ultimate goal of prediction modeling is to provide accurate predictions for independent samples obtained in different settings. Cross-validation within exemplary datasets may not adequately reflect performance in the broader application context.

Methods: We develop and implement a systematic approach to ‘cross-study validation’, to replace or supplement conventional cross-validation when evaluating high-dimensional prediction models in independent datasets. We illustrate it via simulations and in a collection of eight estrogen-receptor positive breast cancer microarray gene-expression datasets, where the objective is predicting distant metastasis-free survival (DMFS). We computed the C-index for all pairwise combinations of training and validation datasets. We evaluate several alternatives for summarizing the pairwise validation statistics, and compare these to conventional cross-validation.

Results: Our data-driven simulations and our application to survival prediction with eight breast cancer microarray datasets, suggest that standard cross-validation produces inflated discrimination accuracy for all algorithms considered, when compared to cross-study validation. Furthermore, the ranking of learning algorithms differs, suggesting that algorithms performing best in cross-validation may be suboptimal when evaluated through independent validation.

Availability: The survHD: Survival in High Dimensions package (http://www.bitbucket.org/lwaldron/survhd) will be made available through Bioconductor.

Contact:levi.waldron@hunter.cuny.edu

Supplementary information:Supplementary data are available at Bioinformatics online.

Design and analysis of metabolomics studies in epidemiologic research: a primer on -omic technologies

Metabolomics is the field of "-omics" research concerned with the comprehensive characterization of the small low-molecular-weight metabolites in biological samples. In epidemiology, it represents an emerging technology and an unprecedented opportunity to measure environmental and other exposures with improved precision and far less measurement error than with standard epidemiologic methods. Advances in the application of metabolomics in large-scale epidemiologic research are now being realized through a combination of improved sample preparation and handling, automated laboratory and processing methods, and reduction in costs. The number of epidemiologic studies that use metabolic profiling is still limited, but it is fast gaining popularity in this area. In the present article, we present a roadmap for metabolomic analyses in epidemiologic studies and discuss the various challenges these data pose to large-scale studies. We discuss the steps of data preprocessing, univariate and multivariate data analysis, correction for multiplicity of comparisons with correlated data, and finally the steps of cross-validation and external validation. As data from metabolomic studies accumulate in epidemiology, there is a need for large-scale replication and synthesis of findings, increased availability of raw data, and a focus on good study design, all of which will highlight the potential clinical impact of metabolomics in this field.

Investigating the prediction ability of survival models based on both clinical and omics data: two case studies

In biomedical literature, numerous prediction models for clinical outcomes have been developed based either on clinical data or, more recently, on high-throughput molecular data (omics data). Prediction models based on both types of data, however, are less common, although some recent studies suggest that a suitable combination of clinical and molecular information may lead to models with better predictive abilities. This is probably due to the fact that it is not straightforward to combine data with different characteristics and dimensions (poorly characterized high-dimensional omics data, well-investigated low-dimensional clinical data). In this paper, we analyze two publicly available datasets related to breast cancer and neuroblastoma, respectively, in order to show some possible ways to combine clinical and omics data into a prediction model of time-to-event outcome. Different strategies and statistical methods are exploited. The results are compared and discussed according to different criteria, including the discriminative ability of the models, computed on a validation dataset.

PredictSNP: robust and accurate consensus classifier for prediction of disease-related mutations

Single nucleotide variants represent a prevalent form of genetic variation. Mutations in the coding regions are frequently associated with the development of various genetic diseases. Computational tools for the prediction of the effects of mutations on protein function are very important for analysis of single nucleotide variants and their prioritization for experimental characterization. Many computational tools are already widely employed for this purpose. Unfortunately, their comparison and further improvement is hindered by large overlaps between the training datasets and benchmark datasets, which lead to biased and overly optimistic reported performances. In this study, we have constructed three independent datasets by removing all duplicities, inconsistencies and mutations previously used in the training of evaluated tools. The benchmark dataset containing over 43,000 mutations was employed for the unbiased evaluation of eight established prediction tools: MAPP, nsSNPAnalyzer, PANTHER, PhD-SNP, PolyPhen-1, PolyPhen-2, SIFT and SNAP. The six best performing tools were combined into a consensus classifier PredictSNP, resulting into significantly improved prediction performance, and at the same time returned results for all mutations, confirming that consensus prediction represents an accurate and robust alternative to the predictions delivered by individual tools. A user-friendly web interface enables easy access to all eight prediction tools, the consensus classifier PredictSNP and annotations from the Protein Mutant Database and the UniProt database. The web server and the datasets are freely available to the academic community at http://loschmidt.chemi.muni.cz/predictsnp.

Increasing value and reducing waste in research design, conduct, and analysis

Correctable weaknesses in the design, conduct, and analysis of biomedical and public health research studies can produce misleading results and waste valuable resources. Small effects can be difficult to distinguish from bias introduced by study design and analyses. An absence of detailed written protocols and poor documentation of research is common. Information obtained might not be useful or important, and statistical precision or power is often too low or used in a misleading way. Insufficient consideration might be given to both previous and continuing studies. Arbitrary choice of analyses and an overemphasis on random extremes might affect the reported findings. Several problems relate to the research workforce, including failure to involve experienced statisticians and methodologists, failure to train clinical researchers and laboratory scientists in research methods and design, and the involvement of stakeholders with conflicts of interest. Inadequate emphasis is placed on recording of research decisions and on reproducibility of research. Finally, reward systems incentivise quantity more than quality, and novelty more than reliability. We propose potential solutions for these problems, including improvements in protocols and documentation, consideration of evidence from studies in progress, standardisation of research efforts, optimisation and training of an experienced and non-conflicted scientific workforce, and reconsideration of scientific reward systems.

An industry statistician's perspective on PHC drug development

In the past decade, the cost of drug development has increased significantly. The estimates vary widely but frequently quoted numbers are staggering-it takes 10-15 years and billions of dollars to bring a drug to patients. To a large extent this is due to many long, expensive and ultimately unsuccessful drug trials. While one approach to combat the low yield on investment could be to continue searching for new blockbusters, an alternative method would lead us to focus on testing new targeted treatments that have a strong underlying scientific rationale and are more likely to provide enhanced clinical benefit in population subsets defined by molecular diagnostics. Development of these new treatments, however, cannot follow the usual established path; new strategies and approaches are required for the co-development of novel therapeutics and the diagnostic. In this paper we will review, from the point of view of industry, the approaches to, and challenges of drug development strategies incorporating predictive biomarkers into clinical programs. We will outline the basic concepts behind co-development with predictive biomarkers and summarize the current regulatory paradigm. We will present guiding principles of personalized health care (PHC) development and review the statistical, strategic, regulatory and operational challenges that statisticians regularly encounter on development programs with a PHC component. Some practical recommendations for team statisticians involved in PHC drug development are included. The majority of the examples and recommendations are drawn from oncology but broader concepts apply across all therapeutic areas.

Genetic variants and their interactions in disease risk prediction - machine learning and network perspectives

A central challenge in systems biology and medical genetics is to understand how interactions among genetic loci contribute to complex phenotypic traits and human diseases. While most studies have so far relied on statistical modeling and association testing procedures, machine learning and predictive modeling approaches are increasingly being applied to mining genotype-phenotype relationships, also among those associations that do not necessarily meet statistical significance at the level of individual variants, yet still contributing to the combined predictive power at the level of variant panels. Network-based analysis of genetic variants and their interaction partners is another emerging trend by which to explore how sub-network level features contribute to complex disease processes and related phenotypes. In this review, we describe the basic concepts and algorithms behind machine learning-based genetic feature selection approaches, their potential benefits and limitations in genome-wide setting, and how physical or genetic interaction networks could be used as a priori information for providing improved predictive power and mechanistic insights into the disease networks. These developments are geared toward explaining a part of the missing heritability, and when combined with individual genomic profiling, such systems medicine approaches may also provide a principled means for tailoring personalized treatment strategies in the future.

Opportunities and challenges for selected emerging technologies in cancer epidemiology: mitochondrial, epigenomic, metabolomic, and telomerase profiling

Remarkable progress has been made in the last decade in new methods for biologic measurements using sophisticated technologies that go beyond the established genome, proteome, and gene expression platforms. These methods and technologies create opportunities to enhance cancer epidemiologic studies. In this article, we describe several emerging technologies and evaluate their potential in epidemiologic studies. We review the background, assays, methods, and challenges and offer examples of the use of mitochondrial DNA and copy number assessments, epigenomic profiling (including methylation, histone modification, miRNAs, and chromatin condensation), metabolite profiling (metabolomics), and telomere measurements. We map the volume of literature referring to each one of these measurement tools and the extent to which efforts have been made at knowledge integration (e.g., systematic reviews and meta-analyses). We also clarify strengths and weaknesses of the existing platforms and the range of type of samples that can be tested with each of them. These measurement tools can be used in identifying at-risk populations and providing novel markers of survival and treatment response. Rigorous analytic and validation standards, transparent availability of massive data, and integration in large-scale evidence are essential in fulfilling the potential of these technologies.

Multi-analyte network markers for tumor prognosis

Deregulation of gene expression, a hallmark of cancer, is caused by both genetic and epigenetic mechanisms. The rapid accumulation of epigenome maps of various cancers suggests a new avenue of research, namely integrating epigenomic data with other types of omic data for cancer diagnosis, prognosis, and biomarker discovery. We introduce the MAPIT algorithm (Multi Analyte Pathway Inference Tool), to enable principled integration of epigenomic, transcriptomic, and protein interactome data. As a proof-of-principle, we apply MAPIT to glioblastoma multiforme (GBM), the most common and aggressive form of brain tumor. Few predictive markers were reported for the prognosis of GBM patients. By integrating mRNA transcriptome, promoter DNA methylome and protein-protein physical interactome, we find ten expression- and three methylation-based network markers, involving 118 genes. When tested on additional GBM patient samples, the prognostic accuracy of the multi-analyte network markers (73.5%) is 9.7% and 8.6% higher than previous prognostic signatures built on gene expression or DNA methylation alone. Our results highlight the critical role of two novel pathways in the prognosis of GBM patients, small GTPase-mediated protein trafficking and ubiquitination-dependent protein degradation. A better understanding of these two pathways could lead to personalized therapies for subgroups of GBM patients. Our study demonstrates that integrating epigenomic, transcriptomic, and interactomic data can improve the accuracy network-based prognosis markers and lead to novel mechanistic understanding of cancer.

Comparison and evaluation of pathway-level aggregation methods of gene expression data

Background

Microarray experiments produce expression measurements in genomic scale. A way to derive functional understanding of the data is to focus on functional sets of genes, such as pathways, instead of individual genes. While a common practice for the pathway-level analysis has been functional enrichment analysis such as over-representation analysis and gene set enrichment analysis, an alternative approach has also been explored. In this approach, gene expression data are first aggregated at pathway level to transform the original data into a compact representation in which each row corresponds to a pathway instead of a gene. Thereafter the pathway expression data can be used for differential expression and classification analyses in pathway space, leveraging existing algorithms usually applied to gene expression data. While several studies have proposed the pathway-level aggregation methods, it remains unclear how they compare with one another, since the evaluations were done to a limited extent. Thus this study presents a comprehensive evaluation of six most prominent aggregation methods.

Results

The compared methods include five existing methods--mean of all member genes (Mean all), mean of condition-responsive genes (Mean CORGs), analysis of sample set enrichment scores (ASSESS), principal component analysis (PCA), and partial least squares (PLS)--and a variant of an existing method (Mean top 50%, averaging top half of member genes). Comprehensive and stringent benchmarking was performed by collecting seven pairs of related but independent datasets encompassing various phenotypes. Aggregation was done in the space of KEGG pathways. Performance of the methods was assessed by classification accuracy validated both internally and externally, and by examining the correlative extent of pathway signatures between the dataset pairs. The assessment revealed that (i) the best accuracy and correlation were obtained from ASSESS and Mean top 50%, (ii) Mean all showed the lowest accuracy, and (iii) Mean CORGs and PLS gave rise to the largest extent of discordance in the pathway signature correlation.

Conclusions

The two best performing method (ASSESS and Mean top 50%) are suggested to be preferred. The benchmarking analysis also suggests that there is both room and necessity for developing a novel method for pathway-level aggregation.

Why your new cancer biomarker may never work: recurrent patterns and remarkable diversity in biomarker failures

Less than 1% of published cancer biomarkers actually enter clinical practice. Although best practices for biomarker development are published, optimistic investigators may not appreciate the statistical near-certainty and diverse modes by which the other 99% (likely including your favorite new marker) do indeed fail. Here, patterns of failure were abstracted for classification from publications and an online database detailing marker failures. Failure patterns formed a hierarchical logical structure, or outline, of an emerging, deeply complex, and arguably fascinating science of biomarker failure. A new cancer biomarker under development is likely to have already encountered one or more of the following fatal features encountered by prior markers: lack of clinical significance, hidden structure in the source data, a technically inadequate assay, inappropriate statistical methods, unmanageable domination of the data by normal variation, implausibility, deficiencies in the studied population or in the investigator system, and its disproof or abandonment for cause by others. A greater recognition of the science of biomarker failure and its near-complete ubiquity is constructive and celebrates a seemingly perpetual richness of biologic, technical, and philosophical complexity, the full appreciation of which could improve the management of scarce research resources.