Design and analysis of classifier learning experiments in bioinformatics: survey and case studies

rMSIcleanup: an open-source tool for matrix-related peak annotation in mass spectrometry imaging and its application to silver-assisted laser desorption/ionization

Mass spectrometry imaging (MSI) has become a mature, widespread analytical technique to perform non-targeted spatial metabolomics. However, the compounds used to promote desorption and ionization of the analyte during acquisition cause spectral interferences in the low mass range that hinder downstream data processing in metabolomics applications. Thus, it is advisable to annotate and remove matrix-related peaks to reduce the number of redundant and non-biologically-relevant variables in the dataset. We have developed rMSIcleanup, an open-source R package to annotate and remove signals from the matrix, according to the matrix chemical composition and the spatial distribution of its ions. To validate the annotation method, rMSIcleanup was challenged with several images acquired using silver-assisted laser desorption ionization MSI (AgLDI MSI). The algorithm was able to correctly classify m/z signals related to silver clusters. Visual exploration of the data using Principal Component Analysis (PCA) demonstrated that annotation and removal of matrix-related signals improved spectral data post-processing. The results highlight the need for including matrix-related peak annotation tools such as rMSIcleanup in MSI workflows.

A New Kind of Nonparametric Test for Statistical Comparison of Multiple Classifiers Over Multiple Datasets

Nonparametric statistical analysis, such as the Friedman test (FT), is gaining more and more attention due to its useful applications in a lot of experimental studies. However, traditional FT for the comparison of multiple learning algorithms on different datasets adopts the naive ranking approach. The ranking is based on the average accuracy values obtained by the set of learning algorithms on the datasets, which neither considers the differences of the results obtained by the learning algorithms on each dataset nor takes into account the performance of the learning algorithms in each run. In this paper, we will first propose three kinds of ranking approaches, which are the weighted ranking approach, the global ranking approach (GRA), and the weighted GRA. Then, a theoretical analysis is performed to explore the properties of the proposed ranking approaches. Next, a set of the modified FTs based on the proposed ranking approaches are designed for the comparison of the learning algorithms. Finally, the modified FTs are evaluated through six classifier ensemble approaches on 34 real-world datasets. The experiments show the effectiveness of the modified FTs.

Choosing the Most Effective Pattern Classification Model under Learning-Time Constraint

Nowadays, large datasets are common and demand faster and more effective pattern analysis techniques. However, methodologies to compare classifiers usually do not take into account the learning-time constraints required by applications. This work presents a methodology to compare classifiers with respect to their ability to learn from classification errors on a large learning set, within a given time limit. Faster techniques may acquire more training samples, but only when they are more effective will they achieve higher performance on unseen testing sets. We demonstrate this result using several techniques, multiple datasets, and typical learning-time limits required by applications.

Visualizing Validation of Protein Surface Classifiers

Many bioinformatics applications construct classifiers that are validated in experiments that compare their results to known ground truth over a corpus. In this paper, we introduce an approach for exploring the results of such classifier validation experiments, focusing on classifiers for regions of molecular surfaces. We provide a tool that allows for examining classification performance patterns over a test corpus. The approach combines a summary view that provides information about an entire corpus of molecules with a detail view that visualizes classifier results directly on protein surfaces. Rather than displaying miniature 3D views of each molecule, the summary provides 2D glyphs of each protein surface arranged in a reorderable, small-multiples grid. Each summary is specifically designed to support visual aggregation to allow the viewer to both get a sense of aggregate properties as well as the details that form them. The detail view provides a 3D visualization of each protein surface coupled with interaction techniques designed to support key tasks, including spatial aggregation and automated camera touring. A prototype implementation of our approach is demonstrated on protein surface classifier experiments.

Diagnostic and prognostic utility of a DNA hypermethylated gene signature in prostate cancer

We aimed to identify a prostate cancer DNA hypermethylation microarray signature (denoted as PHYMA) that differentiates prostate cancer from benign prostate hyperplasia (BPH), high from low-grade and lethal from non-lethal cancers. This is a non-randomized retrospective study in 111 local Asian men (87 prostate cancers and 24 BPH) treated from 1995 to 2009 in our institution. Archival prostate epithelia were laser-capture microdissected and genomic DNA extracted and bisulfite-converted. Samples were profiled using Illumina GoldenGate Methylation microarray, with raw data processed by GenomeStudio. A classification model was generated using support vector machine, consisting of a 55-probe DNA methylation signature of 46 genes. The model was independently validated on an internal testing dataset which yielded cancer detection sensitivity and specificity of 95.3% and 100% respectively, with overall accuracy of 96.4%. Second validation on another independent western cohort yielded 89.8% sensitivity and 66.7% specificity, with overall accuracy of 88.7%. A PHYMA score was developed for each sample based on the state of methylation in the PHYMA signature. Increasing PHYMA score was significantly associated with higher Gleason score and Gleason primary grade. Men with higher PHYMA scores have poorer survival on univariate (p = 0.0038, HR = 3.89) and multivariate analyses when controlled for (i) clinical stage (p = 0.055, HR = 2.57), and (ii) clinical stage and Gleason score (p = 0.043, HR = 2.61). We further performed bisulfite genomic sequencing on 2 relatively unknown genes to demonstrate robustness of the assay results. PHYMA is thus a signature with high sensitivity and specificity for discriminating tumors from BPH, and has a potential role in early detection and in predicting survival.

Biologically inspired intelligent decision making: a commentary on the use of artificial neural networks in bioinformatics

Artificial neural networks (ANNs) are a class of powerful machine learning models for classification and function approximation which have analogs in nature. An ANN learns to map stimuli to responses through repeated evaluation of exemplars of the mapping. This learning approach results in networks which are recognized for their noise tolerance and ability to generalize meaningful responses for novel stimuli. It is these properties of ANNs which make them appealing for applications to bioinformatics problems where interpretation of data may not always be obvious, and where the domain knowledge required for deductive techniques is incomplete or can cause a combinatorial explosion of rules. In this paper, we provide an introduction to artificial neural network theory and review some interesting recent applications to bioinformatics problems.