Reducing the probability of false positive research findings by pre-publication validation - experience with a large multiple sclerosis database

Addressing researcher degrees of freedom through minP adjustment

When different researchers study the same research question using the same dataset they may obtain different and potentially even conflicting results. This is because there is often substantial flexibility in researchers' analytical choices, an issue also referred to as "researcher degrees of freedom". Combined with selective reporting of the smallest p-value or largest effect, researcher degrees of freedom may lead to an increased rate of false positive and overoptimistic results. In this paper, we address this issue by formalizing the multiplicity of analysis strategies as a multiple testing problem. As the test statistics of different analysis strategies are usually highly dependent, a naive approach such as the Bonferroni correction is inappropriate because it leads to an unacceptable loss of power. Instead, we propose using the "minP" adjustment method, which takes potential test dependencies into account and approximates the underlying null distribution of the minimal p-value through a permutation-based procedure. This procedure is known to achieve more power than simpler approaches while ensuring a weak control of the family-wise error rate. We illustrate our approach for addressing researcher degrees of freedom by applying it to a study on the impact of perioperative p a O 2 on post-operative complications after neurosurgery. A total of 48 analysis strategies are considered and adjusted using the minP procedure. This approach allows to selectively report the result of the analysis strategy yielding the most convincing evidence, while controlling the type 1 error-and thus the risk of publishing false positive results that may not be replicable.

Maximizing the Clinical Value of Blood-Based Biomarkers for Mild Traumatic Brain Injury

Mild traumatic brain injury (TBI) and concussion can have serious consequences that develop over time with unpredictable levels of recovery. Millions of concussions occur yearly, and a substantial number result in lingering symptoms, loss of productivity, and lower quality of life. The diagnosis may not be made for multiple reasons, including due to patient hesitancy to undergo neuroimaging and inability of imaging to detect minimal damage. Biomarkers could fill this gap, but the time needed to send blood to a laboratory for analysis made this impractical until point-of-care measurement became available. A handheld blood test is now on the market for diagnosis of concussion based on the specific blood biomarkers glial fibrillary acidic protein (GFAP) and ubiquitin carboxyl terminal hydrolase L1 (UCH-L1). This paper discusses rapid blood biomarker assessment for mild TBI and its implications in improving prediction of TBI course, avoiding repeated head trauma, and its potential role in assessing new therapeutic options. Although we focus on the Abbott i-STAT TBI plasma test because it is the first to be FDA-cleared, our discussion applies to any comparable test systems that may become available in the future. The difficulties in changing emergency department protocols to include new technology are addressed.

Supervised Prediction of Aging-Related Genes From a Context-Specific Protein Interaction Subnetwork

Human aging is linked to many prevalent diseases. The aging process is highly influenced by genetic factors. Hence, it is important to identify human aging-related genes. We focus on supervised prediction of such genes. Gene expression-based methods for this purpose study genes in isolation from each other. While protein-protein interaction (PPI) network-based methods for this purpose account for interactions between genes' protein products, current PPI network data are context-unspecific, spanning different biological conditions. Instead, here, we focus on an aging-specific subnetwork of the entire PPI network, obtained by integrating aging-specific gene expression data and PPI network data. The potential of such data integration has been recognized but mostly in the context of cancer. So, we are the first to propose a supervised learning framework for predicting aging-related genes from an aging-specific PPI subnetwork. In a systematic and comprehensive evaluation, we find that in many of the evaluation tests: (i) using an aging-specific subnetwork indeed yields more accurate aging-related gene predictions than using the entire network, and (ii) predictive methods from our framework that have not previously been used for supervised prediction of aging-related genes outperform existing prominent methods for the same purpose. These results justify the need for our framework.

Cognitive performance shows domain specific associations with regional cortical thickness in multiple sclerosis

•

Cognitive impairment correlates with loss of cortical thickness in MS.

•

Cognitive tests show distinctive regional associations with cortical thickness.

•

Some regions, such as the left insula, correlate with multiple tests.

•

Associations patterns seem reproducible in patients with very mild impairment.

•

Better localization of cognitive functions may improve future MRI studies.

Multiple Sclerosis (MS) patients often suffer from significant cognitive impairment. Earlier research has shown relationships between regional cortical atrophy and cognitive deterioration. However, due to a large number of neuropsychological assessments and a heterogenous pattern of cognitive deficits in MS patients, reported associations patterns are also heterogenous. Using an extensive neuropsychological battery of 23 different tasks, we explored domain (attention/information processing, memory, spatial processing, executive functioning) and task-specific associations with regional cortical thickness in a representative sample of MS patients (N = 97). Cortical regions associated with multiple cognitive tasks in the left hemisphere were predominantly located in the inferior insula (attention p < 0.001, memory p = 0.047, spatial processing p = 0.004, executive functioning p = 0.037), the gyrus frontalis superior (attention p = 0.015, memory p = 0.037, spatial processing p = 0.033, executive functioning p = 0.017) and temporal medial (attention p < 0.001, memory two clusters p = 0.016 and p < 0.001, executive functioning p = 0.016). In the right hemisphere, we detected the strongest association in the sulcus interparietalis with five cluster (attention SDMT p = 0.003 and TAP_DA p < 0.001; memory Rey recall p = 0.013 and VLMT verbal learning p = 0.016; spatial processing Rey copy p < 0.001). We replicated parts of our results in an independent sample of 30 mildly disabled MS patients. Moreover, comparisons to 29 healthy controls showed that the regional associations seemed to represent rather pathophysiological dependency than a physiological one. We believe that our results may prove useful in diagnosis and rehabilitation of cognitive impairments and may serve as guidance in future magnetic resonance imaging (MRI) studies.

Gram-negative and -positive bacteria differentiation in blood culture samples by headspace volatile compound analysis

BACKGROUND

Identification of microorganisms in positive blood cultures still relies on standard techniques such as Gram staining followed by culturing with definite microorganism identification. Alternatively, matrix-assisted laser desorption/ionization time-of-flight mass spectrometry or the analysis of headspace volatile compound (VC) composition produced by cultures can help to differentiate between microorganisms under experimental conditions. This study assessed the efficacy of volatile compound based microorganism differentiation into Gram-negatives and -positives in unselected positive blood culture samples from patients.

METHODS

Headspace gas samples of positive blood culture samples were transferred to sterilized, sealed, and evacuated 20 ml glass vials and stored at -30 °C until batch analysis. Headspace gas VC content analysis was carried out via an auto sampler connected to an ion-molecule reaction mass spectrometer (IMR-MS). Measurements covered a mass range from 16 to 135 u including CO2, H2, N2, and O2. Prediction rules for microorganism identification based on VC composition were derived using a training data set and evaluated using a validation data set within a random split validation procedure.

RESULTS

One-hundred-fifty-two aerobic samples growing 27 Gram-negatives, 106 Gram-positives, and 19 fungi and 130 anaerobic samples growing 37 Gram-negatives, 91 Gram-positives, and two fungi were analysed. In anaerobic samples, ten discriminators were identified by the random forest method allowing for bacteria differentiation into Gram-negative and -positive (error rate: 16.7 % in validation data set). For aerobic samples the error rate was not better than random.

CONCLUSIONS

In anaerobic blood culture samples of patients IMR-MS based headspace VC composition analysis facilitates bacteria differentiation into Gram-negative and -positive.

A measure of the impact of CV incompleteness on prediction error estimation with application to PCA and normalization

Background

In applications of supervised statistical learning in the biomedical field it is necessary to assess the prediction error of the respective prediction rules. Often, data preparation steps are performed on the dataset—in its entirety—before training/test set based prediction error estimation by cross-validation (CV)—an approach referred to as “incomplete CV”. Whether incomplete CV can result in an optimistically biased error estimate depends on the data preparation step under consideration. Several empirical studies have investigated the extent of bias induced by performing preliminary supervised variable selection before CV. To our knowledge, however, the potential bias induced by other data preparation steps has not yet been examined in the literature. In this paper we investigate this bias for two common data preparation steps: normalization and principal component analysis for dimension reduction of the covariate space (PCA). Furthermore we obtain preliminary results for the following steps: optimization of tuning parameters, variable filtering by variance and imputation of missing values.

Methods

We devise the easily interpretable and general measure CVIIM (“CV Incompleteness Impact Measure”) to quantify the extent of bias induced by incomplete CV with respect to a data preparation step of interest. This measure can be used to determine whether a specific data preparation step should, as a general rule, be performed in each CV iteration or whether an incomplete CV procedure would be acceptable in practice. We apply CVIIM to large collections of microarray datasets to answer this question for normalization and PCA.

Results

Performing normalization on the entire dataset before CV did not result in a noteworthy optimistic bias in any of the investigated cases. In contrast, when performing PCA before CV, medium to strong underestimates of the prediction error were observed in multiple settings.

Conclusions

While the investigated forms of normalization can be safely performed before CV, PCA has to be performed anew in each CV split to protect against optimistic bias.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-015-0088-9) contains supplementary material, which is available to authorized users.

Added predictive value of omics data: specific issues related to validation illustrated by two case studies

Background

In the last years, the importance of independent validation of the prediction ability of a new gene signature has been largely recognized. Recently, with the development of gene signatures which integrate rather than replace the clinical predictors in the prediction rule, the focus has been moved to the validation of the added predictive value of a gene signature, i.e. to the verification that the inclusion of the new gene signature in a prediction model is able to improve its prediction ability.

Methods

The high-dimensional nature of the data from which a new signature is derived raises challenging issues and necessitates the modification of classical methods to adapt them to this framework. Here we show how to validate the added predictive value of a signature derived from high-dimensional data and critically discuss the impact of the choice of methods on the results.

Results

The analysis of the added predictive value of two gene signatures developed in two recent studies on the survival of leukemia patients allows us to illustrate and empirically compare different validation techniques in the high-dimensional framework.

Conclusions

The issues related to the high-dimensional nature of the omics predictors space affect the validation process. An analysis procedure based on repeated cross-validation is suggested.

Diverse convergent evidence in the genetic analysis of complex disease: coordinating omic, informatic, and experimental evidence to better identify and validate risk factors

In omic research, such as genome wide association studies, researchers seek to repeat their results in other datasets to reduce false positive findings and thus provide evidence for the existence of true associations. Unfortunately this standard validation approach cannot completely eliminate false positive conclusions, and it can also mask many true associations that might otherwise advance our understanding of pathology. These issues beg the question: How can we increase the amount of knowledge gained from high throughput genetic data? To address this challenge, we present an approach that complements standard statistical validation methods by drawing attention to both potential false negative and false positive conclusions, as well as providing broad information for directing future research. The Diverse Convergent Evidence approach (DiCE) we propose integrates information from multiple sources (omics, informatics, and laboratory experiments) to estimate the strength of the available corroborating evidence supporting a given association. This process is designed to yield an evidence metric that has utility when etiologic heterogeneity, variable risk factor frequencies, and a variety of observational data imperfections might lead to false conclusions. We provide proof of principle examples in which DiCE identified strong evidence for associations that have established biological importance, when standard validation methods alone did not provide support. If used as an adjunct to standard validation methods this approach can leverage multiple distinct data types to improve genetic risk factor discovery/validation, promote effective science communication, and guide future research directions.

Onset of secondary progressive phase and long-term evolution of multiple sclerosis

OBJECTIVES

To assess factors affecting the rate of conversion to secondary progressive (SP) multiple sclerosis (MS) and its subsequent evolution.

METHODS

Among 806 patients with relapsing remitting (RR) onset MS from the London Ontario database, we used Kaplan-Meier, Cox regression and multiple logistic regression analyses to investigate the effect of baseline clinical and demographic features on (1) the probability of, and the time to, SP disease, (2) the time to bedbound status (Disability Status Scale (DSS 8)) from onset of progression.

RESULTS

The risk of entering the SP phase increased proportionally with disease duration (OR=1.07 for each additional year; p<0.001). Shorter latency to SP was associated with shorter times to severe disability. The same association was found even when patients were grouped by number of total relapses before progression. However, the evolution of the SP phase was not influenced by the duration of the RR phase. Male sex (HR=1.41; p<0.001), older age at onset (age ≤20 and 21-30 vs >30 HR=0.52 (p<0.001), 0.65 (p<0.001), respectively) and high early relapse frequency (1-2 attacks vs ≥3 HR=0.63 (p<0.001), 0.75 (p=0.04), respectively) predicted significantly higher risk of SP MS and shorter latency to progression. Times to DSS 8 from onset of progression were significantly shorter among those with high early relapse frequency (≥3 attacks), and among those presenting with cerebellar and brainstem symptoms.

CONCLUSIONS

The onset of SP MS is the dominant determinant of long-term prognosis, and its prevention is the most important target measure for treatment. Baseline clinical features of early relapse frequency and age at onset can be used to select groups at higher risk of developing severe disability based on the probability of their disease becoming progressive within a defined time period.

What is the "normal" fetal heart rate?

Aim. There is no consensus about the normal fetal heart rate. Current international guidelines recommend for the normal fetal heart rate (FHR) baseline different ranges of 110 to 150 beats per minute (bpm) or 110 to 160 bpm. We started with a precise definition of "normality" and performed a retrospective computerized analysis of electronically recorded FHR tracings. Methods. We analyzed all recorded cardiotocography tracings of singleton pregnancies in three German medical centers from 2000 to 2007 and identified 78,852 tracings of sufficient quality. For each tracing, the baseline FHR was extracted by eliminating accelerations/decelerations and averaging based on the "delayed moving windows" algorithm. After analyzing 40% of the dataset as "training set" from one hospital generating a hypothetical normal baseline range, evaluation of external validity on the other 60% of the data was performed using data from later years in the same hospital and externally using data from the two other hospitals. Results. Based on the training data set, the "best" FHR range was 115 or 120 to 160 bpm. Validation in all three data sets identified 120 to 160 bpm as the correct symmetric "normal range". FHR decreases slightly during gestation. Conclusions. Normal ranges for FHR are 120 to 160 bpm. Many international guidelines define ranges of 110 to 160 bpm which seem to be safe in daily practice. However, further studies should confirm that such asymmetric alarm limits are safe, with a particular focus on the lower bound, and should give insights about how to show and further improve the usefulness of the widely used practice of CTG monitoring.

Developing a short-form structured diagnostic interview for common mental disorders using signal detection theory

Diagnostic instruments must be relatively free from respondent burden and cost effective to administer whilst remaining faithful to the psychiatric nomenclature. It seems logical to develop short-form alternatives to rather lengthy and complicated diagnostic interviews to facilitate large scale data collection. The current study examines one method, signal detection theory, for developing a short-form interview based on the Composite International Diagnostic Interview version 3.0. The method was able to retain the smallest number of items to predict a lifetime and 30 day DSM-IV diagnosis for 10 disorders. Concordance analyses between the full-form and the short-form modules, demonstrated an excellent level of agreement in the whole sample and various subsamples of the Australian population as well as in an international comparison sample of the US population. The good concordance between the long form and the short form demonstrates the ability of signal detection theory to assist in the development of valid short forms, which could replace lengthy diagnostic interviews when the aim is to reduce respondent burden and overall research costs.

Age and disability accumulation in multiple sclerosis

OBJECTIVES

We tested the hypothesis that age is a prognostic factor with respect to long-term accumulation of disability in multiple sclerosis (MS).

METHODS

Kaplan-Meier analysis and binary logistic regression models determined the effect of age at disease onset, age at onset of progression, and current age on attainment of severe disability levels (Disability Status Scale [DSS] 6-8-10) from the London, Ontario, database (n = 1,023).

RESULTS

Older age at relapsing-remitting (RR) phase onset was associated with higher risk of reaching advanced DSS scores. This was independent of disease duration and early relapse frequency but secondary to increased risk of conversion to secondary progressive (SP) MS. Onset at age 40 (odds ratio [OR] = 4.22) and at age 50 (OR = 6.04) doubled and tripled risks of developing SP, compared to age 20 (OR = 2.05). Younger age at conversion to SPMS was associated with shorter times to high DSS scores from disease onset. The progressive course, unaffected by age at RR onset, was only modestly affected by age at SP onset. Among primary progressive and RR/SP patients, median ages at attainment of DSS scores were strikingly similar: DSS = 6, 49 vs 48 years; DSS = 8, 58 vs 58 years; and DSS = 10, 78 years for both (p = NS for all comparisons).

CONCLUSIONS

Development of SP is the dominant determinant of long-term prognosis, independent of disease duration and early relapse frequency. Age independently affects disability development primarily by changing probability and latency of SP onset, with little effect on the progressive course.

Association between walking speed and age in healthy, free-living individuals using mobile accelerometry--a cross-sectional study

CONTEXT

Walking speed is a fundamental parameter of human motion and is increasingly considered as an important indicator of individuals' health status.

OBJECTIVE

To evaluate the relationship of gait parameters, and demographic and physical characteristics in healthy men and women.

DESIGN, SETTING, AND PARTICIPANTS

Recruitment of a subsample (n = 358) of male and female blood donors taking part in the Cambridge CardioResource study. Collection of demographic data, measurement of physical characteristics (height, weight and blood pressure) and assessment of 7-day, free-living activity parameters using accelerometry and a novel algorithm to measure walking speed. Participants were a median (interquartile range[IQR]) age of 49 (16) years; 45% women; and had a median (IQR) BMI of 26 (5.4).

MAIN OUTCOME MEASURE

Walking speed.

RESULTS

In this study, the hypothesis that walking speed declines with age was generated using an initial 'open' dataset. This was subsequently validated in a separate 'closed' dataset that showed a decrease of walking speed of -0.0037 m/s per year. This is equivalent to a difference of 1.2 minutes, when walking a distance of 1 km aged 20 compared to 60 years. Associations between walking speed and other participant characteristics (i.e. gender, BMI and blood pressure) were non-significant. BMI was negatively correlated with the number of walking and running steps and longest non-stop distance.

CONCLUSION

This is the first study using accelerometry which shows an association between walking speed and age in free-living, healthy individuals. Absolute values of gait speed are comparable to published normal ranges in clinical settings. This study highlights the potential use of mobile accelerometry to assess gait parameters which may be indicative of future health outcomes in healthy individuals.

HAGIL (Hamburg Vigil Study): a randomized placebo-controlled double-blind study with modafinil for treatment of fatigue in patients with multiple sclerosis

Objective: To reassess the effect of modafinil, a wakefulness-promoting artificial psychostimulant, on fatigue and neuropsychological measures in patients with multiple sclerosis.

Methods: Multiple sclerosis (MS) patients with a baseline score of ≥4 on the Fatigue Severity Scale (FSS) and an Expanded Disability Status Scale score <7 were eligible for the 8-week randomized, double-blind, placebo-controlled study. Modafinil was dosed up to 200 mg/day within 1 week. Assessments were performed at baseline and after 4 and 8 weeks. The primary outcome parameter was the mean change of the FSS mean score. Secondary outcome variables were other questionnaires covering fatigue, daytime sleepiness and sleep quality. Cognitive impairment was assessed by the oral version of the Symbol Digit Modalities Test (SDMT) and the Paced Auditory Serial Addition Test (PASAT).

Results: The study included 121 MS patients. Dropout rate was 9%. Both treatment groups showed improvements through time. While mean FSS at 8 weeks showed a trend difference between groups in the intention-to-treat analysis, the primary endpoint was not met. Assessment of cognitive impairment by SDMT and PASAT showed contradictory results. All other secondary endpoints were not met. There was no major safety concern.

Conclusions: In general, the study does not support modafinil as an effective treatment for MS fatigue. However, the study shows the need for new study designs and endpoints in MS fatigue studies.

Over-optimism in bioinformatics: an illustration

MOTIVATION

In statistical bioinformatics research, different optimization mechanisms potentially lead to 'over-optimism' in published papers. So far, however, a systematic critical study concerning the various sources underlying this over-optimism is lacking.

RESULTS

We present an empirical study on over-optimism using high-dimensional classification as example. Specifically, we consider a 'promising' new classification algorithm, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. While this approach yields poor results in terms of error rate, we quantitatively demonstrate that it can artificially seem superior to existing approaches if we 'fish for significance'. The investigated sources of over-optimism include the optimization of datasets, of settings, of competing methods and, most importantly, of the method's characteristics. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should always be demonstrated on independent validation data.

AVAILABILITY

The R codes and relevant data can be downloaded from http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/overoptimism/, such that the study is completely reproducible.

Prognosis of the individual course of disease: the elements of time, heterogeneity and precision

There is no gold standard in monitoring disease activity for clinical trials in multiple sclerosis. Various outcome measures, including relapses, disability and magnetic resonance imaging (MRI) measures have been used to demonstrate the efficacy of the different available therapies for multiple sclerosis. Recently, the potential limitations of these measures have received increasing attention, and these have stimulated research into more appropriate and sensitive outcome measures for clinical trials. For example, it has been shown that widely-used MRI measures add little, if any, independent information to that provided by more clinically relevant measures such as relapses and disability. Similarly, the Expanded Disability status Scale (EDSS), which is the most widely-used measure of disability related to multiple sclerosis, is insufficiently sensitive to detect robust changes in disability over the timeframes usually used in clinical trials. An alternative to the EDSS is the Multiple Sclerosis Severity Score (MSSS), a severity scale which relates clinical disability to disease duration. The MSSS was originally developed from a database of nearly ten thousand patients from eleven European countries and Australia and has since been reproduced in an independent dataset of 1134 patients from the placebo arms of randomised clinical trials. Based on the MSSS score, disease severity can be defined, which shows stability over time and may provide evidence-based decision support for patient management. Another alternative to measure disability is the objective quantification of physical activity. There is evidence that recent developments in pervasive computing using tiny accelerometers may have the potential to increase the reliability and precision of motor assessment, especially in the mid-range of the EDSS. The outcome measures discussed have potential use as online tools for evidence-based decision support which are increasingly being used in medical research and clinical decision-making.

Optimal classifier selection and negative bias in error rate estimation: an empirical study on high-dimensional prediction

BACKGROUND

In biometric practice, researchers often apply a large number of different methods in a "trial-and-error" strategy to get as much as possible out of their data and, due to publication pressure or pressure from the consulting customer, present only the most favorable results. This strategy may induce a substantial optimistic bias in prediction error estimation, which is quantitatively assessed in the present manuscript. The focus of our work is on class prediction based on high-dimensional data (e.g. microarray data), since such analyses are particularly exposed to this kind of bias.

METHODS

In our study we consider a total of 124 variants of classifiers (possibly including variable selection or tuning steps) within a cross-validation evaluation scheme. The classifiers are applied to original and modified real microarray data sets, some of which are obtained by randomly permuting the class labels to mimic non-informative predictors while preserving their correlation structure.

RESULTS

We assess the minimal misclassification rate over the different variants of classifiers in order to quantify the bias arising when the optimal classifier is selected a posteriori in a data-driven manner. The bias resulting from the parameter tuning (including gene selection parameters as a special case) and the bias resulting from the choice of the classification method are examined both separately and jointly.

CONCLUSIONS

The median minimal error rate over the investigated classifiers was as low as 31% and 41% based on permuted uninformative predictors from studies on colon cancer and prostate cancer, respectively. We conclude that the strategy to present only the optimal result is not acceptable because it yields a substantial bias in error rate estimation, and suggest alternative approaches for properly reporting classification accuracy.

CMA: a comprehensive Bioconductor package for supervised classification with high dimensional data

BACKGROUND

For the last eight years, microarray-based classification has been a major topic in statistics, bioinformatics and biomedicine research. Traditional methods often yield unsatisfactory results or may even be inapplicable in the so-called "p >> n" setting where the number of predictors p by far exceeds the number of observations n, hence the term "ill-posed-problem". Careful model selection and evaluation satisfying accepted good-practice standards is a very complex task for statisticians without experience in this area or for scientists with limited statistical background. The multiplicity of available methods for class prediction based on high-dimensional data is an additional practical challenge for inexperienced researchers.

RESULTS

In this article, we introduce a new Bioconductor package called CMA (standing for "Classification for MicroArrays") for automatically performing variable selection, parameter tuning, classifier construction, and unbiased evaluation of the constructed classifiers using a large number of usual methods. Without much time and effort, users are provided with an overview of the unbiased accuracy of most top-performing classifiers. Furthermore, the standardized evaluation framework underlying CMA can also be beneficial in statistical research for comparison purposes, for instance if a new classifier has to be compared to existing approaches.

CONCLUSION

CMA is a user-friendly comprehensive package for classifier construction and evaluation implementing most usual approaches. It is freely available from the Bioconductor website at (http://bioconductor.org/packages/2.3/bioc/html/CMA.html).

Evaluating microarray-based classifiers: an overview

For the last eight years, microarray-based class prediction has been the subject of numerous publications in medicine, bioinformatics and statistics journals. However, in many articles, the assessment of classification accuracy is carried out using suboptimal procedures and is not paid much attention. In this paper, we carefully review various statistical aspects of classifier evaluation and validation from a practical point of view. The main topics addressed are accuracy measures, error rate estimation procedures, variable selection, choice of classifiers and validation strategy.