Re: Felix K.-H. Chun, Markus Graefen, Alberto Briganti, Andrea Gallina, Julia Hopp, Michael W. Kattan, Hartwig Huland and Pierre I. Karakiewicz. Initial Biopsy Outcome Prediction—Head-to-Head Comparison of a Logistic Regression-Based Nomogram versus Artificial Neural Network. Eur Urol 2007;51:1236–43

External validation of an artificial neural network and two nomograms for prostate cancer detection

Background. Multivariate models are used to increase prostate cancer (PCa) detection rate and to reduce unnecessary biopsies. An external validation of the artificial neural network (ANN) "ProstataClass" (ANN-Charité) was performed with daily routine data. Materials and Methods. The individual ANN predictions were generated with the use of the ANN application for PSA and free PSA assays, which rely on age, tPSA, %fPSA, prostate volume, and DRE (ANN-Charité). Diagnostic validity of tPSA, %fPSA, and the ANN was evaluated by ROC curve analysis and comparisons of observed versus predicted probabilities. Results. Overall, 101 (35.8%) PCa were detected. The areas under the ROC curve (AUCs) were 0.501 for tPSA, 0.669 for %fPSA, 0.694 for ANN-Charité, 0.713 for nomogram I, and 0.742 for nomogram II, showing a significant advantage for nomogram II (P = 0.009) compared with %fPSA while the other model did not differ from %fPSA (P = 0.15 and P = 0.41). All models overestimated the predicted PCa probability. Conclusions. Beside ROC analysis, calibration is an important tool to determine the true value of using a model in clinical practice. The worth of multivariate models is limited when external validations were performed without knowledge of the circumstances of the model's development.

Evaluation of machine learning classifiers in keratoconus detection from orbscan II examinations

PURPOSE

To evaluate the performance of support vector machine, multi-layer perceptron and radial basis function neural network as auxiliary tools to identify keratoconus from Orbscan II maps.

METHODS

A total of 318 maps were selected and classified into four categories: normal (n = 172), astigmatism (n = 89), keratoconus (n = 46) and photorefractive keratectomy (n = 11). For each map, 11 attributes were obtained or calculated from data provided by the Orbscan II. Ten-fold cross-validation was used to train and test the classifiers. Besides accuracy, sensitivity and specificity, receiver operating characteristic (ROC) curves for each classifier were generated, and the areas under the curves were calculated.

RESULTS

The three selected classifiers provided a good performance, and there were no differences between their performances. The area under the ROC curve of the support vector machine, multi-layer perceptron and radial basis function neural network were significantly larger than those for all individual Orbscan II attributes evaluated (p < 0.05).

CONCLUSION

Overall, the results suggest that using a support vector machine, multi-layer perceptron classifiers and radial basis function neural network, these classifiers, trained on Orbscan II data, could represent useful techniques for keratoconus detection.

Internal validation of an artificial neural network for prostate biopsy outcome

OBJECTIVES

To carry out an internal validation of the retrospectively trained artificial neural network (ANN) 'ProstataClass'.

METHODS

A prospectively collected database of 393 patients undergoing 8-12 core prostate biopsy was analyzed. Data of these patients were applied to the online available ANN 'ProstataClass' using the Elecsys total prostate-specific antigen (tPSA) and free PSA (fPSA) assays. Beside the internal validation of the ANN 'ProstataClass' an additional ANN (named as ANN internal validation: ANNiv) only using the 393 prospective patient data was evaluated. The new ANN model was constructed with the MATLAB Neural Network Toolbox. Diagnostic accuracy was evaluated by receiver operator characteristic (ROC) curves comparing the areas under the ROC curves (AUC) and specificities at 90% and 95% sensitivity.

RESULTS

Within a tPSA range of 1.0-22.8 ng/mL, 229 men (58.3%) had prostate cancer (PCa). tPSA, %fPSA and the number of positive digital rectal examinations (DRE) differed significantly from the cohort of patients of the ANN 'ProstataClass', whereas age and prostate volume were comparable. AUCs for tPSA, %fPSA and the ANN 'ProstataClass' were 0.527, 0.726 and 0.747 (P = 0.085 between %fPSA and ANN). The AUC of the ANNiv (0.754) was significantly better compared with %fPSA (P = 0.021), whereas the AUC of two ANN models built on external cohorts (0.726 and 0.729) showed no differences to %fPSA and the other ANN models.

CONCLUSIONS

Significant differences of DRE status and %fPSA medians decrease the power of the 'ProstataClass' ANN in the internal validation cohort. The effect of retrospective data evaluation the 'ProstataClass' cohort and prospective fPSA measurement may be responsible for %fPSA differences. All ANN models built with different PSA and fPSA assays performed equally if applied to the two cohorts.