Validation procedures in radiologic diagnostic models. Neural network and logistic regression

Custom-Made Porous Hydroxyapatite Cranioplasty in Patients with Tumor Versus Traumatic Brain Injury: A Single-Center Case Series

BACKGROUND

Cranioplasty is a common neurosurgical procedure with the goal of restoring skull integrity. Custom-made porous hydroxyapatite prostheses have long been used for cranial reconstruction in patients with traumatic brain injury. We present a large consecutive series of 2 groups of patients undergoing cranioplasty with hydroxyapatite custom bone and compare the adverse events (AEs) between the 2 groups.

METHODS

We examined a series of consecutive patients who underwent cranioplasty using custom-made porous hydroxyapatite implants following tumor resection and traumatic brain injury at a single center between March 2003 and May 2018. The implants were designed and produced according to the surgeon's specifications and based on the patient's computed tomography scan data obtained through a standardized protocol. AEs were recorded.

RESULTS

Information on 38 patients with tumor and 39 patients with traumatic brain injury was collected and analyzed. A significant difference in the timing of surgery was found between the 2 groups; single-stage surgery was performed in 84% of patients in the tumor versus 8% of those in the traumatic brain injury group (P < 0.0001). The rate of AEs was not significantly different between the 2 groups (P = 0.4309) and was not related to the timing of surgery.

CONCLUSIONS

Custom-made hydroxyapatite cranioplasty is a solution for cranial reconstruction in patients with cranial tumors. The low incidence of AEs in a consecutive series of patients with either trauma or tumors demonstrates that these prostheses represent a safe solution independent of the characteristics of cases.

Differential diagnosis of parkinsonian syndromes using dopamine transporter and perfusion SPECT

OBJECTIVE

We aimed to assess whether a combined analysis of dopamine transporter (DAT)- and perfusion-SPECT images (or either) could: (1) distinguish atypical parkinsonian syndromes (APS) from Lewy body diseases (LBD; majority Parkinson disease [PD]), and (2) differentiate among APS subgroups (progressive supranuclear palsy [PSP], corticobasal syndrome [CBS], and multiple system atrophy [MSA]).

METHODS

We recruited consecutive patients with neurodegenerative parkinsonian syndromes (LBD, n = 46; APS, n = 33). Individual [¹²³I]FP-CIT- and [¹²³I]iodoamphetamine-SPECT images were coregistered onto anatomical MRI segmented into brain regions. Striatal DAT activity and regional perfusion were extracted from each brain region for each patient and submitted to logistic regression analyses. Stepwise procedures were used to select predictors that should be included in the models to distinguish APS from LBD, and differentiate among the APS subgroups. Receiver-operating characteristic (ROC) analyses were performed to measure diagnostic power. Leave-one-out cross-validation (LOOCV) was performed to evaluate the diagnostic accuracy.

RESULTS

The model to discriminate APS from LBD showed that the area under the ROC curve (AUC) was 0.923, while the total diagnostic accuracy (TDA) was 86.1% in LOOCV. In the model to distinguish PSP, CBS, and MSA from LBD, the AUC/TDA values were 0.978/94.6%, 0.978/87.0%, and 0.880/80.3%, respectively. In the model to differentiate between CBS and MSA, MSA and PSP, and PSP and CBS, the AUC/TDA values were 0.967/91.3%, 0.920/88.0%, 0.875/77.8%, respectively.

CONCLUSION

An image-based automated classification using striatal DAT activity and regional perfusion patterns provided a good performance in the differential diagnosis of neurodegenerative parkinsonian syndromes without clinical information.

Classifier performance prediction for computer-aided diagnosis using a limited dataset

In a practical classifier design problem, the true population is generally unknown and the available sample is finite-sized. A common approach is to use a resampling technique to estimate the performance of the classifier that will be trained with the available sample. We conducted a Monte Carlo simulation study to compare the ability of the different resampling techniques in training the classifier and predicting its performance under the constraint of a finite-sized sample. The true population for the two classes was assumed to be multivariate normal distributions with known covariance matrices. Finite sets of sample vectors were drawn from the population. The true performance of the classifier is defined as the area under the receiver operating characteristic curve (AUC) when the classifier designed with the specific sample is applied to the true population. We investigated methods based on the Fukunaga-Hayes and the leave-one-out techniques, as well as three different types of bootstrap methods, namely, the ordinary, 0.632, and 0.632+ bootstrap. The Fisher's linear discriminant analysis was used as the classifier. The dimensionality of the feature space was varied from 3 to 15. The sample size n2 from the positive class was varied between 25 and 60, while the number of cases from the negative class was either equal to n2 or 3n2. Each experiment was performed with an independent dataset randomly drawn from the true population. Using a total of 1000 experiments for each simulation condition, we compared the bias, the variance, and the root-mean-squared error (RMSE) of the AUC estimated using the different resampling techniques relative to the true AUC (obtained from training on a finite dataset and testing on the population). Our results indicated that, under the study conditions, there can be a large difference in the RMSE obtained using different resampling methods, especially when the feature space dimensionality is relatively large and the sample size is small. Under this type of conditions, the 0.632 and 0.632+ bootstrap methods have the lowest RMSE, indicating that the difference between the estimated and the true performances obtained using the 0.632 and 0.632+ bootstrap will be statistically smaller than those obtained using the other three resampling methods. Of the three bootstrap methods, the 0.632+ bootstrap provides the lowest bias. Although this investigation is performed under some specific conditions, it reveals important trends for the problem of classifier performance prediction under the constraint of a limited dataset.

Population pharmacokinetic and pharmacodynamic models of remifentanil in healthy volunteers using artificial neural network analysis

AIMS

An ordinary sigmoid E(max) model could not predict overshoot of electroencephalographic approximate entropy (ApEn) during recovery from remifentanil effect in our previous study. The aim of this study was to evaluate the ability of an artificial neural network (ANN) to predict ApEn overshoot and to evaluate the predictive performance of the pharmacokinetic model, and pharmacodynamic models of ANN with respect to data used.

METHODS

Using a reduced number of ApEn instances (n = 1581) to make NONMEM modelling feasible and complete ApEn data (n = 24 509), the presence of overshoot was assessed. A total of 1077 measured remifentanil concentrations and ApEn data, and a total of 24 509 predicted concentrations and ApEn data were used in the pharmacodynamic model A and B of ANN, respectively. The testing subset of model B (n = 7352) was used to evaluate the ability of ANN to predict overshoot of ApEn. Mean squared error (MSE) was calculated to evaluate the predictive performance of the ANN models.

RESULTS

With complete ApEn data, ApEn overshoot was observed in 66.7% of subjects, but only in 37% with a reduced number of ApEn instances. The ANN model B predicted 77.8% of ApEn overshoot. MSE (95% confidence interval) was 57.1 (3.22, 71.03) for the pharmacokinetic model, 0.148 (0.004, 0.007) for model A and 0.0018 (0.0017, 0.0019) for model B.

CONCLUSIONS

The reduced ApEn instances interfered with the approximation of true electroencephalographic response. ANN predicted 77.8% of ApEn overshoot. The predictive performance of model B was significantly better than that of model A.

Risk adjusted surgical audit in gynaecological oncology: P-POSSUM does not predict outcome

AIMS

To assess the Physiological and Operative Severity Score for the enumeration of mortality and morbidity (POSSUM) and its validity for use in gynaecological oncology surgery.

METHODS

All patients undergoing gynaecological oncology surgery at the Northern Gynaecological Oncology Centre (NGOC) Gateshead, UK over a period of 12months (2002-2003) were assessed prospectively. Mortality and morbidity predictions using the Portsmouth modification of the POSSUM algorithm (P-POSSUM) were compared to the actual outcomes. Performance of the model was also evaluated using the Hosmer and Lemeshow Chi square statistic (testing the goodness of fit).

RESULTS

During this period 468 patients were assessed. The P-POSSUM appeared to over predict mortality rates for our patients. It predicted a 7% mortality rate for our patients compared to an observed rate of 2% (35 predicted deaths in comparison to 10 observed deaths), a difference that was statistically significant (H&L chi(2)=542.9, d.f. 8, p<0.05).

CONCLUSION

The P-POSSUM algorithm overestimates the risk of mortality for gynaecological oncology patients undergoing surgery. The P-POSSUM algorithm will require further adjustments prior to adoption for gynaecological cancer surgery as a risk adjusted surgical audit tool.

Prediction of lung adenocarcinoma without vessel invasion: a CT scan volumetric analysis

STUDY OBJECTIVES

Patients with lung adenocarcinoma without vessel invasion have a favorable prognosis after resection and are among the candidates for limited surgery. The purpose of the present study was to predict lung adenocarcinoma without vessel invasion based on a volumetric analysis of the lesion with a CT scan prior to the operation.

METHODS

CT scan images were obtained in 288 consecutive patients with adenocarcinoma of the lung before surgical resection. Total tumor volume, the volume of the nonsolid component, and the proportion occupied by the nonsolid component were calculated by the perimeter method. The performance of the derived logistic regression model and the volumetric results were evaluated by receiver operating characteristic analysis. The model derived for the prediction of tumors without vessel invasion was assessed by means of the leave-one-out cross-validation technique.

RESULTS

The pathologic diagnosis was adenocarcinoma with vessel invasion in 160 cases, and without vessel invasion in 128 cases. The median total tumor volume, the median volume of the nonsolid component, and median proportion occupied by the nonsolid component were 1,123.7 mm(3), 253.4 mm(3), and 58.0%, respectively. With the derivation of the predictive rule, stepwise regression yielded the following five features: the proportion occupied by the nonsolid component; spiculation; pleural indentation; gender; and tumor size. The Az value, a measure of diagnostic power represented as the area under the curve, was 0.957 for prediction of lung adenocarcinoma without vessel invasion. The cross-validation accuracy achieved by applying the rule was 90.3%.

CONCLUSIONS

The proportion occupied by the nonsolid component based on a CT scan volumetric analysis was a reliable predictor of tumors without vessel invasion in patients with adenocarcinoma of the lung.

Clinical application of artificial neural network (ANN) modeling to predict pharmacokinetic parameters of severely ill patients

Artificial neural network (ANN) modeling was used to evaluate the pharmacokinetics of aminoglycosides (arbekacin sulfate and amikacin sulfate) in severely ill patients. The plasma level was predicted by ANN modeling using parameters related to the severity of the patient's condition and the predictive performance was shown to be better than could be achieved using multiple regression analysis. These results indicate that there is a non-linear relationship between the pharmacokinetics of aminoglycosides and the severity of the patient's condition, and this should be taken into account when determining the dose for severely ill patients. Patients whose plasma levels are likely to fall below the effective level can be identified by ANN modeling with a predictive sensitivity and specificity superior to multivariate logistic regression analysis. The predictable range should be inferred from the data structure before the modeling in order to improve the predictive performance. The volume of distribution (Vd) in the normal range was weakly predicted by ANN modeling from the patients' data. Prediction of clearance by ANN modeling was poorer than that obtained from serum creatinine concentration by linear regression analysis. These results suggest that the input-output relationship (linear or non-linear) should be taken into account in selecting the modeling method. Linear modeling can give better predictive performance for linear systems and non-linear modeling can give better predictive performance for non-linear systems. In general, the performance of ANN modeling was superior to linear modeling for PK/PD prediction. For accurate modeling, a predictable range should be inferred from the data structure before the analysis. Restriction of the predictable region, as determined from the data structure, produced an increase in prediction performance. When applying ANN modeling in clinical settings, the predictive performance and predictable region should be investigated in detail to avoid the risk of harm to severely ill patients.

External validity of predictive models: a comparison of logistic regression, classification trees, and neural networks

BACKGROUND AND OBJECTIVE

The utility of predictive models depends on their external validity, that is, their ability to maintain accuracy when applied to patients and settings different from those on which the models were developed. We report a simulation study that compared the external validity of standard logistic regression (LR1), logistic regression with piecewise-linear and quadratic terms (LR2), classification trees, and neural networks (NNETs).

METHODS

We developed predictive models on data simulated from a specified population and on data from perturbed forms of the population not representative of the original distribution. All models were tested on new data generated from the population.

RESULTS

The performance of LR2 was superior to that of the other model types when the models were developed on data sampled from the population (mean receiver operating characteristic [ROC] areas 0.769, 0.741, 0.724, and 0.682, for LR2, LR1, NNETs, and trees, respectively) and when they were developed on nonrepresentative data (mean ROC areas 0.734, 0.713, 0.703, and 0.667). However, when the models developed using nonrepresentative data were compared with models developed from data sampled from the population, LR2 had the greatest loss in performance.

CONCLUSION

Our results highlight the necessity of external validation to test the transportability of predictive models.

Diagnóstico de las lesiones de la calota. Selección de variables por redes neuronales y regresión logística

OBJECTIVES

To establish the minimun set of features needed in the diagnosis of calvarial lesions using computed tomography (CT) and to assess the accuracy of logistic regression (LR) and artificial neural networks (NN) for their diagnosis.

MATERIAL AND METHODS

167 patients with calvarial lesions as the only known disease were enrolled. The clinical and CT data were used for LR and NN models. Both models were tested with the jacknife method. The final results of each model were compared using the area under ROC curves (A 2 ).

RESULTS

The lesions were 73.1 % benign and 26.9% malignant. There was no statistically significant difference between LR and NN in differentiating malignancy. In characterizing the histologic diagnoses, NN was statistically superior to LR. Important NN features needed for malignancy classification were age and edge definition, and for the histologic diagnoses matrix, marginal sclerosis and age.

CONCLUSIONS

A minimum four features is needed to diagnose these lesions, not being important patients' symptoms. NNs offer wide possibilities over statistics for the calvarial lesions study besides a superior diagnostic performance.

Predicting the presence of acute pulmonary embolism: a comparative analysis of the artificial neural network, logistic regression, and threshold models

OBJECTIVE

The objective of this study was to determine whether an artificial neural network, a new data analysis method, offers increased performance over conventional logistic regression in predicting the presence of a pulmonary embolism for patients in a well-known data set.

MATERIALS AND METHODS

Data from the 1064 patients who received an angiographically based diagnosis of pulmonary embolism in the Prospective Investigation of Pulmonary Embolism Diagnosis study were encoded using a previously described method. The 21 input variables represented abnormalities identified on each patient's ventilation-perfusion scan and chest radiograph. Two methods-an artificial neural network with one hidden layer and a multivariate logistic regression-were compared for accuracy in predicting the presence or absence of pulmonary embolism on subsequent pulmonary arteriography.

RESULTS

No significant difference was observed between the two methods. Areas under the receiver operating characteristic curves +/- standard deviation were 0.78 +/- 0.02 for the artificial neural network model and 0.79 +/- 0.02 for the logistic regression model. Furthermore, use of these two methods resulted in no more diagnostic accuracy than did the use of a simple threshold model based only on the number of subsegmental perfusion defects, which was the dominant input variable.

CONCLUSION

In the study population, the usefulness of data from ventilation-perfusion scans as predictors of the presence of a pulmonary embolism was similar for the three analytic methods, a finding that reinforces the importance of making comparisons to simpler or more established methods when performing studies involving complex analytic models, such as artificial neural networks.