The Accuracy of Artificial Neural Networks in Predicting Long-term Outcome After Traumatic Brain Injury

Deep Learning to Predict Traumatic Brain Injury Outcomes in the Low-Resource Setting

OBJECTIVE

Traumatic brain injury (TBI) disproportionately affects low- and middle-income countries (LMICs). In these settings, accurate patient prognostication is both difficult and essential for high-quality patient care. With the ultimate goal of enhancing TBI triage in LMICs, we aim to develop the first deep learning model to predict outcomes after TBI and compare its performance with that of less complex algorithms.

METHODS

TBI patients' data were prospectively collected in Kampala, Uganda, from 2016 to 2020. To predict good versus poor outcome at hospital discharge, we created deep neural network, shallow neural network, and elastic-net regularized logistic regression models. Predictors included 13 easily acquirable clinical variables. We assessed model performance with 5-fold cross-validation to calculate areas under both the receiver operating characteristic curve and precision-recall curve (AUPRC), in addition to standardized partial AUPRC to focus on comparisons at clinically relevant operating points.

RESULTS

We included 2164 patients for model training, of which 12% had poor outcomes. The deep neural network performed best as measured by the area under the receiver operating characteristic curve (0.941) and standardized partial AUPRC in region maximizing recall (0.291), whereas the shallow neural network was best by the area under the precision-recall curve (0.770). In several other comparisons, the elastic-net regularized logistic regression was noninferior to the neural networks.

CONCLUSIONS

We present the first use of deep learning for TBI prognostication, with an emphasis on LMICs, where there is great need for decision support to allocate limited resources. Optimal algorithm selection depends on the specific clinical setting; deep learning is not a panacea, though it may have a role in these efforts.

Machine Learning in the Prediction of Trauma Outcomes: A Systematic Review

STUDY OBJECTIVE

Machine learning models carry unique potential as decision-making aids and prediction tools for improving patient care. Traumatically injured patients provide a uniquely heterogeneous population with severe injuries that can be difficult to predict. Given the relative infancy of machine learning applications in medicine, this systematic review aimed to better understand the current state of machine learning development and implementation to help create a basis for future research.

METHODS

We conducted a systematic review from inception to May 2021, using Embase, MEDLINE through Ovid, Web of Science, Google Scholar, and relevant gray literature, for uses of machine learning in predicting the outcomes of trauma patients. The screening and data extraction were performed by 2 independent reviewers.

RESULTS

Of the 14,694 identified articles screened, 67 were included for data extraction. Artificial neural networks comprised the most commonly used model, and mortality was the most prevalent outcome of interest. In terms of machine learning model development, there was a lack of studies that employed external validation, feature selection methods, and performed formal calibration testing. Significant heterogeneity in reporting was also observed between the machine learning models employed, patient populations, performance metrics, and features employed.

CONCLUSION

This review highlights the heterogeneity in the development and reporting of machine learning models for the prediction of trauma outcomes. While these models present an area of opportunity as an ancillary to clinical decision-making, we recommend more standardization and rigorous guidelines for the development of future models.

Capture rates of comorbidity measures at inpatient rehabilitation facilities after a stroke or brain injury

BACKGROUND

Comorbidity indices have been used to represent the overall medical complexity of patient populations in clinical research; however, it is not known how well they capture the comorbidities of patients with a stroke or brain injury admitted to inpatient rehabilitation facilities (IRFs).

OBJECTIVE

To determine how well commonly used comorbidity indices capture the comorbidities of patients admitted to IRFs after a stroke or brain injury.

DESIGN

Cross-sectional, retrospective study.

SETTING

IRFs nationwide.

PARTICIPANTS

Adults from four impairment groups: (1) hemorrhagic stroke, (2) ischemic stroke, (3) nontraumatic brain injury (NTBI), and (4) traumatic brain injury (TBI).

MAIN OUTCOME MEASURES

International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes were extracted from the Uniform Data System for Medical Rehabilitation (UDSMR) for IRF discharges from October 1, 2015 to December 31, 2017. The percentage of discharges captured by Deyo-Charlson, Elixhauser, and Centers for Medicare and Medicaid Services (CMS) tiers was determined, as was the percentage of comorbidities captured. These measures were also compared with respect to their ability to capture chronic medical complexity by examining the percentage of codes captured after removal of codes deemed to represent hospital complications or sequela of the admission diagnosis.

RESULTS

The percentage of discharges without at least one ICD-10-CM code captured by any index ranged from 0.3%-3.8%. The percentage of comorbidities with a prevalence exceeding 1% captured by at least one index ranged from 37.1%-43.6%. Chronic comorbidities were most likely to be captured by Elixhauser (40.7%-44.4%), followed by Deyo-Charlson (7.8%-9.6%), then CMS tiers (4.5%-6.9%). Existing comorbidity measures capture most IRF discharges related to a brain injury or stroke, whereas most medical comorbidities escape representation. Several common, functionally relevant diagnoses were not captured.

CONCLUSION

The use of comorbidity indices in the IRF neurologic injury population should account for the fact that these measures miss several common, important comorbidities.

Machine learning analyses can differentiate meningioma grade by features on magnetic resonance imaging

OBJECTIVEPrognostication and surgical planning for WHO grade I versus grade II meningioma requires thoughtful decision-making based on radiographic evidence, among other factors. Although conventional statistical models such as logistic regression are useful, machine learning (ML) algorithms are often more predictive, have higher discriminative ability, and can learn from new data. The authors used conventional statistical models and an array of ML algorithms to predict atypical meningioma based on radiologist-interpreted preoperative MRI findings. The goal of this study was to compare the performance of ML algorithms to standard statistical methods when predicting meningioma grade.METHODSThe cohort included patients aged 18-65 years with WHO grade I (n = 94) and II (n = 34) meningioma in whom preoperative MRI was obtained between 1998 and 2010. A board-certified neuroradiologist, blinded to histological grade, interpreted all MR images for tumor volume, degree of peritumoral edema, presence of necrosis, tumor location, presence of a draining vein, and patient sex. The authors trained and validated several binary classifiers: k-nearest neighbors models, support vector machines, naïve Bayes classifiers, and artificial neural networks as well as logistic regression models to predict tumor grade. The area under the curve-receiver operating characteristic curve was used for comparison across and within model classes. All analyses were performed in MATLAB using a MacBook Pro.RESULTSThe authors included 6 preoperative imaging and demographic variables: tumor volume, degree of peritumoral edema, presence of necrosis, tumor location, patient sex, and presence of a draining vein to construct the models. The artificial neural networks outperformed all other ML models across the true-positive versus false-positive (receiver operating characteristic) space (area under curve = 0.8895).CONCLUSIONSML algorithms are powerful computational tools that can predict meningioma grade with great accuracy.

Using an artificial neural network to predict traumatic brain injury

In BriefPediatric traumatic brain injury (TBI) is common, but not all injuries require hospitalization. A computational tool for ruling-in patients who will have clinically relevant TBI (CRTBI) would be valuable, providing an evidence-based mechanism for safe discharge. Here, using data from 12,902 patients from the Pediatric Emergency Care Applied Research Network (PECARN) TBI data set, the authors utilize artificial intelligence to predict CRTBI using radiologist-interpreted CT information with > 99% sensitivity and an AUC of 0.99.

The Impact of Oversampling with SMOTE on the Performance of 3 Classifiers in Prediction of Type 2 Diabetes

OBJECTIVE

To evaluate the impact of the synthetic minority oversampling technique (SMOTE) on the performance of probabilistic neural network (PNN), naïve Bayes (NB), and decision tree (DT) classifiers for predicting diabetes in a prospective cohort of the Tehran Lipid and Glucose Study (TLGS).

METHODS

. Data of the 6647 nondiabetic participants, aged 20 years or older with more than 10 years of follow-up, were used to develop prediction models based on 21 common risk factors. The minority class in the training dataset was oversampled using the SMOTE technique, at 100%, 200%, 300%, 400%, 500%, 600%, and 700% of its original size. The original and the oversampled training datasets were used to establish the classification models. Accuracy, sensitivity, specificity, precision, F-measure, and Youden's index were used to evaluated the performance of classifiers in the test dataset. To compare the performance of the 3 classification models, we used the ROC convex hull (ROCCH).

RESULTS

Oversampling the minority class at 700% (completely balanced) increased the sensitivity of the PNN, DT, and NB by 64%, 51%, and 5%, respectively, but decreased the accuracy and specificity of the 3 classification methods. NB had the best Youden's index before and after oversampling. The ROCCH showed that PNN is suboptimal for any class and cost conditions.

CONCLUSIONS

To determine a classifier with a machine learning algorithm like the PNN and DT, class skew in data should be considered. The NB and DT were optimal classifiers in a prediction task in an imbalanced medical database.

In-hospital mortality after traumatic brain injury surgery: a nationwide population-based comparison of mortality predictors used in artificial neural network and logistic regression models

Object

Most reports compare artificial neural network (ANN) models and logistic regression models in only a single data set, and the essential issue of internal validity (reproducibility) of the models has not been adequately addressed. This study proposes to validate the use of the ANN model for predicting in-hospital mortality after traumatic brain injury (TBI) surgery and to compare the predictive accuracy of ANN with that of the logistic regression model.

Methods

The authors of this study retrospectively analyzed 16,956 patients with TBI nationwide who were surgically treated in Taiwan between 1998 and 2009. For every 1000 pairs of ANN and logistic regression models, the area under the receiver operating characteristic curve (AUC), Hosmer-Lemeshow statistics, and accuracy rate were calculated and compared using paired t-tests. A global sensitivity analysis was also performed to assess the relative importance of input parameters in the ANN model and to rank the variables in order of importance.

Results

The ANN model outperformed the logistic regression model in terms of accuracy in 95.15% of cases, in terms of Hosmer-Lemeshow statistics in 43.68% of cases, and in terms of the AUC in 89.14% of cases. The global sensitivity analysis of in-hospital mortality also showed that the most influential (sensitive) parameters in the ANN model were surgeon volume followed by hospital volume, Charlson comorbidity index score, length of stay, sex, and age.

Conclusions

This work supports the continued use of ANNs for predictive modeling of neurosurgery outcomes. However, further studies are needed to confirm the clinical efficacy of the proposed model.

Predicting Survival in Patients With Brain Metastases Treated With Radiosurgery Using Artificial Neural Networks

BACKGROUND:

Artificial neural networks (ANNs) excel at analyzing challenging data sets and can be exceptional tools for decision support in clinical environments. The present study pilots the use of ANNs for determining prognosis in neuro-oncology patients.

OBJECTIVE:

To determine whether ANNs perform better at predicting 1-year survival in a group of patients with brain metastasis compared with traditional predictive tools.

METHODS:

ANNs were trained on a multi-institutional data set of radiosurgery patients to predict 1-year survival on the basis of several input factors. A single ANN, an ensemble of 5 ANNs, and logistic regression analyses were compared for efficacy. Sensitivity analysis was used to identify important variables in the ANN model.

RESULTS:

A total of 196 patients were divided up into training, testing, and validation data sets consisting of 98, 49, and 49 patients, respectively. Patients surviving at 1 year tended to be female (P = .001) and of good performance status (P = .01) and to have favorable primary tumor histology (P = .001). The pooled voting of 5 ANNs performed significantly better than the multivariate logistic regression model (P = .02), with areas under the curve of 84% and 75%, respectively. The ensemble also significantly outperformed 2 commonly used prognostic indexes. Primary tumor subtype and performance status were identified on sensitivity analysis to be the most important variables for the ANN.

CONCLUSION:

ANNs outperform traditional statistical tools and scoring indexes for predicting individual patient prognosis. Their facile implementation, robustness in the presence of missing data, and ability to continuously learn make them excellent choices for use in complicated clinical environments.

Comparisons of prediction models of quality of life after laparoscopic cholecystectomy: a longitudinal prospective study

BACKGROUND

Few studies of laparoscopic cholecystectomy (LC) outcome have used longitudinal data for more than two years. Moreover, no studies have considered group differences in factors other than outcome such as age and nonsurgical treatment. Additionally, almost all published articles agree that the essential issue of the internal validity (reproducibility) of the artificial neural network (ANN), support vector machine (SVM), Gaussian process regression (GPR) and multiple linear regression (MLR) models has not been adequately addressed. This study proposed to validate the use of these models for predicting quality of life (QOL) after LC and to compare the predictive capability of ANNs with that of SVM, GPR and MLR.

METHODOLOGY/PRINCIPAL FINDINGS

A total of 400 LC patients completed the SF-36 and the Gastrointestinal Quality of Life Index at baseline and at 2 years postoperatively. The criteria for evaluating the accuracy of the system models were mean square error (MSE) and mean absolute percentage error (MAPE). A global sensitivity analysis was also performed to assess the relative significance of input parameters in the system model and to rank the variables in order of importance. Compared to SVM, GPR and MLR models, the ANN model generally had smaller MSE and MAPE values in the training data set and test data set. Most ANN models had MAPE values ranging from 4.20% to 8.60%, and most had high prediction accuracy. The global sensitivity analysis also showed that preoperative functional status was the best parameter for predicting QOL after LC.

CONCLUSIONS/SIGNIFICANCE

Compared with SVM, GPR and MLR models, the ANN model in this study was more accurate in predicting patient-reported QOL and had higher overall performance indices. Further studies of this model may consider the effect of a more detailed database that includes complications and clinical examination findings as well as more detailed outcome data.

Application of data mining to the identification of critical factors in patient falls using a web-based reporting system

PURPOSE

The implementation of an information system has become a trend in healthcare institutions. How to identify variables related to patient safety among accumulated data has been viewed as a main issue. The purpose of this study was to identify critical factors related to patient falls through the application of data mining to available data through a hospital information system.

METHOD

Data on a total of 725 patient falls were obtained from a web-based nursing incident reporting system at a medical center in Taiwan. In the process of data mining, feature selection was applied as the first step, after which 10 critical factors were selected to predict the dependent variables (injury versus non-injury). An artificial neural network (ANN) analysis was applied to develop a predictive model and a multivariate stepwise logistic regression was performed for comparison purposes.

RESULTS

The ANN model produced the following results: a Receiver-Operating-Character (ROC) curve indicated 77% accuracy, the positive predictive value (PPV) was 68%, and the negative predictive value (NPV) was 72%; while the multivariate stepwise logistic regression only identified 3 variables (fall assessment, anti-psychosis medication and diuretics) as significant predictors with ROC curve of 42%, PPV of 26.24%, and NPV of 87.12%.

CONCLUSION

In addition to medication use such as anti-psychotic and diuretics, nursing intervention where a fall assessment is conducted could represent a critical factor related to outcomes of fall incidence.

Use of an artificial neural network to predict head injury outcome

Object

The authors describe the artificial neural network (ANN) as an innovative and powerful modeling tool that can be increasingly applied to develop predictive models in neurosurgery. They aimed to demonstrate the utility of an ANN in predicting survival following traumatic brain injury and compare its predictive ability with that of regression models and clinicians.

Methods

The authors designed an ANN to predict in-hospital survival following traumatic brain injury. The model was generated with 11 clinical inputs and a single output. Using a subset of the National Trauma Database, the authors “trained” the model to predict outcome by providing the model with patients for whom 11 clinical inputs were paired with known outcomes, which allowed the ANN to “learn” the relevant relationships that predict outcome. The model was tested against actual outcomes in a novel subset of 100 patients derived from the same database. For comparison with traditional forms of modeling, 2 regression models were developed using the same training set and were evaluated on the same testing set. Lastly, the authors used the same 100-patient testing set to evaluate 5 neurosurgery residents and 4 neurosurgery staff physicians on their ability to predict survival on the basis of the same 11 data points that were provided to the ANN. The ANN was compared with the clinicians and the regression models in terms of accuracy, sensitivity, specificity, and discrimination.

Results

Compared with regression models, the ANN was more accurate (p < 0.001), more sensitive (p < 0.001), as specific (p = 0.260), and more discriminating (p < 0.001). There was no difference between the neurosurgery residents and staff physicians, and all clinicians were pooled to compare with the 5 best neural networks. The ANNs were more accurate (p < 0.0001), more sensitive (p < 0.0001), as specific (p = 0.743), and more discriminating (p < 0.0001) than the clinicians.

Conclusions

When given the same limited clinical information, the ANN significantly outperformed regression models and clinicians on multiple performance measures. While this paradigm certainly does not adequately reflect a real clinical scenario, this form of modeling could ultimately serve as a useful clinical decision support tool. As the model evolves to include more complex clinical variables, the performance gap over clinicians and logistic regression models will persist or, ideally, further increase.

Hierarchical log linear analysis of admission blood parameters and clinical outcome following traumatic brain injury

PURPOSE

Numerous statistical methods have been utilised to generate predictive models that identify clinical and biochemical parameters of prognostic value following traumatic brain injury. While these methods provide an accurate statistical description between these variables and outcome, they are difficult to interpret intuitively. Hierarchical log linear analysis can be utilised to present the complex interactions between these variables and outcome visually.

METHODS

We compiled a database of 327 traumatic brain injury patients, their admission blood parameters, clinical admission parameters, and 6-month Glasgow Outcome Score. Seven variables (age, injury severity, Glasgow Coma Score, glucose, albumin, haemoglobin, white cell count) that correlated with outcome in a univariate analysis and two further variables, included on the basis of biological plausibility, (abnormal clotting and magnesium) were used to derive and present a hierarchical log linear model.

RESULTS

Seventeen (out of an original forty-five possible) inter-relationships between the chosen variables were identified as remaining in the hierarchical log linear model. This data is presented pictorially in a hierarchy demonstrating the directness of the statistical association between each of the variables and dichotomised outcome. Four variables within the hierarchical log linear model (age, raised serum glucose, low haemoglobin, Glasgow Coma Score) had a direct independent statistical relationship with outcome. The remaining five variables only had a statistical relationship with outcome via at least one other variable.

CONCLUSIONS

Hierarchical log linear analysis allows the presentation of multivariate, categorical data sets in a pictorial and more easily interpretable fashion.

Clinician vs mathematical statistical models: which is better at predicting an abnormal chest radiograph finding in injured patients?

OBJECTIVE

The purpose of this study was to determine if statistical models for prediction of chest injuries would outperform the clinician's (MD) ability to identify injured patients at risk for a thoracic injury diagnosed by chest radiograph (CXR).

DESIGN

A prospective observational study was done during a 12-month period.

SETTING

The study was conducted in a level I trauma center.

PATIENTS

Injured patients meeting trauma team activation criteria were enrolled to the study.

INTERVENTIONS

Physical examination findings by a clinician were interpreted and CXR was performed.

OUTCOME MEASURES

The accuracy of 2 mathematical models is compared against the accuracy of clinician's clinical judgment in predicting an injury by CXR. Two newly constructed multivariate models, binary logistic regression (LR) and classification and regression tree (CaRT) analysis, are compared to previously published data of clinician clinical assessment of probability of thoracic injury identified by CXR.

RESULTS

Data for 757 patients were analyzed. Classification and regression tree analysis developed a stepwise decision tree to determine which signs/symptoms were indicative of an abnormal CXR finding. The sensitivity (CaRT, 36.6%; LR, 36.3%; MD, 58.7%), specificity (CaRT, 98.3%; LR, 98.2%; MD, 96.4%), and error rates (CaRT, 0.93; LR, 0.94; MD, 0.82) show that the mathematical decision aids are less sensitive and risk more misclassification compared to clinician judgment in predicting an injury by CXR.

CONCLUSION

Clinician judgment was superior to mathematical decision aids for predicting an abnormal CXR finding in injured patients with chest trauma.