Letter to the Editor: “Bridging Machine Learning and Clinical Practice in Neurosurgery: Hurdles and Solutions”

Machine learning for outcome prediction of neurosurgical aneurysm treatment: Current methods and future directions

INTRODUCTION

Machine learning algorithms have received increased attention in neurosurgical literature for improved accuracy over traditional predictive methods. In this review, the authors sought to assess current applications of machine learning for outcome prediction of neurosurgical treatment of intracranial aneurysms and identify areas for future research.

METHODS

A PRISMA-compliant systematic review of the PubMed, MEDLINE, and EMBASE databases was conducted for all studies utilizing machine learning for outcome prediction of intracranial aneurysm treatment. Patient characteristics, machine learning methods, outcomes of interest, and accuracy metrics were recorded from included studies.

RESULTS

16 studies were ultimately included in qualitative synthesis. Studies primarily analyzed angiographic outcomes, functional outcomes, or complication prediction using clinical, radiological, or composite variables. The majority of included studies utilized supervised learning algorithms for analysis of dichotomized outcomes.

CONCLUSIONS

Commonly included variables were demographics, presentation variables (including ruptured or unruptured status), and treatment used. Areas for future research include increased generalizability across institutions and for smaller datasets, as well as development of front-end tools for clinical applicability of published algorithms.

Code-free machine learning for object detection in surgical video: a benchmarking, feasibility, and cost study

OBJECTIVE

While the utilization of machine learning (ML) for data analysis typically requires significant technical expertise, novel platforms can deploy ML methods without requiring the user to have any coding experience (termed AutoML). The potential for these methods to be applied to neurosurgical video and surgical data science is unknown.

METHODS

AutoML, a code-free ML (CFML) system, was used to identify surgical instruments contained within each frame of endoscopic, endonasal intraoperative video obtained from a previously validated internal carotid injury training exercise performed on a high-fidelity cadaver model. Instrument-detection performances using CFML were compared with two state-of-the-art ML models built using the Python coding language on the same intraoperative video data set.

RESULTS

The CFML system successfully ingested surgical video without the use of any code. A total of 31,443 images were used to develop this model; 27,223 images were uploaded for training, 2292 images for validation, and 1928 images for testing. The mean average precision on the test set across all instruments was 0.708. The CFML model outperformed two standard object detection networks, RetinaNet and YOLOv3, which had mean average precisions of 0.669 and 0.527, respectively, in analyzing the same data set. Significant advantages to the CFML system included ease of use, relatively low cost, displays of true/false positives and negatives in a user-friendly interface, and the ability to deploy models for further analysis with ease. Significant drawbacks of the CFML model included an inability to view the structure of the trained model, an inability to update the ML model once trained with new examples, and the inability for robust downstream analysis of model performance and error modes.

CONCLUSIONS

This first report describes the baseline performance of CFML in an object detection task using a publicly available surgical video data set as a test bed. Compared with standard, code-based object detection networks, CFML exceeded performance standards. This finding is encouraging for surgeon-scientists seeking to perform object detection tasks to answer clinical questions, perform quality improvement, and develop novel research ideas. The limited interpretability and customization of CFML models remain ongoing challenges. With the further development of code-free platforms, CFML will become increasingly important across biomedical research. Using CFML, surgeons without significant coding experience can perform exploratory ML analyses rapidly and efficiently.

Machine Learning-Based Prediction of Early Recurrence in Glioblastoma Patients: A Glance Towards Precision Medicine

BACKGROUND

Ability to thrive and time-to-recurrence following treatment are important parameters to assess in patients with glioblastoma multiforme (GBM), given its dismal prognosis. Though there is an ongoing debate whether it can be considered an appropriate surrogate endpoint for overall survival in clinical trials, progression-free survival (PFS) is routinely used for clinical decision-making.

OBJECTIVE

To investigate whether machine learning (ML)-based models can reliably stratify newly diagnosed GBM patients into prognostic subclasses on PFS basis, identifying those at higher risk for an early recurrence (≤6 mo).

METHODS

Data were extracted from a multicentric database, according to the following eligibility criteria: histopathologically verified GBM and follow-up >12 mo: 474 patients met our inclusion criteria and were included in the analysis. Relevant demographic, clinical, molecular, and radiological variables were selected by a feature selection algorithm (Boruta) and used to build a ML-based model.

RESULTS

Random forest prediction model, evaluated on an 80:20 split ratio, achieved an AUC of 0.81 (95% CI: 0.77; 0.83) demonstrating high discriminative ability. Optimizing the predictive value derived from the linear and nonlinear combinations of the selected input features, our model outperformed across all performance metrics multivariable logistic regression.

CONCLUSION

A robust ML-based prediction model that identifies patients at high risk for early recurrence was successfully trained and internally validated. Considerable effort remains to integrate these predictions in a patient-centered care context.

Discrepancies in Stroke Distribution and Dataset Origin in Machine Learning for Stroke

BACKGROUND

Machine learning algorithms depend on accurate and representative datasets for training in order to become valuable clinical tools that are widely generalizable to a varied population. We aim to conduct a review of machine learning uses in stroke literature to assess the geographic distribution of datasets and patient cohorts used to train these models and compare them to stroke distribution to evaluate for disparities.

AIMS

582 studies were identified on initial searching of the PubMed database. Of these studies, 106 full texts were assessed after title and abstract screening which resulted in 489 papers excluded. Of these 106 studies, 79 were excluded due to using cohorts from outside the United States or being review articles or editorials. 27 studies were thus included in this analysis.

SUMMARY OF REVIEW

Of the 27 studies included, 7 (25.9%) used patient data from California, 6 (22.2%) were multicenter, 3 (11.1%) were in Massachusetts, 2 (7.4%) each in Illinois, Missouri, and New York, and 1 (3.7%) each from South Carolina, Washington, West Virginia, and Wisconsin. 1 (3.7%) study used data from Utah and Texas. These were qualitatively compared to a CDC study showing the highest distribution of stroke in Mississippi (4.3%) followed by Oklahoma (3.4%), Washington D.C. (3.4%), Louisiana (3.3%), and Alabama (3.2%) while the prevalence in California was 2.6%.

CONCLUSIONS

It is clear that a strong disconnect exists between the datasets and patient cohorts used in training machine learning algorithms in clinical research and the stroke distribution in which clinical tools using these algorithms will be implemented. In order to ensure a lack of bias and increase generalizability and accuracy in future machine learning studies, datasets using a varied patient population that reflects the unequal distribution of stroke risk factors would greatly benefit the usability of these tools and ensure accuracy on a nationwide scale.