Development and external validation of predictive algorithms for six-week mortality in spinal metastasis using 4,304 patients from five institutions

Machine Learning in Spine Oncology: A Narrative Review

STUDY DESIGN

Narrative Review.

OBJECTIVE

Machine learning (ML) is one of the latest advancements in artificial intelligence used in medicine and surgery with the potential to significantly impact the way physicians diagnose, prognose, and treat spine tumors. In the realm of spine oncology, ML is utilized to analyze and interpret medical imaging and classify tumors with incredible accuracy. The authors present a narrative review that specifically addresses the use of machine learning in spine oncology.

METHODS

This study was conducted in accordance with the Preferred Reporting Items of Systematic Reviews and Meta-Analysis (PRISMA) methodology. A systematic review of the literature in the PubMed, EMBASE, Web of Science, Scopus, and Cochrane Library databases since inception was performed to present all clinical studies with the search terms '[[Machine Learning] OR [Artificial Intelligence]] AND [[Spine Oncology] OR [Spine Cancer]]'. Data included studies that were extracted and included algorithms, training and test size, outcomes reported. Studies were separated based on the type of tumor investigated using the machine learning algorithms into primary, metastatic, both, and intradural. A minimum of 2 independent reviewers conducted the study appraisal, data abstraction, and quality assessments of the studies.

RESULTS

Forty-five studies met inclusion criteria out of 480 references screened from the initial search results. Studies were grouped by metastatic, primary, and intradural tumors. The majority of ML studies relevant to spine oncology focused on utilizing a mixture of clinical and imaging features to risk stratify mortality and frailty. Overall, these studies showed that ML is a helpful tool in tumor detection, differentiation, segmentation, predicting survival, predicting readmission rates of patients with either primary, metastatic, or intradural spine tumors.

CONCLUSION

Specialized neural networks and deep learning algorithms have shown to be highly effective at predicting malignant probability and aid in diagnosis. ML algorithms can predict the risk of tumor recurrence or progression based on imaging and clinical features. Additionally, ML can optimize treatment planning, such as predicting radiotherapy dose distribution to the tumor and surrounding normal tissue or in surgical resection planning. It has the potential to significantly enhance the accuracy and efficiency of health care delivery, leading to improved patient outcomes.

Predictive Modeling for Spinal Metastatic Disease

Spinal metastasis is exceedingly common in patients with cancer and its prevalence is expected to increase. Surgical management of symptomatic spinal metastasis is indicated for pain relief, preservation or restoration of neurologic function, and mechanical stability. The overall prognosis is a major driver of treatment decisions; however, clinicians' ability to accurately predict survival is limited. In this narrative review, we first discuss the NOMS decision framework used to guide decision making in the treatment of patients with spinal metastasis. Given that decision making hinges on prognosis, multiple scoring systems have been developed over the last three decades to predict survival in patients with spinal metastasis; these systems have largely been developed using expert opinions or regression modeling. Although these tools have provided significant advances in our ability to predict prognosis, their utility is limited by the relative lack of patient-specific survival probability. Machine learning models have been developed in recent years to close this gap. Employing a greater number of features compared to models developed with conventional statistics, machine learning algorithms have been reported to predict 30-day, 6-week, 90-day, and 1-year mortality in spinal metastatic disease with excellent discrimination. These models are well calibrated and have been externally validated with domestic and international independent cohorts. Despite hypothesized and realized limitations, the role of machine learning methodology in predicting outcomes in spinal metastatic disease is likely to grow.

Are Current Survival Prediction Tools Useful When Treating Subsequent Skeletal-related Events From Bone Metastases?

BACKGROUND

Bone metastasis in advanced cancer is challenging because of pain, functional issues, and reduced life expectancy. Treatment planning is complex, with consideration of factors such as location, symptoms, and prognosis. Prognostic models help guide treatment choices, with Skeletal Oncology Research Group machine-learning algorithms (SORG-MLAs) showing promise in predicting survival for initial spinal metastases and extremity metastases treated with surgery or radiotherapy. Improved therapies extend patient lifespans, increasing the risk of subsequent skeletal-related events (SREs). Patients experiencing subsequent SREs often suffer from disease progression, indicating a deteriorating condition. For these patients, a thorough evaluation, including accurate survival prediction, is essential to determine the most appropriate treatment and avoid aggressive surgical treatment for patients with a poor survival likelihood. Patients experiencing subsequent SREs often suffer from disease progression, indicating a deteriorating condition. However, some variables in the SORG prediction model, such as tumor histology, visceral metastasis, and previous systemic therapies, might remain consistent between initial and subsequent SREs. Given the prognostic difference between patients with and without a subsequent SRE, the efficacy of established prognostic models-originally designed for individuals with an initial SRE-in addressing a subsequent SRE remains uncertain. Therefore, it is crucial to verify the model's utility for subsequent SREs.

QUESTION/PURPOSE

We aimed to evaluate the reliability of the SORG-MLAs for survival prediction in patients undergoing surgery or radiotherapy for a subsequent SRE for whom both the initial and subsequent SREs occurred in the spine or extremities.

METHODS

We retrospectively included 738 patients who were 20 years or older who received surgery or radiotherapy for initial and subsequent SREs at a tertiary referral center and local hospital in Taiwan between 2010 and 2019. We excluded 74 patients whose initial SRE was in the spine and in whom the subsequent SRE occurred in the extremities and 37 patients whose initial SRE was in the extremities and the subsequent SRE was in the spine. The rationale was that different SORG-MLAs were exclusively designed for patients who had an initial spine metastasis and those who had an initial extremity metastasis, irrespective of whether they experienced metastatic events in other areas (for example, a patient experiencing an extremity SRE before his or her spinal SRE would also be regarded as a candidate for an initial spinal SRE). Because these patients were already validated in previous studies, we excluded them in case we overestimated our result. Five patients with malignant primary bone tumors and 38 patients in whom the metastasis's origin could not be identified were excluded, leaving 584 patients for analysis. The 584 included patients were categorized into two subgroups based on the location of initial and subsequent SREs: the spine group (68% [399]) and extremity group (32% [185]). No patients were lost to follow-up. Patient data at the time they presented with a subsequent SRE were collected, and survival predictions at this timepoint were calculated using the SORG-MLAs. Multiple imputation with the Missforest technique was conducted five times to impute the missing proportions of each predictor. The effectiveness of SORG-MLAs was gauged through several statistical measures, including discrimination (measured by the area under the receiver operating characteristic curve [AUC]), calibration, overall performance (Brier score), and decision curve analysis. Discrimination refers to the model's ability to differentiate between those with the event and those without the event. An AUC ranges from 0.5 to 1.0, with 0.5 indicating the worst discrimination and 1.0 indicating perfect discrimination. An AUC of 0.7 is considered clinically acceptable discrimination. Calibration is the comparison between the frequency of observed events and the predicted probabilities. In an ideal calibration, the observed and predicted survival rates should be congruent. The logarithm of observed-to-expected survival ratio [log(O:E)] offers insight into the model's overall calibration by considering the total number of observed (O) and expected (E) events. The Brier score measures the mean squared difference between the predicted probability of possible outcomes for each individual and the observed outcomes, ranging from 0 to 1, with 0 indicating perfect overall performance and 1 indicating the worst performance. Moreover, the prevalence of the outcome should be considered, so a null-model Brier score was also calculated by assigning a probability equal to the prevalence of the outcome (in this case, the actual survival rate) to each patient. The benefit of the prediction model is determined by comparing its Brier score with that of the null model. If a prediction model's Brier score is lower than the null model's Brier score, the prediction model is deemed as having good performance. A decision curve analysis was performed for models to evaluate the "net benefit," which weighs the true positive rate over the false positive rate against the "threshold probabilities," the ratio of risk over benefit after an intervention was derived based on a comprehensive clinical evaluation and a well-discussed shared-decision process. A good predictive model should yield a higher net benefit than default strategies (treating all patients and treating no patients) across a range of threshold probabilities.

RESULTS

For the spine group, the algorithms displayed acceptable AUC results (median AUCs of 0.69 to 0.72) for 42-day, 90-day, and 1-year survival predictions after treatment for a subsequent SRE. In contrast, the extremity group showed median AUCs ranging from 0.65 to 0.73 for the corresponding survival periods. All Brier scores were lower than those of their null model, indicating the SORG-MLAs' good overall performances for both cohorts. The SORG-MLAs yielded a net benefit for both cohorts; however, they overestimated 1-year survival probabilities in patients with a subsequent SRE in the spine, with a median log(O:E) of -0.60 (95% confidence interval -0.77 to -0.42).

CONCLUSION

The SORG-MLAs maintain satisfactory discriminatory capacity and offer considerable net benefits through decision curve analysis, indicating their continued viability as prediction tools in this clinical context. However, the algorithms overestimate 1-year survival rates for patients with a subsequent SRE of the spine, warranting consideration of specific patient groups. Clinicians and surgeons should exercise caution when using the SORG-MLAs for survival prediction in these patients and remain aware of potential mispredictions when tailoring treatment plans, with a preference for less invasive treatments. Ultimately, this study emphasizes the importance of enhancing prognostic algorithms and developing innovative tools for patients with subsequent SREs as the life expectancy in patients with bone metastases continues to improve and healthcare providers will encounter these patients more often in daily practice.

LEVEL OF EVIDENCE

Level III, prognostic study.

Machine Learning in Neurosurgery: Toward Complex Inputs, Actionable Predictions, and Generalizable Translations

Machine learning can predict neurosurgical diagnosis and outcomes, power imaging analysis, and perform robotic navigation and tumor labeling. State-of-the-art models can reconstruct and generate images, predict surgical events from video, and assist in intraoperative decision-making. In this review, we will detail the neurosurgical applications of machine learning, ranging from simple to advanced models, and their potential to transform patient care. As machine learning techniques, outputs, and methods become increasingly complex, their performance is often more impactful yet increasingly difficult to evaluate. We aim to introduce these advancements to the neurosurgical audience while suggesting major potential roadblocks to their safe and effective translation. Unlike the previous generation of machine learning in neurosurgery, the safe translation of recent advancements will be contingent on neurosurgeons' involvement in model development and validation.

Limitations in Evaluating Machine Learning Models for Imbalanced Binary Outcome Classification in Spine Surgery: A Systematic Review

Clinical prediction models for spine surgery applications are on the rise, with an increasing reliance on machine learning (ML) and deep learning (DL). Many of the predicted outcomes are uncommon; therefore, to ensure the models' effectiveness in clinical practice it is crucial to properly evaluate them. This systematic review aims to identify and evaluate current research-based ML and DL models applied for spine surgery, specifically those predicting binary outcomes with a focus on their evaluation metrics. Overall, 60 papers were included, and the findings were reported according to the PRISMA guidelines. A total of 13 papers focused on lengths of stay (LOS), 12 on readmissions, 12 on non-home discharge, 6 on mortality, and 5 on reoperations. The target outcomes exhibited data imbalances ranging from 0.44% to 42.4%. A total of 59 papers reported the model's area under the receiver operating characteristic (AUROC), 28 mentioned accuracies, 33 provided sensitivity, 29 discussed specificity, 28 addressed positive predictive value (PPV), 24 included the negative predictive value (NPV), 25 indicated the Brier score with 10 providing a null model Brier, and 8 detailed the F1 score. Additionally, data visualization varied among the included papers. This review discusses the use of appropriate evaluation schemes in ML and identifies several common errors and potential bias sources in the literature. Embracing these recommendations as the field advances may facilitate the integration of reliable and effective ML models in clinical settings.

Machine learning-based algorithms to predict severe psychological distress among cancer patients with spinal metastatic disease

BACKGROUND CONTEXT

Metastatic spinal disease is an advanced stage of cancer patients and often suffer from terrible psychological health status; however, the ability to estimate the risk probability of this adverse outcome using current available data is very limited.

PURPOSE

The goal of this study was to propose a precise model based on machine learning techniques to predict psychological status among cancer patients with spinal metastatic disease.

STUDY DESIGN/SETTING

A prospective cohort study.

PATIENT SAMPLE

A total of 1043 cancer patients with spinal metastatic disease were included.

OUTCOME MEASURES

The main outcome was severe psychological distress.

METHODS

The total of patients was randomly divided into a training dataset and a testing dataset on a ratio of 9:1. Patients' demographics, lifestyle choices, cancer-related features, clinical manifestations, and treatments were collected as potential model predictors in the study. Five machine learning algorithms, including XGBoosting machine, random forest, gradient boosting machine, support vector machine, and ensemble prediction model, as well as a logistic regression model were employed to train and optimize models in the training set, and their predictive performance was assessed in the testing set.

RESULTS

Up to 21.48% of all patients who were recruited had severe psychological distress. Elderly patients (p<0.001), female (p =0.045), current smoking (p=0.002) or drinking (p=0.003), a lower level of education (p<0.001), a stronger spiritual desire (p<0.001), visceral metastasis (p=0.005), and a higher Eastern Cooperative Oncology Group (ECOG) score (p<0.001) were significantly associated with worse psychological health. With an area under the curve (AUC) of 0.865 (95% CI: 0.788-0.941) and an accuracy of up to 0.843, the gradient boosting machine algorithm performed best in the prediction of the outcome, followed by the XGBooting machine algorithm (AUC: 0.851, 95% CI: 0.768-0.934; Accuracy: 0.826) and ensemble prediction (AUC: 0.851, 95% CI: 0.770-0.932; Accuracy: 0.809) in the testing set. In contrast, the AUC of the logistic regression model was only 0.836 (95% CI: 0.756-0.916; Accuracy: 0.783).

CONCLUSIONS

Machine learning models have greater predictive power and can offer useful tools to identify individuals with spinal metastatic disease who are experiencing severe psychological distress.

A Machine Learning Algorithm for Predicting 6-Week Survival in Spinal Metastasis: An External Validation Study Using 2,768 Taiwanese Patients

INTRODUCTION

There are predictive algorithms for predicting 3-month and 1-year survival in patients with spinal metastasis. However, advance in surgical technique, immunotherapy, and advanced radiation therapy has enabled shortening of postoperative recovery, which returns dividends to the overall quality-adjusted life-year. As such, the Skeletal Oncology Research Group machine learning algorithm (SORG-MLA) was proposed to predict 6-week survival in patients with spinal metastasis, whereas its utility for patients treated with nonsurgical treatment was untested externally. This study aims to validate the survival prediction of the 6-week SORG-MLA for patients with spinal metastasis and provide the measurement of model consistency (MC).

METHODS

Discrimination using area under the receiver operating characteristic curve, calibration, Brier score, and decision curve analysis were conducted to assess the model's performance in the Taiwanese-based cohort. MC was also applied to detect the proportion of paradoxical predictions among 6-week, 3-month, and 1-year survival predictions. The long-term prognosis should not be better than the shorter-term prognosis in that of an individual.

RESULTS

The 6-week survival rate was 84.2%. The SORG-MLA retained good discrimination with an area under the receiver operating characteristic curve of 0.78 (95% confidence interval, 0.75 to 0.80) and good prediction accuracy with a Brier score of 0.11 (null model Brier score 0.13). There is an underestimation of the 6-week survival rate when the predicted survival rate is less than 50%. Decision curve analysis showed that the model was suitable for use over all threshold probabilities. MC showed suboptimal consistency between 6-week and 90-day survival prediction (78%).

CONCLUSIONS

The results of this study supported the utility of the algorithm. The online tool ( https://sorg-apps.shinyapps.io/spinemetssurvival/ ) can be used by both clinicians and patients in informative decision-making discussion before management of spinal metastasis.

More help than harm: surgery for metastatic spinal cord compression is associated with more favorable overall survival within a propensity score analysis

PURPOSE

Indication for surgical decompression in metastatic spinal cord compression (MSCC) is often based on prognostic scores such as the modified Bauer score (mBs), with favorable prognosis suggestive of surgery and poor prognosis of non-surgical management. This study aimed to clarify if (1) surgery may directly affect overall survival (OS) aside from short-term neurologic outcome, (2) explore whether selected patient subgroups with poor mBs might still benefit from surgery, and (3) gauge putative adverse effects of surgery on short-term oncologic outcomes.

METHODS

Single-center propensity score analyses with inverse-probability-of-treatment-weights (IPTW) of OS and short-term neurologic outcomes in MSCC patients treated with or without surgery between 2007 and 2020.

RESULTS

Among 398 patients with MSCC, 194 (49%) underwent surgery. During a median follow-up of 5.8 years, 355 patients (89%) died. MBs was the most important predictor for spine surgery (p < 0.0001) and the strongest predictor of favorable OS (p < 0.0001). Surgery was associated with improved OS after accounting for selection bias with the IPTW method (p = 0.021) and emerged as the strongest determinant of short-term neurological improvement (p < 0.0001). Exploratory analyses delineated a subgroup of patients with an mBs of 1 point who still benefited from surgery, and surgery did not result in a higher risk of short-term oncologic disease progression.

CONCLUSION

This propensity score analysis corroborates the concept that spine surgery for MSCC associates with more favorable neurological and OS outcomes. Selected patients with poor prognosis might also benefit from surgery, suggesting that even those with low mBs may be considered for this intervention.