Variability in the utility of predictive models in predicting patient-reported outcomes following spine surgery for degenerative conditions: a systematic review

Assessment of the External Validity of Dialogue Support for Predicting Lumbar Spine Surgery Outcomes in a US Cohort

STUDY DESIGN

External validation using prospectively collected data.

OBJECTIVES

To determine the model performance of "Dialogue Support" (DS) in predicting outcomes after lumbar spine surgery.

SUMMARY OF BACKGROUND DATA

To help clinicians discuss risk versus benefit with patients considering lumbar fusion surgery, DS has been made available online. As DS was created using a Swedish sample, there is a need to study how well DS performs in alternative populations.

PATIENTS AND METHODS

Preoperative data from patients enrolled in the Quality Outcomes Database were entered into DS. The probability for each patient to report satisfaction, achieve success (leg pain improvement ≥3), or have no leg pain 12 months after surgery was extracted and compared with their actual 12-month postoperative data. The ability of DS to identify patients in the Quality Outcomes Database who report satisfaction, achieve success, or have no leg pain 12 months after surgery was determined using Receiver operating characteristic curve analysis, goodness-of-fit tests, and calibration plots.

RESULTS

There was a significant improvement in all outcomes in 23,928 cases included in the analysis from baseline to 12 months postoperative. Most (84%) reported satisfaction, 67% achieved success, and 44% were pain-free 12 months postoperative. Receiver operating characteristic analysis showed that DS had a low ability to predict satisfaction [area under the curve (AUC) = 0.606], success (AUC = 0.546), and being pain-free (AUC = 0.578) at 12 months postoperative; poor fit for satisfaction (<0.001) and being pain-free ( P = 0.004), but acceptable fit for success ( P = 0.052). Calibration plots showed underestimation for satisfaction and success, but acceptable estimates for being pain-free.

CONCLUSION

DS is not directly transferable to predict satisfaction and success after lumbar surgery in a US population. This may be due to differences in patient characteristics, weights of the variables included, or the exclusion of unknown variables associated with outcomes. Future studies to better understand and improve the transferability of these models are needed.

Machine learning in clinical diagnosis, prognostication, and management of acute traumatic spinal cord injury (SCI): A systematic review

Background

Machine learning has been applied to improve diagnosis and prognostication of acute traumatic spinal cord injury. We investigate potential for clinical integration of machine learning in this patient population to navigate variability in injury and recovery.

Materials and methods

We performed a systematic review using PRISMA guidelines through PubMed database to identify studies that use machine learning algorithms for clinical application toward improvements in diagnosis, management, and predictive modeling.

Results

Of the 132 records identified, a total of 13 articles met inclusion criteria and were included in final analysis. Of the 13 articles, 5 focused on diagnostic accuracy and 8 were related to prognostication or management of traumatic spinal cord injury. Across studies, 1983 patients with spinal cord injury were evaluated with most classifying as ASIA C or D. Retrospective designs were used in 10 of 13 studies and 3 were prospective. Studies focused on MRI evaluation and segmentation for diagnostic accuracy and prognostication, investigation of mean arterial pressure in acute care and intraoperative settings, prediction of ambulatory and functional ability, chronic complication prevention, and psychological quality of life assessments. Decision tree, random forests (RF), support vector machines (SVM), hierarchical cluster tree analysis (HCTA), artificial neural networks (ANN), convolutional neural networks (CNN) machine learning subtypes were used.

Conclusions

Machine learning represents a platform technology with clinical application in traumatic spinal cord injury diagnosis, prognostication, management, rehabilitation, and risk prevention of chronic complications and mental illness. SVM models showed improved accuracy when compared to other ML subtypes surveyed. Inherent variability across patients with SCI offers unique opportunity for ML and personalized medicine to drive desired outcomes and assess risks in this patient population.

A Smartphone Application Using Artificial Intelligence Is Superior To Subject Self-Reporting When Assessing Stool Form

INTRODUCTION

Stool form assessment relies on subjective patient reports using the Bristol Stool Scale (BSS). In a novel smartphone application (app), trained artificial intelligence (AI) characterizes digital images of users' stool. In this study, we evaluate this AI for accuracy in assessing stool characteristics.

METHODS

Subjects with diarrhea-predominant irritable bowel syndrome image-captured every stool for 2 weeks using the app, which assessed images for 5 visual characteristics (BSS, consistency, fragmentation, edge fuzziness, and volume). In the validation phase, using 2 expert gastroenterologists as a gold standard, sensitivity, specificity, accuracy, and diagnostic odds ratios of subject-reported vs AI-graded BSS scores were compared. In the implementation phase, agreements between AI-graded and subject-reported daily average BSS scores were determined, and subject BSS and AI stool characteristics scores were correlated with diarrhea-predominant irritable bowel syndrome symptom severity scores.

RESULTS

In the validation phase (n = 14), there was good agreement between the 2 experts and AI characterizations for BSS (intraclass correlation coefficients [ICC] = 0.782-0.852), stool consistency (ICC = 0.873-0.890), edge fuzziness (ICC = 0.836-0.839), fragmentation (ICC = 0.837-0.863), and volume (ICC = 0.725-0.851). AI outperformed subjects' self-reports in categorizing daily average BSS scores as constipation, normal, or diarrhea. In the implementation phase (n = 25), the agreement between AI and self-reported BSS scores was moderate (ICC = 0.61). AI stool characterization also correlated better than subject reports with diarrhea severity scores.

DISCUSSION

A novel smartphone application can determine BSS and other visual stool characteristics with high accuracy compared with the 2 expert gastroenterologists. Moreover, trained AI was superior to subject self-reporting of BSS. AI assessments could provide more objective outcome measures for stool characterization in gastroenterology.

The use of electronic PROMs provides same outcomes as paper version in a spine surgery registry. Results from a prospective cohort study

BACKGROUND AND PURPOSE

Patient-Reported Measured Outcomes (PROMs) are essential to gain a full understanding of a patient's condition, and in spine surgery, these questionnaires are of help when tailoring a surgical strategy. Electronic registries allow for a systematic collection and storage of PROMs, making them readily available for clinical and research purposes. This study aimed to investigate the reliability between the electronic and paper form of ODI (Oswestry Disability Index), SF-36 (Short Form Health Survey 36) and COMI-back (Core Outcome Measures Index for the back) questionnaires.

METHODS

A prospective analysis was performed of ODI, SF-36 and COMI-back questionnaires collected in paper and electronic format in two patients' groups: Pre-Operatively (PO) or at follow-up (FU). All patients, in both groups, completed the three questionnaires in paper and electronic form. The correlation between both methods was assessed with the Intraclass Correlation Coefficients (ICC).

RESULTS

The data from 100 non-consecutive, volunteer patients with a mean age of 55.6 ± 15.0 years were analysed. For all of the three PROMs, the reliability between paper and electronic questionnaires results was excellent (ICC: ODI = 0.96; COMI = 0.98; SF36-MCS = 0.98; SF36-PCS = 0.98. For all p < 0.001).

CONCLUSIONS

This study proved an excellent reliability between the electronic and paper versions of ODI, SF-36 and COMI-back questionnaires collected using a spine registry. This validation paves the way for stronger widespread use of electronic PROMs. They offer numerous advantages in terms of accessibility, storage, and data analysis compared to paper questionnaires.

Machine learning-based preoperative predictive analytics for lumbar spinal stenosis

OBJECTIVEPatient-reported outcome measures (PROMs) following decompression surgery for lumbar spinal stenosis (LSS) demonstrate considerable heterogeneity. Individualized prediction tools can provide valuable insights for shared decision-making. The authors aim to evaluate the feasibility of predicting short- and long-term PROMs, reoperations, and perioperative parameters by machine learning (ML) methods.METHODSData were derived from a prospective registry. All patients had undergone single- or multilevel mini-open facet-sparing decompression for LSS. The prediction models were trained using various ML-based algorithms to predict the endpoints of interest. Models were selected by area under the receiver operating characteristic curve (AUC). The endpoints were dichotomized by minimum clinically important difference (MCID) and included 6-week and 12-month numeric rating scales for back pain (NRS-BP) and leg pain (NRS-LP) severity and the Oswestry Disability Index (ODI), as well as prolonged surgery (> 45 minutes), extended length of hospital stay (> 28 hours), and reoperations.RESULTSA total of 635 patients were included. The average age was 62 ± 10 years, and 333 patients (52%) were male. At 6 weeks, MCID was seen in 63%, 76%, and 61% of patients for ODI, NRS-LP, and NRS-BP, respectively. At internal validation, the models predicted MCID in these variables with accuracies of 69%, 76%, and 85%, and with AUCs of 0.75, 0.79, and 0.92. At 12 months, 66%, 63%, and 51% of patients reported MCID; the observed accuracies were 62%, 74%, and 66%, with AUCs of 0.68, 0.72, and 0.79. Reoperations occurred in 60 patients (9.5%), of which 27 (4.3%) occurred at the index level. Overall and index-level reoperations were predicted with 69% and 63% accuracy, respectively, and with AUCs of 0.66 and 0.61. In 15%, a length of surgery greater than 45 minutes was observed and predicted with 78% accuracy and AUC of 0.54. Only 15% of patients were admitted to the hospital for longer than 28 hours. The developed ML-based model enabled prediction of extended hospital stay with an accuracy of 77% and AUC of 0.58.CONCLUSIONSPreoperative prediction of a range of clinically relevant endpoints in decompression surgery for LSS using ML is feasible, and may enable enhanced informed patient consent and personalized shared decision-making. Access to individualized preoperative predictive analytics for outcome and treatment risks may represent a further step in the evolution of surgical care for patients with LSS.

Evaluation of Predictive Models for Complications following Spinal Surgery

BACKGROUND

 Complications rates vary across spinal surgery procedures and are difficult to predict due to heterogeneity in patient characteristics, surgical methods, and hospital volume. Incorporation of predictive models for complications may guide surgeon decision making and improve outcomes.

METHODS

 We evaluate current independently validated predictive models for complications in spinal surgery with respect to study design and model generation, accuracy, reliability, and utility. We conducted our search using Preferred Reporting Items for Systematic Review and Meta-analysis guidelines and the Participants, Intervention, Comparison, Outcomes, Study Design model through the PubMed and Ovid Medline databases.

RESULTS

 A total of 18 articles met inclusion criteria including 30 validated predictive models of complications after adult spinal surgery. National registry databases were used in 12 studies. Validation cohorts were used in seven studies for verification; three studies used other methods including random sample bootstrapping techniques or cross-validation. Reported area under the curve (AUC) values ranged from 0.37 to 1.0. Studies described treatment for deformity, degenerative conditions, inclusive spinal surgery (neoplasm, trauma, infection, deformity, degenerative), and miscellaneous (disk herniation, spinal epidural abscess). The most commonly cited risk factors for complications included in predictive models included age, body mass index, diabetes, sex, and smoking. Those models in the deformity subset that included radiographic and anatomical grading features reported higher AUC values than those that included patient demographics or medical comorbidities alone.

CONCLUSIONS

 We identified a cohort of 30 validated predictive models of complications following spinal surgery for degenerative conditions, deformity, infection, and trauma. Accurate evidence-based predictive models may enhance shared decision making, improve rehabilitation, reduce adverse events, and inform best practices.

SpineCloud: image analytics for predictive modeling of spine surgery outcomes

Purpose: Data-intensive modeling could provide insight on the broad variability in outcomes in spine surgery. Previous studies were limited to analysis of demographic and clinical characteristics. We report an analytic framework called "SpineCloud" that incorporates quantitative features extracted from perioperative images to predict spine surgery outcome. Approach: A retrospective study was conducted in which patient demographics, imaging, and outcome data were collected. Image features were automatically computed from perioperative CT. Postoperative 3- and 12-month functional and pain outcomes were analyzed in terms of improvement relative to the preoperative state. A boosted decision tree classifier was trained to predict outcome using demographic and image features as predictor variables. Predictions were computed based on SpineCloud and conventional demographic models, and features associated with poor outcome were identified from weighting terms evident in the boosted tree. Results: Neither approach was predictive of 3- or 12-month outcomes based on preoperative data alone in the current, preliminary study. However, SpineCloud predictions incorporating image features obtained during and immediately following surgery (i.e., intraoperative and immediate postoperative images) exhibited significant improvement in area under the receiver operating characteristic (AUC): AUC = 0.72 ( CI 95 = 0.59 to 0.83) at 3 months and AUC = 0.69 ( CI 95 = 0.55 to 0.82) at 12 months. Conclusions: Predictive modeling of lumbar spine surgery outcomes was improved by incorporation of image-based features compared to analysis based on conventional demographic data. The SpineCloud framework could improve understanding of factors underlying outcome variability and warrants further investigation and validation in a larger patient cohort.