FDA-approved machine learning algorithms in neuroradiology: A systematic review of the current evidence for approval

Over the past decade, machine learning (ML) and artificial intelligence (AI) have become increasingly prevalent in the medical field. In the United States, the Food and Drug Administration (FDA) is responsible for regulating AI algorithms as "medical devices" to ensure patient safety. However, recent work has shown that the FDA approval process may be deficient. In this study, we evaluate the evidence supporting FDA-approved neuroalgorithms, the subset of machine learning algorithms with applications in the central nervous system (CNS), through a systematic review of the primary literature. Articles covering the 53 FDA-approved algorithms with applications in the CNS published in PubMed, EMBASE, Google Scholar and Scopus between database inception and January 25, 2022 were queried. Initial searches identified 1505 studies, of which 92 articles met the criteria for extraction and inclusion. Studies were identified for 26 of the 53 neuroalgorithms, of which 10 algorithms had only a single peer-reviewed publication. Performance metrics were available for 15 algorithms, external validation studies were available for 24 algorithms, and studies exploring the use of algorithms in clinical practice were available for 7 algorithms. Papers studying the clinical utility of these algorithms focused on three domains: workflow efficiency, cost savings, and clinical outcomes. Our analysis suggests that there is a meaningful gap between the FDA approval of machine learning algorithms and their clinical utilization. There appears to be room for process improvement by implementation of the following recommendations: the provision of compelling evidence that algorithms perform as intended, mandating minimum sample sizes, reporting of a predefined set of performance metrics for all algorithms and clinical application of algorithms prior to widespread use. This work will serve as a baseline for future research into the ideal regulatory framework for AI applications worldwide.

Cost effectiveness of implanting a prosthesis after anterior cervical discectomy for radiculopathy: results of the NECK randomized controlled trial

BACKGROUND CONTEXT

In the treatment of cervical radiculopathy due to a herniated disc, potential surgical treatments include: anterior cervical discectomy (ACD), ACD and fusion using a cage (ACDF), and anterior cervical disc arthroplasty (ACDA). Previous publications yielded comparable clinical and radiological outcome data for the various implants, but research on their comparative costutility has been inconclusive.

PURPOSE

To evaluate the cost utility of ACD, ACDF, and ACDA.

STUDY DESIGN

Cost-utility analysis.

PATIENT SAMPLE

About 109 patients with cervical radiculopathy randomized to undergo ACD, ACDF, or ACDA as part of the NEtherlands Cervical Kinetics trial.

OUTCOME MEASURES

Quality-adjusted life-years (QALYs) estimated from patient-reported utilities using the EuroQol-5D questionnaire and EuroQol Visual Analogue Scale (EQ VAS), measured at baseline, 2, 4, 8, 12, 26, 52, and 104 weeks postprocedure. Societal costs including admissions to hospital (related and otherwise), GP visits, specialist visits, physical therapy, medications, home care, aids, informal care, productivity losses, and out of pocket condition-related expenses.

METHODS

The cost utility of the competing strategies over 1 and 2 years was assessed following a net benefit (NB) approach, whereby the intervention with the highest NB among competing strategies is preferred. Cost effectiveness acceptability curves were produced to reflect the probability of each strategy being the most cost effective across various willingness-to-pay (WTP) thresholds. Five sensitivity analyses were conducted to assess the robustness of results.

RESULTS

ACDF was more likely to be the most cost-effective strategy at WTP thresholds of €20,000 to 50,000/QALY in all but one of the analyses. The mean QALYs during the first year were 0.750, 0.817, and 0.807 for ACD, ACDF, and ACDA, respectively, with no significant differences between groups. Total healthcare costs over the first year were significantly higher for ACDA, largely due to the higher surgery and implant costs. The total societal costs of the three strategies were €12,173 for ACD, €11,195 for ACDF, and €13,746 for ACDA, with no significant differences between groups.

CONCLUSION

Our findings demonstrate that ACDF is likely to be more cost-effective than ACDA or ACD at most WTP thresholds, and this conclusion is robust to most sensitivity analyses conducted. It is demonstrated that the difference in costs is mainly caused by the initial surgical costs and that there are only minimal differences in other costs during follow-up. Since clinical data are comparable between the groups, it is to the judgment of the patient and surgeon which intervention is applied.

Long-term results of the NECK trial-implanting a disc prosthesis after cervical anterior discectomy cannot prevent adjacent segment disease: five-year clinical follow-up of a double-blinded randomised controlled trial

BACKGROUND CONTEXT

Motion preserving anterior cervical disc arthroplasty (ACDA) in patients with cervical radiculopathy was introduced to prevent symptomatic adjacent segment disease as compared to anterior cervical discectomy and fusion (ACDF).

PURPOSE

To evaluate the long-term outcome in patients with cervical radiculopathy due to a herniated disc undergoing ACDA, ACDF or ACD (no cage, no plate) in terms of clinical outcome measured by the Neck Disability Index (NDI). Likewise, clinically relevant adjacent segment disease is assessed as a long-term result.

STUDY DESIGN

Double-blinded randomized controlled trial.

PATIENT SAMPLE

A total of 109 patients with one level herniated disc were randomized to one of the following treatments: ACDA, ACDF with intervertebral cage, ACD without cage.

OUTCOME MEASURES

Clinical outcome was measured by patients' self-reported NDI, Visual Analogue Scale (VAS) neck pain, VAS arm pain, SF36, EQ-5D, perceived recovery and reoperation rate. Radiological outcome was assessed by radiographic cervical curvature and adjacent segment degeneration (ASD) parameters at baseline and up until five years after surgery.

METHODS

To account for the correlation between repeated measurements of the same individual Generalized Estimated Equations (GEE) were used to calculate treatment effects, expressed in difference in marginal mean values for NDI per treatment group.

RESULTS

Clinical outcome parameters were comparable in the ACDA and ACDF group, but significantly worse in the ACD group, though not reaching clinical relevance. Annual reoperation rate was 3.6% in the first two years after surgery, declined to 1.9% in the years thereafter. The number of reoperations for ASD was not lower in the ACDA group, while the number of reoperations at the index level was higher after ACD, when compared to ACDF and ACDA.

CONCLUSIONS

A persisting absence of clinical superiority was demonstrated for the cervical disc prosthesis five years after surgery. Specifically, clinically relevant adjacent level disease was not prevented by implanting a prosthesis. Single level ACD without implanting an intervertebral device provided worse clinical outcome, which was hypothesized to be caused by delayed fusion. This stresses the need for focusing on timely fusion in future research.

The Clinical Relevance of the Cervical Disc Prosthesis: Combining Clinical Results of Two RCTs

STUDY DESIGN

Retrospective analysis was performed on data from 251 patients that were included in two randomized, double-blinded clinical trials comparing clinical results of anterior cervical discectomy and arthroplasty (ACDA) to anterior cervical discectomy and fusion (ACDF), and anterior cervical discectomy (ACD), for single-level disc herniation.

OBJECTIVE

This study aimed to investigate whether the ACDA procedure offers superior clinical results 2 years after surgery, to either ACDF or ACD without instrumentation, in the entire group of patients or in a particular subgroup of patients.

SUMMARY OF BACKGROUND DATA

The cervical disc prosthesis was introduced to provide superior clinical outcomes after ACD.

METHODS

Neck Disability Index (NDI), and subscales of the 36-item short-form health survey (SF-36) and McGill pain score were collected at baseline, 1 year and 2 years after surgery. Reoperations and complications were also evaluated. A preliminary subgroup analysis was performed for age, disc height, body mass index (BMI), smoking, and sex.

RESULTS

The NDI decreased comparably in all treatment arms to circa 50% of the baseline value and marginal mean NDI differences varied from 0.4 to 1.1 on a 100 point NDI scale, with confidence intervals never exceeding the 20-point minimal clinical important difference (MCID). Secondary outcome parameters showed comparable results. Preliminary subgroup analysis could not demonstrate clinically relevant differences in NDI between treatments after 2 years.

CONCLUSION

After combining data from two Randomized Controlled Trials it can be concluded that there is no clinical benefit for ACDA, when compared with ACDF or ACD 2 years after surgery. Preliminary subgroup analysis indicated outcomes were similar between treatment groups, and that no subgroup could be appointed that benefited more from either ACD, ACDF, or ACDA.Level of Evidence: 1.

The impact of mental health on outcome after anterior cervical discectomy: cohort study assessing the influence of mental health using predictive modelling

BACKGROUND

Depression and anxiety are common mental disorders among patients with chronic pain. It is hypothesised that patients suffering from these disorders benefit less from cervical spine surgery than mentally healthy patients. Therefore, this study aimed to quantify the effect of mental health status on functional outcome after anterior cervical discectomy in a post hoc analysis on RCT data.

METHODS

One hundred eight patients from the NECK trial, with radiculopathy due to a one-level herniated disc, underwent anterior cervical discectomy and were included into this analysis. Functional outcome was quantified using the Neck Disability Index (NDI), and mental health status was measured using the Hospital Anxiety and Depression Score (HADS) questionnaire. NDI differences were assessed using generalised estimated equations (GEE), crude means, a predictive linear mixed model (LMM) using baseline scores and over time with an explanatory LMM.

RESULTS

At baseline, 24% and 32% of patients were respectively depressed and anxious and had statistically significant and clinically relevant higher NDI scores during follow-up. However, in those patients in which the HADS returned to normal during follow-up, NDI values decreased comparably to the non-depression or non-anxiety cases. Those patients that demonstrated persisting high HADS values had convincingly worse NDI scores. A predictive LMM showed that combining baseline NDI and HADS scores was highly predictive of NDI during follow-up. The R shiny application enabled the effective, visual communication of results from the predictive LMM.

CONCLUSION

This study shows that mental health status and disability are strongly associated and provides insight into the size of the effect, as well as a way to use this relation to improve preoperative patient counselling. These findings give rise to the suggestion that incorporating mental health screening in the preoperative assessment of patients could help to adequately manage patients' expectations for functional recovery.

TRIAL REGISTRATION

Dutch Trial Register Number: NTR1289.

Deep Learning for Adjacent Segment Disease at Preoperative MRI for Cervical Radiculopathy

Background Patients who undergo surgery for cervical radiculopathy are at risk for developing adjacent segment disease (ASD). Identifying patients who will develop ASD remains challenging for clinicians. Purpose To develop and validate a deep learning algorithm capable of predicting ASD by using only preoperative cervical MRI in patients undergoing single-level anterior cervical diskectomy and fusion (ACDF). Materials and Methods In this Health Insurance Portability and Accountability Act-compliant study, retrospective chart review was performed for 1244 patients undergoing single-level ACDF in two tertiary care centers. After application of inclusion and exclusion criteria, 344 patients were included, of whom 60% (n = 208) were used for training and 40% for validation (n = 43) and testing (n = 93). A deep learning-based prediction model with 48 convolutional layers was designed and trained by using preoperative T2-sagittal cervical MRI. To validate model performance, a neuroradiologist and neurosurgeon independently provided ASD predictions for the test set. Validation metrics included accuracy, areas under the curve, and F1 scores. The difference in proportion of wrongful predictions between the model and clinician was statistically tested by using the McNemar test. Results A total of 344 patients (median age, 48 years; interquartile range, 41-58 years; 182 women) were evaluated. The model predicted ASD on the 93 test images with an accuracy of 88 of 93 (95%; 95% CI: 90, 99), sensitivity of 12 of 15 (80%; 95% CI: 60, 100), and specificity of 76 of 78 (97%; 95% CI: 94, 100). The neuroradiologist and neurosurgeon provided predictions with lower accuracy (54 of 93; 58%; 95% CI: 48, 68), sensitivity (nine of 15; 60%; 95% CI: 35, 85), and specificity (45 of 78; 58%; 95% CI: 56, 77) compared with the algorithm. The McNemar test on the contingency table demonstrated that the proportion of wrongful predictions was significantly lower by the model (test statistic, 2.000; P < .001). Conclusion A deep learning algorithm that used only preoperative cervical T2-weighted MRI outperformed clinical experts at predicting adjacent segment disease in patients undergoing surgery for cervical radiculopathy. © RSNA, 2021 An earlier incorrect version appeared online. This article was corrected on September 22, 2021.

Single- versus Dual-Attending Surgeon Approach for Spine Deformity: A Systematic Review and Meta-Analysis

BACKGROUND

Surgical management of spine deformity is associated with significant morbidity. Recent literature has inconsistently demonstrated better outcomes after utilizing 2 attending surgeons for spine deformity.

OBJECTIVE

To conduct a systematic review and meta-analysis on studies reporting outcomes following single- vs dual-attending surgeons for spine deformity.

METHODS

MEDLINE, Embase, Web of science, and Cochrane databases were last searched on July 16, 2020. A total of 1013 records were identified excluding duplicates. After screening, 10 studies (4 cohort, 6 case series) were included in the meta-analysis. Random-effect models were used to pool the effect estimates by study design. When feasible, further subgroup analysis by deformity type was conducted.

RESULTS

A total of 953 patients were analyzed. Pooled results from propensity score-matched cohort studies revealed that the single-surgeon approach was unfavorably associated with a nonstatistically significant higher blood loss (mean difference = 421.0 mL; 95% CI: -28.2, 870.2), a statistically significant higher operative time (mean difference = 94.3 min; 95% CI: 54.9, 133), length of stay (mean difference = 0.84 d; 95% CI: 0.46, 1.22), and an increased risk of complications (Mantel-Haenszel risk ratio = 2.93; 95% CI: 1.12, 7.66). Data from pooled case series demonstrated similar results for all outcomes. Moreover, these results did not differ significantly between deformity types (adolescent idiopathic scoliosis and adult spinal deformity).

CONCLUSION

Dual-attending surgeon approach appeared to be associated with reduced operative time, shorter hospital stays, and reduced risk of complications. These findings may potentially improve outcomes in surgical treatment of spine deformity.

Criteria for success after surgery for cervical radiculopathy-estimates for a substantial amount of improvement in core outcome measures

BACKGROUND CONTEXT

Defining clinically meaningful success criteria from patient-reported outcome measures (PROMs) is crucial for clinical audits, research and decision-making.

PURPOSE

We aimed to define criteria for a successful outcome 3 and 12 months after surgery for cervical degenerative radiculopathy on recommended PROMs.

STUDY DESIGN

Prospective cohort study with 12 months follow-up.

PATIENT SAMPLE

Patients operated at one or two levels for cervical radiculopathy included in the Norwegian Registry for Spine Surgery (NORspine) from 2011 to 2016.

OUTCOME MEASURES

Neck disability index (NDI), Numeric Rating Scale for neck pain (NRS-NP) and arm pain (NRS-AP), health-related quality-of-life EuroQol 3L (EQ-5D), general health status (EQ-VAS).

METHODS

We included 2,868 consecutive cervical degenerative radiculopathy patients operated for cervical radiculopathy in one or two levels and included in the Norwegian Registry for Spine Surgery (NORspine). External criterion to determine accuracy and optimal cut-off values for success in the PROMs was the global perceived effect scale. Success was defined as "much better" or "completely recovered." Cut-off values were assessed by analyzing the area under the receiver operating curves for follow-up scores, mean change scores, and percentage change scores.

RESULTS

All PROMs showed high accuracy in defining success and nonsuccess and only minor differences were found between 3- and 12-month scores. At 12 months, the area under the receiver operating curves for follow-up scores were 0.86 to 0.91, change scores were 0.74 to 0.87, and percentage change scores were 0.74 to 0.91. Percentage scores of NDI and NRS-AP showed the best accuracy. The optimal cut-off values for each PROM showed considerable overlap across those operated due to disc herniation and spondylotic foraminal stenosis.

CONCLUSIONS

All PROMs, especially NDI and NRS-AP, showed good to excellent discriminative ability in distinguishing between a successful and nonsuccessful outcome after surgery due to cervical radiculopathy. Percentage change scores are recommended for use in research and clinical practice.

The NECK trial: Effectiveness of anterior cervical discectomy with or without interbody fusion and arthroplasty in the treatment of cervical disc herniation; a double-blinded randomized controlled trial

BACKGROUND

Motion preserving anterior cervical disc arthroplasty (ACDA) in patients with cervical radiculopathy was introduced to prevent symptomatic adjacent disc degeneration as compared with anterior cervical discectomy and fusion (ACDF). Prior reports suggest that ACDF is not more effective than anterior cervical discectomy (ACD) alone for the treatment of cervical radiculopathy.

PURPOSE

To evaluate whether patients with cervical radiculopathy due to a herniated disc benefit more from undergoing ACDA, ACDF, or ACD in terms of clinical outcome measured by the neck disability index (NDI).

STUDY DESIGN

Double-blinded randomized controlled trial.

METHODS

One hundred-nine patients with one level herniated disc were randomized to one of the following treatments: ACDA, ACDF with intervertebral cage, ACD without fusion. Clinical and radiological outcome was measured by NDI, Visual Analogue Scale (VAS) neck pain, VAS arm pain, SF36, EQ-5D, patients' self-reported perceived recovery, radiographic cervical curvature, and adjacent segment degeneration parameters at baseline and until two years after surgery. BBraun Medical paid €298.837 to cover the costs for research nurses.

RESULTS

The NDI declined from 41 to 47 points at baseline to 19 ± 15 in the ACD group, 19 ± 18 in the ACDF group, and 20 ± 22 in the ACDA group after surgery (p=.929). VAS arm and neck pain declined to half its baseline value and decreased below the critical value of 40 mm. Quality of life, measured by the EQ-5D, increased in all three groups. Adjacent segment degeneration parameters were comparable in all three groups as well. No statistical differences were demonstrated between the treatment groups.

CONCLUSIONS

The hypothesis that ACDA would lead to superior clinical outcome in comparison to ACDF or ACD could not be confirmed during a 2-year follow-up time period. Single level ACD without implanting an intervertebral device may be a reasonable alternative to ACDF or ACDA.