Machine Learning in Differentiating Gliomas from Primary CNS Lymphomas: A Systematic Review, Reporting Quality, and Risk of Bias Assessment

BACKGROUND

Differentiating gliomas and primary CNS lymphoma represents a diagnostic challenge with important therapeutic ramifications. Biopsy is the preferred method of diagnosis, while MR imaging in conjunction with machine learning has shown promising results in differentiating these tumors.

PURPOSE

Our aim was to evaluate the quality of reporting and risk of bias, assess data bases with which the machine learning classification algorithms were developed, the algorithms themselves, and their performance.

DATA SOURCES

Ovid EMBASE, Ovid MEDLINE, Cochrane Central Register of Controlled Trials, and the Web of Science Core Collection were searched according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

STUDY SELECTION

From 11,727 studies, 23 peer-reviewed studies used machine learning to differentiate primary CNS lymphoma from gliomas in 2276 patients.

DATA ANALYSIS

Characteristics of data sets and machine learning algorithms were extracted. A meta-analysis on a subset of studies was performed. Reporting quality and risk of bias were assessed using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) and Prediction Model Study Risk Of Bias Assessment Tool.

DATA SYNTHESIS

The highest area under the receiver operating characteristic curve (0.961) and accuracy (91.2%) in external validation were achieved by logistic regression and support vector machines models using conventional radiomic features. Meta-analysis of machine learning classifiers using these features yielded a mean area under the receiver operating characteristic curve of 0.944 (95% CI, 0.898-0.99). The median TRIPOD score was 51.7%. The risk of bias was high for 16 studies.

LIMITATIONS

Exclusion of abstracts decreased the sensitivity in evaluating all published studies. Meta-analysis had high heterogeneity.

CONCLUSIONS

Machine learning-based methods of differentiating primary CNS lymphoma from gliomas have shown great potential, but most studies lack large, balanced data sets and external validation. Assessment of the studies identified multiple deficiencies in reporting quality and risk of bias. These factors reduce the generalizability and reproducibility of the findings.

NIMG-67. A SYSTEMATIC REVIEW ON THE DEVELOPMENT OF MACHINE LEARNING MODELS FOR DIFFERENTIATING PCNSL FROM GLIOMAS

PURPOSE

Differentiating gliomas and Primary CNS Lymphomas (PCNSL) represents a diagnostic challenge with important therapeutic ramifications. MR imaging combined with Machine Learning (ML) has shown promising results in differentiating tumors non-invasively. The purpose of this systematic review is to evaluate and synthesize the findings on the application of ML in differentiating PCNSL and gliomas.

MATERIALS AND METHODS

A systematic search of literature was performed in October 2020 and February 2021 on Ovid Embase, Ovid MEDLINE, Cochrane trials, and Web of Science – Core Collection. The search strategy included keywords and controlled vocabulary including the terms: gliomas, artificial intelligence, machine learning, and related terms. Publications were reviewed and screened by four different reviewers in accordance with TRIPOD.

RESULTS

The literature search yielded 11,727 studies and 1,135 underwent full-text review. Data was extracted from 16 publications showing that 10 ML and 3 deep learning (DL) algorithms were tested. The analyzed databases had an average size of 118 patients per study. 50% of the publications validated the algorithm in an independent test cohort. The most commonly tested ML and DL algorithms were support vector machines and Convolutional Neural Networks, respectively. In internal (external) datasets, ML algorithms reached an average AUC of 89% (83%); and DL 74% (77%). Preliminary TRIPOD bias analysis yielded an average score of 0.5 (range 0.31-0.62), with most papers showing deficiencies in reporting model specifications, and funding details among other items.

CONCLUSIONS

AI-based methods for differentiating gliomas and PCNSL have been reported and show that ML methods result in accuracy = > 85%.With few studies using DL algorithms, further research into novel DL-based approaches is recommended. Additionally, most studies lack large datasets and external validation, thus increasing the risk of overfitting. Bias analysis of the published studies using TRIPOD identified reporting deficiencies, and close adherence to reporting criteria is recommended.

NIMG-46. SYSTEMATIC LITERATURE REVIEW OF ARTIFICIAL INTELLIGENCE ALGORITHMS USING PRE-THERAPY MR IMAGING FOR GLIOMA MOLECULAR SUBTYPE CLASSIFICATION

PURPOSE

Identifying molecular subtypes in gliomas has prognostic and therapeutic value, traditionally after invasive neurosurgical tumor resection or biopsy. Recent advances using artificial intelligence (AI) show promise in using pre-therapy imaging for predicting molecular subtype. We performed a systematic review of recent literature on AI methods used to predict molecular subtypes of gliomas.

METHODS

Literature review conforming to PRSIMA guidelines was performed for publications prior to February 2021 using 4 databases: Ovid Embase, Ovid MEDLINE, Cochrane trials (CENTRAL), and Web of Science core-collection. Keywords included: artificial intelligence, machine learning, deep learning, radiomics, magnetic resonance imaging, glioma, and glioblastoma. Non-machine learning and non-human studies were excluded. Screening was performed using Covidence software. Bias analysis was done using TRIPOD guidelines.

RESULTS

11,727 abstracts were retrieved. After applying initial screening exclusion criteria, 1,135 full text reviews were performed, with 82 papers remaining for data extraction. 57% used retrospective single center hospital data, 31.6% used TCIA and BRATS, and 11.4% analyzed multicenter hospital data. An average of 146 patients (range 34-462 patients) were included. Algorithms predicting IDH status comprised 51.8% of studies, MGMT 18.1%, and 1p19q 6.0%. Machine learning methods were used in 71.4%, deep learning in 27.4%, and 1.2% directly compared both methods. The most common algorithm for machine learning were support vector machine (43.3%), and for deep learning convolutional neural network (68.4%). Mean prediction accuracy was 76.6%.

CONCLUSION

Machine learning is the predominant method for image-based prediction of glioma molecular subtypes. Major limitations include limited datasets (60.2% with under 150 patients) and thus limited generalizability of findings. We recommend using larger annotated datasets for AI network training and testing in order to create more robust AI algorithms, which will provide better prediction accuracy to real world clinical datasets and provide tools that can be translated to clinical practice.

NIMG-17. SYSTEMATIC REVIEW OF LITERATURE EVALUATING MACHINE LEARNING ALGORITHMS TO DEVELOP OUTCOME PREDICTION MODELS IN GLIOMA USING MOLECULAR IMAGING WITH AMINO ACID PET

PURPOSE

Machine learning (ML) algorithms demonstrate accurate prediction of tumor segmentation, molecular pathology, and outcomes in gliomas using MRI and recently application of ML tools has expanded into molecular imaging with PET. We performed a systematic review to evaluate the role and applications of ML in characterization of gliomas with PET.

METHODS

Four databases were searched by medical school librarian and confirmed by an independent librarian: Ovid Embase, Ovid MEDLINE, Cochrane trials (CENTRAL), and Web of Science-Core Collection. The search strategy used keywords and controlled vocabulary combining the terms for: artificial intelligence, machine learning, deep learning, radiomics, magnetic resonance imaging, glioma, and related terms. All articles were reviewed by at least 2 independent reviewers at abstract screening, full text review, data extraction, and bias analysis using TRIPOD.

RESULTS

An initial 11,727 publications were imported to Covidence for screening. After review, 1135 studies moved to full-text review and 715 articles were included. Twelve publications included PET imaging of gliomas. All publications used single-center databases (3-73 patients) with distribution of tracers being [18F]-FDG (1), [18F]-FET (6), [11C]-MET (3), [18F]-FDOPA (1), and [18F]-AMP (1). All but 2 papers used supervised machine learning algorithms. Number of features ranged from 4-19,284. Nine papers manually extracted semiquantitative features TBRmax, TBRmean, SUV, TTP, in addition to demographics. Study outcomes included prediction of treatment response, survival, molecular subtypes, tumor grade, segmentation, and accuracy of image fusion. Accuracy ranged from 0.64-0.95 with AUC 0.43-0.9.

CONCLUSION

ML can be used on small datasets of PET imaging of brain tumors. While majority of the clinical scans are performed with FDG-PET, the machine learning approaches are being applied to mostly amino acid tracers. Extending ML approaches to FDG-PET, which is more common in clinical practice, is recommended. Overall, ML has potential as a useful tool for predicting patient outcomes and improving image postprocessing.

NIMG-23. MACHINE LEARNING METHODS IN GLIOMA GRADE PREDICTION: A SYSTEMATIC REVIEW

PURPOSE

Machine learning (ML) technologies have demonstrated highly accurate prediction of glioma grade, though it is unclear which methods and algorithms are superior. We have conducted a systematic review of the literature in order to identify the ML applications most promising for future research and clinical implementation.

MATERIALS AND METHODS

A literature review, in agreement with PRISMA, was conducted by a university librarian in October 2020 and verified by a second librarian in February 2021 using four databases: Cochrane trials (CENTRAL), Ovid Embase, Ovid MEDLINE, and Web of Science core-collection. Keywords and controlled vocabulary included artificial intelligence, machine learning, deep learning, radiomics, magnetic resonance imaging, glioma, and glioblastoma. Screening of publications was done in Covidence, and TRIPOD was used for bias assessment.

RESULTS

The search identified 11,727 candidate articles with 1,135 articles undergoing full text review. 86 articles published since 1995 met the criteria for our study. 79% of the articles were published between 2018 and 2020. The average glioma prediction accuracy of the highest performing model in each study was 90% (range: 53% to 100%). The most common algorithm used for cML studies was Support Vector Machine (SVM) and for DL studies was Convolutional Neural Network (CNN). BRATS and TCIA datasets were used in 47% of the studies, with the average patient number of study datasets being 186 (range: 23 to 662). The average number of features used in machine learning prediction was 55 (range: 2 to 580). Classical machine learning (cML) was the primary machine learning model in 68% of studies, with deep learning (DL) used in 32%.

CONCLUSIONS

Using multimodal sequences in ML methods delivers significantly higher grading accuracies than single sequences. Potential areas of improvement for ML glioma grade prediction studies include increasing sample size, incorporating molecular subtypes, and validating on external datasets.

NIMG-71. IDENTIFYING CLINICALLY APPLICABLE MACHINE LEARNING ALGORITHMS FOR GLIOMA SEGMENTATION USING A SYSTEMATIC LITERATURE REVIEW

PURPOSE

Nowadays Machine learning (ML) algorithms are often used for segmentation of gliomas, but which algorithms provide the most accurate method for implementation into clinical practice has not fully been identified. We performed a systematic review of the literature to characterize the methods used for glioma segmentation and their accuracy.

METHODS

In accordance to PRISMA, a literature review was performed on four databases, Ovid Embase, Ovid MEDLINE, Cochrane trials (CENTRAL) and Web of science core-collection first in October 2020 and in February 2021. Keywords and controlled vocabulary included artificial intelligence, machine learning, deep learning, radiomics, magnetic resonance imaging, glioma, and glioblastoma. Publications were screened in Covidence and the bias analysis was done in agreement with TRIPOD.

RESULTS

Sixty-six articles were used for data extraction. BRATS and TCIA datasets were used in 36.6% of all studies, with average number of patients being 141 (range: 1 to 622). ML methods represented 45.3% of studies, with deep learning used in 54.7%; Dice score for the tumor core ranged from 0.72 to 0.95. The most common algorithm used in the machine learning papers was support vector machines (SVM) and for deep learning papers, it was Convolutional Neural Networks (CNN). Preliminary TRIPOD analysis yielded an average score from 12 (range: 7-16) with the majority of papers demonstrating deficiencies in description of the ML algorithm, funding role, data acquisition and measures of model performance.

CONCLUSION

In the last years, many articles were published on segmentation of gliomas using machine learning, thus establishing this method for tumor segmentation with high accuracy. However, the major limitations for clinically applicable use of ML in glioma segmentation include more than one-third of publications use the same datasets, thus limiting generalizability, increase the likelihood of overfitting, show and lack of ML network description and standardization in accuracy reporting.

NIMG-38. MEASURING ADHERENCE TO TRIPOD OF ARTIFICIAL INTELLIGENCE PAPERS IN THE GLIOMA SEGMENTATION

PURPOSE

Generalizability, reproducibility and objectivity are critical elements that need to be considered when translating machine learning models into clinical practice. While a large body of literature has been published on machine learning methods for segmentation of brain tumors, a systematic evaluation of paper quality and reproducibility has not been done. We investigated the use of “Transparent Reporting of studies on prediction models for Individual Prognosis Or Diagnosis” (TRIPOD) items, among papers published in this relatively new and growing field.

METHODS

According to PRISMA a literature review was performed on four databases, Ovid Embase, Ovid MEDLINE, Cochrane trials (CENTRAL) and Web of science core-collection first in October 2020 and a second time in February 2021. Keywords and controlled vocabulary included artificial intelligence, machine learning, deep learning, radiomics, magnetic resonance imaging, glioma, and glioblastoma. The publications were assessed in order to the TRIPOD items.

RESULTS

37 publications from our database search were screened in TRIPOD and yielded an average score of 12.08 with the maximum score being 16 and the minimum score 7. The best scoring item was interpretation (item 19) where all papers scored a point. The lowest scoring items were the title, the abstract, risk groups and the model performance (items number 1, 2, 11 and 16), where no paper scored a point. Less than 1% of the papers discussed the problem of missing data (item 9) and the funding of research (item 22).

CONCLUSION

TRIPOD analysis showed that a majority of the papers do not score high on critical elements that allow reproducibility, translation, and objectivity of research. An average score of 12.08 (40%) indicates that the publications usually achieve a relatively low score. The categories that were consistently poorly described include the ML network description, measuring model performance, title details and inclusion of information into the abstract.