Deep learning in knee imaging: a systematic review utilizing a Checklist for Artificial Intelligence in Medical Imaging (CLAIM)

Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update

To address the rapid evolution of artificial intelligence in medical imaging, the authors present the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) 2024 Update.

A Comprehensive Evaluation of Deep Learning Models on Knee MRIs for the Diagnosis and Classification of Meniscal Tears: A Systematic Review and Meta-Analysis

OBJECTIVES

This study delves into the cutting-edge field of deep learning techniques, particularly deep convolutional neural networks (DCNNs), which have demonstrated unprecedented potential in assisting radiologists and orthopedic surgeons in precisely identifying meniscal tears. This research aims to evaluate the effectiveness of deep learning models in recognizing, localizing, describing, and categorizing meniscal tears in magnetic resonance images (MRIs).

MATERIALS AND METHODS

This systematic review was rigorously conducted, strictly following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Extensive searches were conducted on MEDLINE (PubMed), Web of Science, Cochrane Library, and Google Scholar. All identified articles underwent a comprehensive risk of bias analysis. Predictive performance values were either extracted or calculated for quantitative analysis, including sensitivity and specificity. The meta-analysis was performed for all prediction models that identified the presence and location of meniscus tears.

RESULTS

This study's findings underscore that a range of deep learning models exhibit robust performance in detecting and classifying meniscal tears, in one case surpassing the expertise of musculoskeletal radiologists. Most studies in this review concentrated on identifying tears in the medial or lateral meniscus and even precisely locating tears-whether in the anterior or posterior horn-with exceptional accuracy, as demonstrated by AUC values ranging from 0.83 to 0.94.

CONCLUSIONS

Based on these findings, deep learning models have showcased significant potential in analyzing knee MR images by learning intricate details within images. They offer precise outcomes across diverse tasks, including segmenting specific anatomical structures and identifying pathological regions. Contributions: This study focused exclusively on DL models for identifying and localizing meniscus tears. It presents a meta-analysis that includes eight studies for detecting the presence of a torn meniscus and a meta-analysis of three studies with low heterogeneity that localize and classify the menisci. Another novelty is the analysis of arthroscopic surgery as ground truth. The quality of the studies was assessed against the CLAIM checklist, and the risk of bias was determined using the QUADAS-2 tool.

Self-reporting with checklists in artificial intelligence research on medical imaging: a systematic review based on citations of CLAIM

OBJECTIVE

To evaluate the usage of a well-known and widely adopted checklist, Checklist for Artificial Intelligence in Medical imaging (CLAIM), for self-reporting through a systematic analysis of its citations.

METHODS

Google Scholar, Web of Science, and Scopus were used to search for citations (date, 29 April 2023). CLAIM's use for self-reporting with proof (i.e., filled-out checklist) and other potential use cases were systematically assessed in research papers. Eligible papers were evaluated independently by two readers, with the help of automatic annotation. Item-by-item confirmation analysis on papers with checklist proof was subsequently performed.

RESULTS

A total of 391 unique citations were identified from three databases. Of the 118 papers included in this study, 12 (10%) provided a proof of self-reported CLAIM checklist. More than half (70; 59%) only mentioned some sort of adherence to CLAIM without providing any proof in the form of a checklist. Approximately one-third (36; 31%) cited the CLAIM for reasons unrelated to their reporting or methodological adherence. Overall, the claims on 57 to 93% of the items per publication were confirmed in the item-by-item analysis, with a mean and standard deviation of 81% and 10%, respectively.

CONCLUSION

Only a small proportion of the publications used CLAIM as checklist and supplied filled-out documentation; however, the self-reported checklists may contain errors and should be approached cautiously. We hope that this systematic citation analysis would motivate artificial intelligence community about the importance of proper self-reporting, and encourage researchers, journals, editors, and reviewers to take action to ensure the proper usage of checklists.

CLINICAL RELEVANCE STATEMENT

Only a small percentage of the publications used CLAIM for self-reporting with proof (i.e., filled-out checklist). However, the filled-out checklist proofs may contain errors, e.g., false claims of adherence, and should be approached cautiously. These may indicate inappropriate usage of checklists and necessitate further action by authorities.

KEY POINTS

• Of 118 eligible papers, only 12 (10%) followed the CLAIM checklist for self-reporting with proof (i.e., filled-out checklist). More than half (70; 59%) only mentioned some kind of adherence without providing any proof. • Overall, claims on 57 to 93% of the items were valid in item-by-item confirmation analysis, with a mean and standard deviation of 81% and 10%, respectively. • Even with the checklist proof, the items declared may contain errors and should be approached cautiously.

The endorsement of general and artificial intelligence reporting guidelines in radiological journals: a meta-research study

BACKGROUND

Complete reporting is essential for clinical research. However, the endorsement of reporting guidelines in radiological journals is still unclear. Further, as a field extensively utilizing artificial intelligence (AI), the adoption of both general and AI reporting guidelines would be necessary for enhancing quality and transparency of radiological research. This study aims to investigate the endorsement of general reporting guidelines and those for AI applications in medical imaging in radiological journals, and explore associated journal characteristic variables.

METHODS

This meta-research study screened journals from the Radiology, Nuclear Medicine & Medical Imaging category, Science Citation Index Expanded of the 2022 Journal Citation Reports, and excluded journals not publishing original research, in non-English languages, and instructions for authors unavailable. The endorsement of fifteen general reporting guidelines and ten AI reporting guidelines was rated using a five-level tool: "active strong", "active weak", "passive moderate", "passive weak", and "none". The association between endorsement and journal characteristic variables was evaluated by logistic regression analysis.

RESULTS

We included 117 journals. The top-five endorsed reporting guidelines were CONSORT (Consolidated Standards of Reporting Trials, 58.1%, 68/117), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, 54.7%, 64/117), STROBE (STrengthening the Reporting of Observational Studies in Epidemiology, 51.3%, 60/117), STARD (Standards for Reporting of Diagnostic Accuracy, 50.4%, 59/117), and ARRIVE (Animal Research Reporting of In Vivo Experiments, 35.9%, 42/117). The most implemented AI reporting guideline was CLAIM (Checklist for Artificial Intelligence in Medical Imaging, 1.7%, 2/117), while other nine AI reporting guidelines were not mentioned. The Journal Impact Factor quartile and publisher were associated with endorsement of reporting guidelines in radiological journals.

CONCLUSIONS

The general reporting guideline endorsement was suboptimal in radiological journals. The implementation of reporting guidelines for AI applications in medical imaging was extremely low. Their adoption should be strengthened to facilitate quality and transparency of radiological study reporting.

Reporting Quality of Research Studies on AI Applications in Medical Images According to the CLAIM Guidelines in a Radiology Journal With a Strong Prominence in Asia

OBJECTIVE

We aimed to evaluate the reporting quality of research articles that applied deep learning to medical imaging. Using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines and a journal with prominence in Asia as a sample, we intended to provide an insight into reporting quality in the Asian region and establish a journal-specific audit.

MATERIALS AND METHODS

A total of 38 articles published in the Korean Journal of Radiology between June 2018 and January 2023 were analyzed. The analysis included calculating the percentage of studies that adhered to each CLAIM item and identifying items that were met by ≤ 50% of the studies. The article review was initially conducted independently by two reviewers, and the consensus results were used for the final analysis. We also compared adherence rates to CLAIM before and after December 2020.

RESULTS

Of the 42 items in the CLAIM guidelines, 12 items (29%) were satisfied by ≤ 50% of the included articles. None of the studies reported handling missing data (item #13). Only one study respectively presented the use of de-identification methods (#12), intended sample size (#19), robustness or sensitivity analysis (#30), and full study protocol (#41). Of the studies, 35% reported the selection of data subsets (#10), 40% reported registration information (#40), and 50% measured inter and intrarater variability (#18). No significant changes were observed in the rates of adherence to these 12 items before and after December 2020.

CONCLUSION

The reporting quality of artificial intelligence studies according to CLAIM guidelines, in our study sample, showed room for improvement. We recommend that the authors and reviewers have a solid understanding of the relevant reporting guidelines and ensure that the essential elements are adequately reported when writing and reviewing the manuscripts for publication.

Artificial intelligence in knee osteoarthritis: A comprehensive review for 2022

Objective

The aim of this literature review is to yield a comprehensive and exhaustive overview of the existing evidence and up-to-date applications of artificial intelligence for knee osteoarthritis.

Methods

A literature review was performed by using PubMed, Google Scholar, and IEEE databases for articles published in peer-reviewed journals in 2022. The articles focusing on the use of artificial intelligence in diagnosis and prognosis of knee osteoarthritis and accelerating the image acquisition were selected. For each selected study, the code availability, considered number of patients and knees, imaging type, covariates, grading type of osteoarthritis, models, validation approaches, objectives, and results were reviewed.

Results

395 articles were screened, and 35 of them were reviewed. Eight articles were based on diagnosis, six on prognosis prediction, three on classification, three on accelerated image acquisition, and 15 on segmentation of knee osteoarthritis. 57% of the articles used MRI, 26% radiography, 6% MRI together with radiography, 6% ultrasonography, and 6% only clinical data. 23% of the articles made the computer codes available for their study, and 26% used clinical data. External validation and nested cross-validation were used in 17% and 14% of articles, respectively.

Conclusions

The use of artificial intelligence provided a promising potential to enhance the detection and management of knee osteoarthritis. Translating the developed models into clinics is still in the early stages of development. The translation of artificial intelligence models is expected to be further examined in prospective studies to support clinicians in improving routine healthcare practice.

Past, present, and future in sports imaging: how to drive in a three-lane freeway

KEY POINTS

• Morphological evaluation of SRIs is still nowadays the clinical standard in daily practice.• New functional imaging modalities show potential to add valuable physiopathological information about the insights of SRIs in specific clinical scenarios.• In the era of personalized medicine, AI algorithms may help athletes and all professionals involved in their care to improve the evaluation of SRIs through a definitive quantitative metric approach.

The Low Rate of Adherence to Checklist for Artificial Intelligence in Medical Imaging Criteria Among Published Prostate MRI Artificial Intelligence Algorithms

OBJECTIVE

To determine the rigor, generalizability, and reproducibility of published classification and detection artificial intelligence (AI) models for prostate cancer (PCa) on MRI using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines, a 42-item checklist that is considered a measure of best practice for presenting and reviewing medical imaging AI research.

MATERIALS AND METHODS

This review searched English literature for studies proposing PCa AI detection and classification models on MRI. Each study was evaluated with the CLAIM checklist. The additional outcomes for which data were sought included measures of AI model performance (eg, area under the curve [AUC], sensitivity, specificity, free-response operating characteristic curves), training and validation and testing group sample size, AI approach, detection versus classification AI, public data set utilization, MRI sequences used, and definition of gold standard for ground truth. The percentage of CLAIM checklist fulfillment was used to stratify studies into quartiles. Wilcoxon's rank-sum test was used for pair-wise comparisons.

RESULTS

In all, 75 studies were identified, and 53 studies qualified for analysis. The original CLAIM items that most studies did not fulfill includes item 12 (77% no): de-identification methods; item 13 (68% no): handling missing data; item 15 (47% no): rationale for choosing ground truth reference standard; item 18 (55% no): measurements of inter- and intrareader variability; item 31 (60% no): inclusion of validated interpretability maps; item 37 (92% no): inclusion of failure analysis to elucidate AI model weaknesses. An AUC score versus percentage CLAIM fulfillment quartile revealed a significant difference of the mean AUC scores between quartile 1 versus quartile 2 (0.78 versus 0.86, P = .034) and quartile 1 versus quartile 4 (0.78 versus 0.89, P = .003) scores. Based on additional information and outcome metrics gathered in this study, additional measures of best practice are defined. These new items include disclosure of public dataset usage, ground truth definition in comparison to other referenced works in the defined task, and sample size power calculation.

CONCLUSION

A large proportion of AI studies do not fulfill key items in CLAIM guidelines within their methods and results sections. The percentage of CLAIM checklist fulfillment is weakly associated with improved AI model performance. Additions or supplementations to CLAIM are recommended to improve publishing standards and aid reviewers in determining study rigor.

Democratization of deep learning for segmenting cartilage from MRIs of human knees: Application to data from the osteoarthritis initiative

In this study, we aimed to democratize access to convolutional neural networks (CNN) for segmenting cartilage volumes, generating state-of-the-art results for specialized, real-world applications in hospitals and research. Segmentation of cross-sectional and/or longitudinal magnetic resonance (MR) images of articular cartilage facilitates both clinical management of joint damage/disease and fundamental research. Manual delineation of such images is a time-consuming task susceptible to high intra- and interoperator variability and prone to errors. Thus, enabling reliable and efficient analyses of MRIs of cartilage requires automated segmentation of cartilage volumes. Two main limitations arise in the development of hospital- or population-specific deep learning (DL) models for image segmentation: specialized knowledge and specialized hardware. We present a relatively easy and accessible implementation of a DL model to automatically segment MRIs of human knees with state-of-the-art accuracy. In representative examples, we trained CNN models in 6-8 h and obtained results quantitatively comparable to state-of-the-art for every anatomical structure. We established and evaluated our methods using two publicly available MRI data sets originating from the Osteoarthritis Initiative, Stryker Imorphics, and Zuse Institute Berlin (ZIB), as representative test cases. We use Google Colabfor editing and adapting the Python codes and selecting the runtime environment leveraging high-performance graphical processing units. We designed our solution for novice users to apply to any data set with relatively few adaptations requiring only basic programming skills. To facilitate the adoption of our methods, we provide a complete guideline for using our methods and software, as well as the software tools themselves. Clinical significance: We establish and detail methods that clinical personal can apply to create their own DL models without specialized knowledge of DL nor specialized hardware/infrastructure and obtain results comparable with the state-of-the-art to facilitate both clinical management of joint damage/disease and fundamental research.

An updated systematic review of radiomics in osteosarcoma: utilizing CLAIM to adapt the increasing trend of deep learning application in radiomics

Objective

To update the systematic review of radiomics in osteosarcoma.

Methods

PubMed, Embase, Web of Science, China National Knowledge Infrastructure, and Wanfang Data were searched to identify articles on osteosarcoma radiomics until May 15, 2022. The studies were assessed by Radiomics Quality Score (RQS), Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Checklist for Artificial Intelligence in Medical Imaging (CLAIM), and modified Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. The evidence supporting radiomics application for osteosarcoma was rated according to meta-analysis results.

Results

Twenty-nine articles were included. The average of the ideal percentage of RQS, the TRIPOD adherence rate and the CLAIM adherence rate were 29.2%, 59.2%, and 63.7%, respectively. RQS identified a radiomics-specific issue of phantom study. TRIPOD addressed deficiency in blindness of assessment. CLAIM and TRIPOD both pointed out shortness in missing data handling and sample size or power calculation. CLAIM identified extra disadvantages in data de-identification and failure analysis. External validation and open science were emphasized by all the above three tools. The risk of bias and applicability concerns were mainly related to the index test. The meta-analysis of radiomics predicting neoadjuvant chemotherapy response by MRI presented a diagnostic odds ratio (95% confidence interval) of 28.83 (10.27–80.95) on testing datasets and was rated as weak evidence.

Conclusions

The quality of osteosarcoma radiomics studies is insufficient. More investigation is needed before using radiomics to optimize osteosarcoma treatment. CLAIM is recommended to guide the design and reporting of radiomics research.

A systematic review of radiomics in pancreatitis: applying the evidence level rating tool for promoting clinical transferability

Background

Multiple tools have been applied to radiomics evaluation, while evidence rating tools for this field are still lacking. This study aims to assess the quality of pancreatitis radiomics research and test the feasibility of the evidence level rating tool.

Results

Thirty studies were included after a systematic search of pancreatitis radiomics studies until February 28, 2022, via five databases. Twenty-four studies employed radiomics for diagnostic purposes. The mean ± standard deviation of the adherence rate was 38.3 ± 13.3%, 61.3 ± 11.9%, and 37.1 ± 27.2% for the Radiomics Quality Score (RQS), the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist, and the Image Biomarker Standardization Initiative (IBSI) guideline for preprocessing steps, respectively. The median (range) of RQS was 7.0 (− 3.0 to 18.0). The risk of bias and application concerns were mainly related to the index test according to the modified Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool. The meta-analysis on differential diagnosis of autoimmune pancreatitis versus pancreatic cancer by CT and mass-forming pancreatitis versus pancreatic cancer by MRI showed diagnostic odds ratios (95% confidence intervals) of, respectively, 189.63 (79.65–451.48) and 135.70 (36.17–509.13), both rated as weak evidence mainly due to the insufficient sample size.

Conclusions

More research on prognosis of acute pancreatitis is encouraged. The current pancreatitis radiomics studies have insufficient quality and share common scientific disadvantages. The evidence level rating is feasible and necessary for bringing the field of radiomics from preclinical research area to clinical stage.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13244-022-01279-4.

Automated detection of knee cystic lesions on magnetic resonance imaging using deep learning

Background

Cystic lesions are frequently observed in knee joint diseases and are usually associated with joint pain, degenerative disorders, or acute injury. Magnetic resonance imaging-based, artificial intelligence-assisted cyst detection is an effective method to improve the whole knee joint analysis. However, few studies have investigated this method. This study is the first attempt at auto-detection of knee cysts based on deep learning methods.

Methods

This retrospective study collected data from 282 subjects with knee cysts confirmed at our institution from January to October 2021. A Squeeze-and-Excitation (SE) inception attention-based You only look once version 5 (SE-YOLOv5) model was developed based on a self-attention mechanism for knee cyst-like lesion detection and differentiation from knee effusions, both characterized by high T2-weighted signals in magnetic resonance imaging (MRI) scans. Model performance was evaluated via metrics including accuracy, precision, recall, mean average precision (mAP), F1 score, and frames per second (fps).

Results

The deep learning model could accurately identify knee MRI scans and auto-detect both obvious cyst lesions and small ones with inconspicuous contrasts. The SE-YOLO V5 model constructed in this study yielded superior performance (F1 = 0.879, precision = 0.887, recall = 0.872, all class mAP0.5 = 0.944, effusion mAP = 0.945, cyst mAP = 0.942) and improved detection speed compared to a traditional YOLO model.

Conclusion

This proof-of-concept study examined whether deep learning models could detect knee cysts and distinguish them from knee effusions. The results demonstrated that the classical Yolo V5 and proposed SE-Yolo V5 models could accurately identify cysts.

3D CAIPIRINHA SPACE versus standard 2D TSE for routine knee MRI: a large-scale interchangeability study

OBJECTIVE

To perform a large-scale interchangeability study comparing 3D controlled aliasing in parallel imaging results in higher acceleration (CAIPIRINHA) sampling perfection with application optimized contrast using different flip angle evolutions (SPACE) TSE with standard 2D TSE for knee MRI.

METHODS

In this prospective study, 250 patients underwent 3 T knee MRI, including a multicontrast 3D CAIPIRINHA SPACE TSE (9:26 min) and a standard 2D TSE protocol (12:14 min). Thirty-three (13%) patients had previous anterior cruciate ligament and/or meniscus surgery. Two radiologists assessed MRIs for image quality and identified pathologies of menisci, ligaments, and cartilage by using a 4-point Likert scale according to the level of diagnostic confidence. Interchangeability of the protocols was tested under the same-reader scenario using a bootstrap percentile confidence interval. Interreader reliability and intermethod concordance were also evaluated.

RESULTS

Despite higher image quality and diagnostic confidence for standard 2D TSE compared to 3D CAIPIRINHA SPACE TSE, the protocols were found interchangeable for diagnosing knee abnormalities, except for patellar (6.8% difference; 95% CI: 4.0, 9.6) and trochlear (3.6% difference; 95% CI: 0.8, 6.6) cartilage defects. The interreader reliability was substantial to almost perfect for 2D and 3D MRI (range κ, 0.785-1 and κ, 0.725-0.964, respectively). Intermethod concordance was almost perfect for all diagnoses (range κ, 0.817-0.986).

CONCLUSION

Multicontrast 3D CAIPIRINHA SPACE TSE and standard 2D TSE protocols perform interchangeably for diagnosing knee abnormalities, except for patellofemoral cartilage defects. Despite the radiologist's preference for 2D TSE imaging, a pursuit towards time-saving 3D TSE knee MRI is justified for routine practice.

KEY POINTS

• Multicontrast 3D CAIPIRINHA SPACE and standard 2D TSE protocols perform interchangeably for diagnosing knee abnormalities, except for patellofemoral cartilage defects. • Radiologists are more confident in diagnosing knee abnormalities on 2D TSE than on 3D CAIPIRINHA SPACE TSE MRI. • Despite the radiologist's preference for 2D TSE, a pursuit towards accelerated 3D TSE knee MRI is justified for routine practice.