Computer-Aided Diagnosis System for the Evaluation of Thyroid Nodules on Ultrasonography: Prospective Non-Inferiority Study according to the Experience Level of Radiologists

Avoidable biopsies? Validating artificial intelligence-based decision support software in indeterminate thyroid nodules

BACKGROUND

Multiple artificial intelligence (AI) systems have been approved to risk-stratify thyroid nodules through sonographic characterization. We sought to validate the ability of one such AI system, Koios DS (Koios Medical, Chicago, IL), to aid in improving risk stratification of indeterminate thyroid nodules.

METHODS

A retrospective single-institution dataset was compiled of 28 cytologically indeterminate thyroid nodules having undergone molecular testing and surgical resection, with surgical pathology categorized as malignant or benign. Nodules were retrospectively evaluated with Koios DS. After nodule selection, automated and AI-adapter-derived Thyroid Imaging Reporting and Data System (TI-RADS) levels were recorded, and agreement with radiologist-derived levels was assessed using Cohen's κ statistic. The performance of malignancy classification was compared between the radiologist and AI-adapter. Biopsy thresholds were re-evaluated using the AI-adapter.

RESULTS

In this cohort, 7 (25%) nodules were malignant on surgical pathology. The median nodule size was 2.4 cm (interquartile range: 1.8-2.9 cm). Median radiologist and automated TI-RADS levels were both 4, with κ 0.25 ("fair agreement"). Malignancy classification by the radiologist provided sensitivity 100%, specificity 33.3%, positive predictive value (PPV) 33.3%, and negative predictive value (NPV) 100%, compared with the AI-adapter's performance with sensitivity 85.7%, specificity 76.2%, PPV 54.5%, and NPV 94.1%. Using the AI-adapter, 14 of 28 biopsies would have been deferred, 13 of which were surgically benign.

CONCLUSION

Koios automated and radiologist-derived TI-RADS levels were in consistent agreement for indeterminate thyroid nodules. Malignancy reclassification with the AI-adapter improved PPV at minimal cost to NPV. Risk stratification with the addition of the AI-adapter may allow for more accurate patient counseling and the avoidance of biopsies in select cases that would otherwise be cytologically indeterminate.

Integrating artificial intelligence (S-Detect software) and contrast-enhanced ultrasound for enhanced diagnosis of thyroid nodules: A comprehensive evaluation study

PURPOSE

This study aims to assess the diagnostic efficacy of Korean Thyroid imaging reporting and data system (K-TIRADS), S-Detect software and contrast-enhanced ultrasound (CEUS) when employed individually, as well as their combined application, for the evaluation of thyroid nodules, with the objective of identifying the optimal method for diagnosing thyroid nodules.

METHODS

Two hundred and sixty eight cases pathologically proven of thyroid nodules were retrospectively enrolled. Each nodule was classified according to K-TIRADS. S-Detect software was utilized for intelligent analysis. CEUS was employed to acquire contrast-enhanced features.

RESULTS

The area under curve (AUC) values for diagnosing benign and malignant thyroid nodules using K-TIRADS alone, S-Detect software alone, CEUS alone, the combined application of K-TIRADS and CEUS, the combined application of S-Detect software and CEUS were 0.668, 0.668, 0.719, 0.741, and 0.759, respectively (p < 0.001). The sensitivity rate of S-Detect software was 89.9% (p < 0.001). It was the highest of the five diagnostic methods above.

CONCLUSION

The utilization of S-Detect software can be served as a powerful tool for early screening. Notably, the combined utilization of S-Detect software with CEUS demonstrates superior diagnostic performance compared to employing K-TIRADS, S-Detect software, CEUS used individually, as well as the combined application of K-TIRADS with CEUS.

Improving the diagnostic performance of inexperienced readers for thyroid nodules through digital self-learning and artificial intelligence assistance

Background

Data-driven digital learning could improve the diagnostic performance of novice students for thyroid nodules.

Objective

To evaluate the efficacy of digital self-learning and artificial intelligence-based computer-assisted diagnosis (AI-CAD) for inexperienced readers to diagnose thyroid nodules.

Methods

Between February and August 2023, a total of 26 readers (less than 1 year of experience in thyroid US from various departments) from 6 hospitals participated in this study. Readers completed an online learning session comprising 3,000 thyroid nodules annotated as benign or malignant independently. They were asked to assess a test set consisting of 120 thyroid nodules with known surgical pathology before and after a learning session. Then, they referred to AI-CAD and made their final decisions on the thyroid nodules. Diagnostic performances before and after self-training and with AI-CAD assistance were evaluated and compared between radiology residents and readers from different specialties.

Results

AUC (area under the receiver operating characteristic curve) improved after the self-learning session, and it improved further after radiologists referred to AI-CAD (0.679 vs 0.713 vs 0.758, p<0.05). Although the 18 radiology residents showed improved AUC (0.7 to 0.743, p=0.016) and accuracy (69.9% to 74.2%, p=0.013) after self-learning, the readers from other departments did not. With AI-CAD assistance, sensitivity (radiology 70.3% to 74.9%, others 67.9% to 82.3%, all p<0.05) and accuracy (radiology 74.2% to 77.1%, others 64.4% to 72.8%, all p <0.05) improved in all readers.

Conclusion

While AI-CAD assistance helps improve the diagnostic performance of all inexperienced readers for thyroid nodules, self-learning was only effective for radiology residents with more background knowledge of ultrasonography.

Clinical Impact

Online self-learning, along with AI-CAD assistance, can effectively enhance the diagnostic performance of radiology residents in thyroid cancer.

[Artificial intelligence in ultrasound diagnosis of thyroid nodules]

OBJECTIVE

To analyze the efficacy of the S-Detect AI system of the Samsung RS85 ultrasound scanner (South Korea) in stratifying thyroid nodules compared to data obtained by specialist of ultrasound diagnostics.

MATERIAL AND METHODS

In 2024, we analyzed ultrasound data of 80 patients with thyroid nodules who required fine-needle aspiration biopsy (FNAB). Patients underwent thyroid ultrasound with nodule assessment according to the EU TI-RADS classification (GE Voluson E8 scanner, USA). After that, each patient underwent repeated thyroid ultrasound (Samsung RS85 scanner with built-in AI algorithms) to stratify nodules according to the EU TI-RADS classification. After ultrasound, all patients underwent FNAB of nodules with subsequent assessment of cytological results according to the Bethesda classification (2023). We assessed sensitivity, specificity and prognostic value of assessment by expert and AI.

RESULTS. S

Ensitivity, specificity and diagnostic value were high for both expert opinion and AI. However, AI was prone to overdiagnosis. At the same time, expert assessment was characterized by more common false negative results. Diagnostic value of studies conducted by expert and AI was similar.

CONCLUSION

At the modern stage, AI algorithms do not show significant advantages in stratifying thyroid tumors over ultrasound diagnostician with extensive experience.

Learnability of Thyroid Nodule Assessment on Ultrasonography: Using a Big Data Set

OBJECTIVE

The aims of the work described here were to evaluate the learnability of thyroid nodule assessment on ultrasonography (US) using a big data set of US images and to evaluate the diagnostic utilities of artificial intelligence computer-aided diagnosis (AI-CAD) used by readers with varying experience to differentiate benign and malignant thyroid nodules.

METHODS

Six college freshmen independently studied the "learning set" composed of images of 13,560 thyroid nodules, and their diagnostic performance was evaluated after their daily learning sessions using the "test set" composed of images of 282 thyroid nodules. The diagnostic performance of two residents and an experienced radiologist was evaluated using the same "test set." After an initial diagnosis, all readers once again evaluated the "test set" with the assistance of AI-CAD.

RESULTS

Diagnostic performance of almost all students increased after the learning program. Although the mean areas under the receiver operating characteristic curves (AUROCs) of residents and the experienced radiologist were significantly higher than those of students, the AUROCs of five of the six students did not differ significantly compared with that of the one resident. With the assistance of AI-CAD, sensitivity significantly increased in three students, specificity in one student, accuracy in four students and AUROC in four students. Diagnostic performance of the two residents and the experienced radiologist was better with the assistance of AI-CAD.

CONCLUSION

A self-learning method using a big data set of US images has potential as an ancillary tool alongside traditional training methods. With the assistance of AI-CAD, the diagnostic performance of readers with varying experience in thyroid imaging could be further improved.

Artificial intelligence in thyroid ultrasound

Artificial intelligence (AI), particularly deep learning (DL) algorithms, has demonstrated remarkable progress in image-recognition tasks, enabling the automatic quantitative assessment of complex medical images with increased accuracy and efficiency. AI is widely used and is becoming increasingly popular in the field of ultrasound. The rising incidence of thyroid cancer and the workload of physicians have driven the need to utilize AI to efficiently process thyroid ultrasound images. Therefore, leveraging AI in thyroid cancer ultrasound screening and diagnosis cannot only help radiologists achieve more accurate and efficient imaging diagnosis but also reduce their workload. In this paper, we aim to present a comprehensive overview of the technical knowledge of AI with a focus on traditional machine learning (ML) algorithms and DL algorithms. We will also discuss their clinical applications in the ultrasound imaging of thyroid diseases, particularly in differentiating between benign and malignant nodules and predicting cervical lymph node metastasis in thyroid cancer. Finally, we will conclude that AI technology holds great promise for improving the accuracy of thyroid disease ultrasound diagnosis and discuss the potential prospects of AI in this field.

Artificial Intelligence for Pre-operative Diagnosis of Malignant Thyroid Nodules Based on Sonographic Features and Cytology Category

BACKGROUND

Current diagnosis and classification of thyroid nodules are susceptible to subjective factors. Despite widespread use of ultrasonography (USG) and fine needle aspiration cytology (FNAC) to assess thyroid nodules, the interpretation of results is nuanced and requires specialist endocrine surgery input. Using readily available pre-operative data, the aims of this study were to develop artificial intelligence (AI) models to classify nodules into likely benign or malignant and to compare the diagnostic performance of the models.

METHODS

Patients undergoing surgery for thyroid nodules between 2010 and 2020 were recruited from our institution's database into training and testing groups. Demographics, serum TSH level, cytology, ultrasonography features and histopathology data were extracted. The training group USG images were re-reviewed by a study radiologist experienced in thyroid USG, who reported the relevant features and supplemented with data extracted from existing reports to reduce sampling bias. Testing group USG features were extracted solely from existing reports to reflect real-life practice of a non-thyroid specialist. We developed four AI models based on classification algorithms (k-Nearest Neighbour, Support Vector Machine, Decision Tree, Naïve Bayes) and evaluated their diagnostic performance of thyroid malignancy.

RESULTS

In the training group (n = 857), 75% were female and 27% of cases were malignant. The testing group (n = 198) consisted of 77% females and 17% malignant cases. Mean age was 54.7 ± 16.2 years for the training group and 50.1 ± 17.4 years for the testing group. Following validation with the testing group, support vector machine classifier was found to perform best in predicting final histopathology with an accuracy of 89%, sensitivity 89%, specificity 83%, F-score 94% and AUROC 0.86.

CONCLUSION

We have developed a first of its kind, pilot AI model that can accurately predict malignancy in thyroid nodules using USG features, FNAC, demographics and serum TSH. There is potential for a model like this to be used as a decision support tool in under-resourced areas as well as by non-thyroid specialists.

Comparison of S-Detect and thyroid imaging reporting and data system classifications in the diagnosis of cytologically indeterminate thyroid nodules

Purpose

The aim of this study was to investigate the value of S-Detect for predicting the malignant risk of cytologically indeterminate thyroid nodules (CITNs).

Methods

The preoperative prediction of 159 CITNs (Bethesda III, IV and V) were performed using S-Detect, Thyroid Imaging Reporting and Data System of American College of Radiology (ACR TI-RADS) and Chinese TI-RADS (C-TIRADS). First, Linear-by-Linear Association test and chi-square test were used to analyze the malignant risk of CITNs. McNemar's test and receiver operating characteristic curve were used to compare the diagnostic efficacy of S-Detect and the two TI-RADS classifications for CITNs. In addition, the McNemar's test was used to compare the diagnostic accuracy of the above three methods for different pathological types of nodules.

Results

The maximum diameter of the benign nodules was significantly larger than that of malignant nodules [0.88(0.57-1.42) vs 0.57(0.46-0.81), P=0.002]. The risk of malignant CITNs in Bethesda system and the two TI-RADS classifications increased with grade (all P for trend<0.001). In all the enrolled CITNs, the diagnostic results of S-Detect were significantly different from those of ACR TI-RADS and C-TIRADS, respectively (P=0.021 and P=0.007). The sensitivity and accuracy of S-Detect [95.9%(90.1%-98.5%) and 88.1%(81.7%-92.5%)] were higher than those of ACR TI-RADS [87.6%(80.1%-92.7%) and 81.8%(74.7%-87.3%)] (P=0.006 and P=0.021) and C-TIRADS [84.3%(76.3%-90.0%) and 78.6%(71.3%-84.5%)] (P=0.001 and P=0.001). Moreover, the negative predictive value and the area under curve value of S-Detect [82.8% (63.5%-93.5%) and 0.795%(0.724%-0.855%)] was higher than that of C-TIRADS [54.8%(38.8%-69.8%) and 0.724%(0.648%-0.792%] (P=0.024 and P=0.035). However, the specificity and positive predictive value of S-Detect were similar to those of ACR TI-RADS (P=1.000 and P=0.154) and C-TIRADS (P=1.000 and P=0.072). There was no significant difference in all the evaluated indicators between ACR TI-RADS and C-TIRADS (all P>0.05). The diagnostic accuracy of S-Detect (97.4%) for papillary thyroid carcinoma (PTC) was higher than that of ACR TI-RADS (90.4%) and C-TIRADS (87.8%) (P=0.021 and P=0.003).

Conclusion

The diagnostic performance of S-Detect in differentiating CITNs was similar to ACR TI-RADS and superior to C-TIRADS, especially for PTC.

Clinical value of artificial intelligence in thyroid ultrasound: a prospective study from the real world

OBJECTIVE

To evaluate the diagnostic performance of a commercial artificial intelligence (AI)-assisted ultrasonography (US) for thyroid nodules and to validate its value in real-world medical practice.

MATERIALS AND METHODS

From March 2021 to July 2021, 236 consecutive patients with 312 suspicious thyroid nodules were prospectively enrolled in this study. One experienced radiologist performed US examinations with a real-time AI system (S-Detect). US images and AI reports of the nodules were recorded. Nine residents and three senior radiologists were invited to make a "benign" or "malignant" diagnosis based on recorded US images without knowing the AI reports. After referring to AI reports, the diagnosis was made again. The diagnostic performance of AI, residents, and senior radiologists with and without AI reports were analyzed.

RESULTS

The sensitivity, accuracy, and AUC of the AI system were 0.95, 0.84, and 0.753, respectively, and were not statistically different from those of the experienced radiologists, but were superior to those of the residents (all p < 0.01). The AI-assisted resident strategy significantly improved the accuracy and sensitivity for nodules ≤ 1.5 cm (all p < 0.01), while reducing the unnecessary biopsy rate by up to 27.7% for nodules > 1.5 cm (p = 0.034).

CONCLUSIONS

The AI system achieved performance, for cancer diagnosis, comparable to that of an average senior thyroid radiologist. The AI-assisted strategy can significantly improve the overall diagnostic performance for less-experienced radiologists, while increasing the discovery of thyroid cancer ≤ 1.5 cm and reducing unnecessary biopsies for nodules > 1.5 cm in real-world medical practice.

KEY POINTS

• The AI system reached a senior radiologist-like level in the evaluation of thyroid cancer and could significantly improve the overall diagnostic performance of residents. • The AI-assisted strategy significantly improved ≤ 1.5 cm thyroid cancer screening AUC, accuracy, and sensitivity of the residents, leading to an increased detection of thyroid cancer while maintaining a comparable specificity to that of radiologists alone. • The AI-assisted strategy significantly reduced the unnecessary biopsy rate for thyroid nodules > 1.5 cm by the residents, while maintaining a comparable sensitivity to that of radiologists alone.

Patients undergoing endocrine consultation and first diagnosis of nodular disease: Indications of thyroid ultrasound and completeness of ultrasound reports

PURPOSE

To evaluate reasons for performing ultrasonography (US) and completeness of US reports in patients undergoing endocrine consultation with the first diagnosis of nodular disease.

METHODS

Since January 1 to June 30, 2021, we prospectively collected patient data (age and thyroid-stimulating hormone concentrations), reasons for performing thyroid US, and completeness of reports regarding the description of the thyroid gland and nodules. In the case of multiple nodules, we considered the nodule suspected of malignancy and the largest one. To evaluate the accuracy of thyroid nodule description, we referred to the five characteristics suggested by the ACR TI-RADS system.

RESULTS

A total of 341 patients with thyroid nodules received endocrine consultation (female, 78%). The most frequent reasons for performing thyroid US were unrelated to a suspected thyroid disease (31.7%), followed by incidentaloma (23.5%), dysfunction or positivity for thyroid antibodies (19.1%), symptomatic or visible nodules (17.6%), and family history of any thyroid disease (8.2%). Gland texture was not reported in 41.9%. The depth of the lobes was the dimension reported most frequently (42.2%), but any diameter was not reported in 57.8% of the cases. As regards the description of the most relevant nodule, length was reported more frequently (75.9%). Margins and echogenicity were more frequently described (54.5% and 44.3%, respectively) than other characteristics (composition: 27%; shape: 8.8%; echogenic foci: 6.7%). No reports had indicated the malignancy risk stratification.

CONCLUSIONS

The results of the study demonstrate that in patients undergoing endocrine consultation with first detected thyroid nodules, US was mostly performed in asymptomatic cases, US reports were incomplete, and no risk stratification system was reported.

Investigating the diagnostic efficiency of a computer-aided diagnosis system for thyroid nodules in the context of Hashimoto's thyroiditis

Objectives

This study aims to investigate the efficacy of a computer-aided diagnosis (CAD) system in distinguishing between benign and malignant thyroid nodules in the context of Hashimoto's thyroiditis (HT) and to evaluate the role of the CAD system in reducing unnecessary biopsies of benign lesions.

Methods

We included a total of 137 nodules from 137 consecutive patients (mean age, 43.5 ± 11.8 years) who were histopathologically diagnosed with HT. The two-dimensional ultrasound images and videos of all thyroid nodules were analyzed by the CAD system and two radiologists with different experiences according to ACR TI-RADS. The diagnostic cutoff values of ACR TI-RADS were divided into two categories (TR4 and TR5), and then the sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) of the CAD system and the junior and senior radiologists were compared in both cases. Moreover, ACR TI-RADS classification was revised according to the results of the CAD system, and the efficacy of recommended fine-needle aspiration (FNA) was evaluated by comparing the unnecessary biopsy rate and the malignant rate of punctured nodules.

Results

The accuracy, sensitivity, specificity, PPV, and NPV of the CAD system were 0.876, 0.905, 0.830, 0.894, and 0.846, respectively. With TR4 as the cutoff value, the AUCs of the CAD system and the junior and senior radiologists were 0.867, 0.628, and 0.722, respectively, and the CAD system had the highest AUC (P < 0.0001). With TR5 as the cutoff value, the AUCs of the CAD system and the junior and senior radiologists were 0.867, 0.654, and 0.812, respectively, and the CAD system had a higher AUC than the junior radiologist (P < 0.0001) but comparable to the senior radiologist (P = 0.0709). With the assistance of the CAD system, the number of TR4 nodules was decreased by both junior and senior radiologists, the malignant rate of punctured nodules increased by 30% and 22%, and the unnecessary biopsies of benign lesions were both reduced by nearly half.

Conclusions

The CAD system based on deep learning can improve the diagnostic performance of radiologists in identifying benign and malignant thyroid nodules in the context of Hashimoto's thyroiditis and can play a role in FNA recommendations to reduce unnecessary biopsy rates.

Machine Learning-Aided Chronic Kidney Disease Diagnosis Based on Ultrasound Imaging Integrated with Computer-Extracted Measurable Features

Although ultrasound plays an important role in the diagnosis of chronic kidney disease (CKD), image interpretation requires extensive training. High operator variability and limited image quality control of ultrasound images have made the application of computer-aided diagnosis (CAD) challenging. This study assessed the effect of integrating computer-extracted measurable features with the convolutional neural network (CNN) on the ultrasound image CAD accuracy of CKD. Ultrasound images from patients who visited Severance Hospital and Gangnam Severance Hospital in South Korea between 2011 and 2018 were used. A Mask regional CNN model was used for organ segmentation and measurable feature extraction. Data on kidney length and kidney-to-liver echogenicity ratio were extracted. The ResNet18 model classified kidney ultrasound images into CKD and non-CKD. Experiments were conducted with and without the input of the measurable feature data. The performance of each model was evaluated using the area under the receiver operating characteristic curve (AUROC). A total of 909 patients (mean age, 51.4 ± 19.3 years; 414 [49.5%] men and 495 [54.5%] women) were included in the study. The average AUROC from the model trained using ultrasound images achieved a level of 0.81. Image training with the integration of automatically extracted kidney length and echogenicity features revealed an improved average AUROC of 0.88. This value further increased to 0.91 when the clinical information of underlying diabetes was also included in the model trained with CNN and measurable features. The automated step-wise machine learning-aided model segmented, measured, and classified the kidney ultrasound images with high performance. The integration of computer-extracted measurable features into the machine learning model may improve CKD classification.

Ultrasonic S-Detect mode for the evaluation of thyroid nodules: A meta-analysis

OBJECTIVES

This meta-analysis aimed to evaluate the value of ultrasonic S-Detect mode for the evaluation of thyroid nodules.

METHODS

We searched PubMed, Cochrane Library, and Chinese biomedical databases from inception to August 31, 2021. Meta-analysis was conducted using STATA version 14.0 and Meta-Disc version 1.4 software. We calculated the summary statistics for sensitivity (Sen), specificity (Spe), summary receiver operating characteristic curve, and the area under the curve, and compared the area under the curve between ultrasonic S-Detect mode and thyroid imaging report and data system (TI-RADS) for the diagnosis of thyroid nodules. As a systematic review summarizing the results of previous studies, this study does not need the informed consent of patients or the approval of the ethics review committee.

RESULTS

Fifteen studies that met all inclusion criteria were included in this meta-analysis. A total of 924 thyroid malignant nodules and 1228 thyroid benign nodules were assessed. All thyroid nodules were histologically confirmed after examination. The pooled Sen and Spe of TI-RADS were 0.89 (95% confidence interval [CI] = 0.85-0.91) and 0.85 (95% CI = 0.78-0.90), respectively; the pooled Sen and Spe of S-Detect were 0.88 (95% CI = 0.85-0.90) and 0.73 (95% CI = 0.63-0.81), respectively. The areas under the summary receiver operating characteristic curve of TI-RADS and S-Detect were 0.9370 (standard error [SE] = 0.0110) and 0.9128 (SE = 0.0147), respectively, between which there was no significant difference (Z = 1.318; SE = 0.0184; P = .1875). We found no evidence of publication bias (t = 0.36, P = .72).

CONCLUSIONS

Our meta-analysis indicates that ultrasonic S-Detect mode may have high diagnostic accuracy and may have certain clinical application value, especially for young doctors.

Diagnostic accuracy of S-Detect in distinguishing benign and malignant thyroid nodules: A meta-analysis

Objectives

In this meta-analysis study, the main objective was to determine the accuracy of S-detect in effectively distinguishing malignant thyroid nodules from benign thyroid nodules.

Methods

We searched the PubMed, Cochrane Library, and CBM databases from inception to August 1, 2021. Meta-analysis was conducted using STATA version 14.0 and Meta-Disc version 1.4 softwares. We calculated summary statistics for sensitivity (Sen), specificity (Spe), positive and negative likelihood ratio (LR⁺/LR⁻), diagnostic odds ratio(DOR), and receiver operating characteristic (SROC) curves. Cochran’s Q-statistic and I² test were used to evaluate potential heterogeneity between studies. A sensitivity analysis was performed to evaluate the influence of single studies on the overall estimate. We also performed meta-regression analyses to investigate the potential sources of heterogeneity.

Results

In this study, a total of 17 studies meeting the requirements of the standard were used. The number of benign and malignant nodules analyzed and evaluated in this paper was 1595 and 1118 respectively. This paper mainly completes the required histological confirmation through s-detect. The pooled Sen and pooled Spe were 0.87 and 0.74, respectively, (95%CI = 0.84–0.89) and (95%CI = 0.66–0.81). Furthermore, the pooled LR⁺ and negative LR⁻ were determined to be 3.37 (95%CI = 2.53–4.50) and 0.18 (95%CI = 0.15–0.21), respectively. The experimental results showed that the pooled DOR of thyroid nodules was 18.83 (95% CI = 13.21–26.84). In addition, area under SROC curve was determined to be 0.89 (SE = 0.0124). It should be pointed out that there is no evidence of bias (i.e. t = 0.25, P = 0.80).

Conclusions

Through this meta-analysis, it can be seen that the accuracy of s-detect is relatively high for the effective distinction between malignant thyroid nodules and benign thyroid nodules.

[Value of ultrasonic S-Detect technique in diagnosis of breast masses]

OBJECTIVE

To evaluate the value of ultrasound S-Detect in the diagnosis of breast masses.

METHODS

A total of 85 breast masses in 62 female patients were diagnosed by S-Detect technique and conventional ultrasound. The diagnostic efficacy of conventional ultrasound and S-Detect technique was analyzed and compared with postoperative pathological results as the gold standard.

RESULTS

When operated by junior physicians, the diagnostic efficacy of conventional ultrasound was significantly lower than that of S-Detect technique (P < 0.05), but this difference was not observed in moderately experienced and senior physicians (P>0.05). S-Detect technique was positively correlated with the diagnostic results of senior physicians (r=0.97). Using S-Detect technique, the diagnostic efficacy did not differ significantly between the long axis section and its vertical section (P>0.05). Routine ultrasound showed a better diagnostic efficacy than S-Detect for breast masses with a diameter below 20 mm (P < 0.05), but for larger breast masses, its diagnostic efficacy was significantly lower than that of SDetect (P < 0.05).

CONCLUSION

S-Detect can be used in differential diagnosis of benign and malignant breast masses, and its diagnostic efficiency can be comparable with that of BI-RADS classification for moderately experienced and senior physicians, but its diagnostic efficacy can be low for breast masses less than 20 mm in diameter.

A comparison of artificial intelligence versus radiologists in the diagnosis of thyroid nodules using ultrasonography: a systematic review and meta-analysis

BACKGROUND

Thyroid nodules are common. Ultrasonography (US) is the first investigation for thyroid nodules. Artificial Intelligence (AI) is widely integrated into medical diagnosis to provide additional information. The primary objective of this study was to accumulate the pooled sensitivity and specificity between all available AI and radiologists using thyroid US imaging. The secondary objective was to compare AI's diagnostic performance to that of radiologists.

MATERIALS AND METHODS

A systematic review meta-analysis. PubMed, Scopus, Web of Science, and Cochrane Library data were searched for studies from inception until June 11, 2020.

RESULTS

Twenty five studies were included in this meta-analysis. The pooled sensitivity and specificity of AI were 0.86 (95% CI 0.81-0.91) and 0.78 (95% CI 0.73-0.83), respectively. The pooled sensitivity and specificity of radiologists were 0.85 (95% CI 0.80-0.89) and 0.82 (95% CI 0.77-0.86), respectively. The accuracy of AI and radiologists is equivalent in terms of AUC [AI 0.89 (95% CI 0.86-0.92), radiologist 0.91 (95% CI 0.88-0.93)]. The diagnostic odd ratio (DOR) between AI 23.10 (95% CI 14.20-37.58) and radiologists 27.12 (95% CI 17.45-42.16) had no statistically significant difference (P = 0.56). Meta-regression analysis revealed that Deep Learning AI had significantly greater sensitivity and specificity than classic machine learning AI (P < 0.001).

CONCLUSION

AI demonstrated comparable performance to radiologists in diagnosing benign and malignant thyroid nodules using ultrasonography. Additional research to establish its equivalency should be conducted.

Artificial Intelligence (AI) Tools for Thyroid Nodules on Ultrasound, From the AJR Special Series on AI Applications

Artificial intelligence (AI) methods for evaluating thyroid nodules on ultrasound have been widely described in the literature, with reported performance of AI tools matching or in some instances surpassing radiologists. As these data have accumulated, products for classification and risk stratification of thyroid nodules on ultrasound have become commercially available. This article reviews FDA-approved products currently on the market, with a focus on product features, reported performance, and considerations for implementation. The products perform risk stratification primarily using the Thyroid Imaging Reporting and Data System (TI-RADS), though may provide additional prediction tools independent of TI-RADS. Key issues in implementation include integration with radiologist interpretation, impact on workflow and efficiency, and performance monitoring. AI applications beyond nodule classification, including report construction and incidental findings follow-up, are also described. Anticipated future directions of research and development in AI tools for thyroid nodules are highlighted.

Reliability of a computer-aided system in the evaluation of indeterminate ultrasound images of thyroid nodules

INTRODUCTION

Computer-aided diagnostic (CAD) programs for malignancy risk stratification from ultrasound (US) imaging of thyroid nodules are being validated both experimentally and in real-world practice. However, they have not been tested for reliability in analyzing difficult or unclear images.

METHODS

US images with indeterminate characteristics were evaluated by five observers with different experience in US examination and by a commercial CAD program. The nodules, on which the observers widely agreed, were considered concordant and, if there was little agreement, not concordant or difficult to assess. The diagnostic performance of the readers and the CAD program was calculated and compared in both groups of nodule images.

RESULTS

In the group of concordant thyroid nodules (n = 37), the clinicians and the CAD system obtained similar levels of accuracy (77.0% vs 74.2%, respectively; P = 0.7) and no differences were found in sensitivity (SEN) (95.0% vs 87.5%, P = 0.2), specificity (SPE) (45.5 vs 49.4, respectively; P = 0.7), positive predictive value (PPV) (75.2% vs 77.7%, respectively; P = 0.8), nor negative predictive value (NPV) (85.6 vs 77.7, respectively; P = 0.3). When analyzing the non-concordant nodules (n = 43), the CAD system presented a decrease in accuracy of 4.2%, which was significantly lower than that observed by the experts (19.9%, P = 0.02).

CONCLUSIONS

Clinical observers are similar to the CAD system in the US assessment of the risk of thyroid nodules. However, the AI system for thyroid nodules AmCAD-UT® showed more reliability in the analysis of unclear or misleading images.

Diagnosing thyroid nodules with atypia of undetermined significance/follicular lesion of undetermined significance cytology with the deep convolutional neural network

To compare the diagnostic performances of physicians and a deep convolutional neural network (CNN) predicting malignancy with ultrasonography images of thyroid nodules with atypia of undetermined significance (AUS)/follicular lesion of undetermined significance (FLUS) results on fine-needle aspiration (FNA). This study included 202 patients with 202 nodules ≥ 1 cm AUS/FLUS on FNA, and underwent surgery in one of 3 different institutions. Diagnostic performances were compared between 8 physicians (4 radiologists, 4 endocrinologists) with varying experience levels and CNN, and AUS/FLUS subgroups were analyzed. Interobserver variability was assessed among the 8 physicians. Of the 202 nodules, 158 were AUS, and 44 were FLUS; 86 were benign, and 116 were malignant. The area under the curves (AUCs) of the 8 physicians and CNN were 0.680–0.722 and 0.666, without significant differences (P > 0.05). In the subgroup analysis, the AUCs for the 8 physicians and CNN were 0.657–0.768 and 0.652 for AUS, 0.469–0.674 and 0.622 for FLUS. Interobserver agreements were moderate (k = 0.543), substantial (k = 0.652), and moderate (k = 0.455) among the 8 physicians, 4 radiologists, and 4 endocrinologists. For thyroid nodules with AUS/FLUS cytology, the diagnostic performance of CNN to differentiate malignancy with US images was comparable to that of physicians with variable experience levels.

Applications of machine learning and deep learning to thyroid imaging: where do we stand?

Ultrasonography (US) is the primary diagnostic tool used to assess the risk of malignancy and to inform decision-making regarding the use of fine-needle aspiration (FNA) and postFNA management in patients with thyroid nodules. However, since US image interpretation is operator-dependent and interobserver variability is moderate to substantial, unnecessary FNA and/or diagnostic surgery are common in practice. Artificial intelligence (AI)-based computeraided diagnosis (CAD) systems have been introduced to help with the accurate and consistent interpretation of US features, ultimately leading to a decrease in unnecessary FNA. This review provides a developmental overview of the AI-based CAD systems currently used for thyroid nodules and describes the future developmental directions of these systems for the personalized and optimized management of thyroid nodules.

Use of artificial intelligence and machine learning for estimating malignancy risk of thyroid nodules

PURPOSE OF REVIEW

Current methods for thyroid nodule risk stratification are subjective, and artificial intelligence algorithms have been used to overcome this shortcoming. In this review, we summarize recent developments in the application of artificial intelligence algorithms for estimating the risks of malignancy in a thyroid nodule.

RECENT FINDINGS

Artificial intelligence have been used to predict malignancy in thyroid nodules using ultrasound images, cytopathology images, and molecular markers. Recent clinical trials have shown that artificial intelligence model's performance matched that of experienced radiologists and pathologists. Explainable artificial intelligence models are being developed to avoid the black box problem. Risk stratification algorithms using artificial intelligence for thyroid nodules are now commercially available in many countries.

SUMMARY

Artificial intelligence models could become a useful tool in a thyroidolgist's armamentarium as a decision support tool. Increased adoption of this emerging technology will depend upon increased awareness of the potential benefits and pitfalls in using artificial intelligence.

A Clinical Assessment of an Ultrasound Computer-Aided Diagnosis System in Differentiating Thyroid Nodules With Radiologists of Different Diagnostic Experience

Introduction

This study aimed to assess the diagnostic performance and the added value to radiologists of different levels of a computer-aided diagnosis (CAD) system for the detection of thyroid cancers.

Methods

303 patients who underwent thyroidectomy from October 2018 to July 2019 were retrospectively reviewed. The diagnostic performance of the senior radiologist, the junior radiologist, and the CAD system were compared. The added value of the CAD system was assessed and subgroup analyses were performed according to the size of thyroid nodules.

Results

In total, 186 malignant thyroid nodules, and 179 benign thyroid nodules were included; 168 were papillary thyroid carcinoma (PTC), 7 were medullary thyroid carcinoma (MTC), 11 were follicular carcinoma (FTC), 127 were follicular adenoma (FA) and 52 were nodular goiters. The CAD system showed a comparable specificity as the senior radiologist (86.0% vs. 86.0%, p > 0.99), but a lower sensitivity and a lower area under the receiver operating characteristic (AUROC) curve (sensitivity: 71.5% vs. 95.2%, p < 0.001; AUROC: 0.788 vs. 0.906, p < 0.001). The CAD system improved the diagnostic sensitivities of both the senior and the junior radiologists (97.8% vs. 95.2%, p = 0.063; 88.2% vs. 75.3%, p < 0.001).

Conclusion

The use of the CAD system using artificial intelligence is a potential tool to distinguish malignant thyroid nodules and is preferable to serve as a second opinion for less experienced radiologists to improve their diagnosis performance.

S-Detect Software vs. EU-TIRADS Classification: A Dual-Center Validation of Diagnostic Performance in Differentiation of Thyroid Nodules

Computer-aided diagnosis (CAD) and other risk stratification systems may improve ultrasound image interpretation. This prospective study aimed to compare the diagnostic performance of CAD and the European Thyroid Imaging Reporting and Data System (EU-TIRADS) classification applied by physicians with S-Detect 2 software CAD based on Korean Thyroid Imaging Reporting and Data System (K-TIRADS) and combinations of both methods (MODELs 1 to 5). In all, 133 nodules from 88 patients referred to thyroidectomy with available histopathology or with unambiguous results of cytology were included. The S-Detect system, EU-TIRADS, and mixed MODELs 1–5 for the diagnosis of thyroid cancer showed a sensitivity of 89.4%, 90.9%, 84.9%, 95.5%, 93.9%, 78.9% and 93.9%; a specificity of 80.6%, 61.2%, 88.1%, 53.7%, 73.1%, 89.6% and 80.6%; a positive predictive value of 81.9%, 69.8%, 87.5%, 67%, 77.5%, 88.1% and 82.7%; a negative predictive value of 88.5%, 87.2%, 85.5%, 92.3%, 92.5%, 81.1% and 93.1%; and an accuracy of 85%, 75.9%, 86.5%, 74.4%, 83.5%, 84.2%, and 87.2%, respectively. Comparison showed superiority of the similar MODELs 1 and 5 over other mixed models as well as EU-TIRADS and S-Detect used alone (p-value < 0.05). S-Detect software is characterized with high sensitivity and good specificity, whereas EU-TIRADS has high sensitivity, but rather low specificity. The best diagnostic performance in malignant thyroid nodule (TN) risk stratification was obtained for the combined model of S-Detect (“possibly malignant” nodule) and simultaneously obtaining 4 or 5 points (MODEL 1) or exactly 5 points (MODEL 5) on the EU-TIRADS scale.