Variability in the interpretation of grey-scale ultrasound features in assessing thyroid nodules: A systematic review and meta-analysis

The value of ACR, European, Korean, and ATA ultrasound risk stratification systems combined with RAS mutations for detecting thyroid carcinoma in cytologically indeterminate and suspicious for malignancy thyroid nodules

PURPOSE

The aim of this study was to evaluate the diagnostic value of four commonly utilized ultrasound (US) RSSs, namely, the American College of Radiology [ACR], European [EU], Korean [K] TI-RADSs and American Thyroid Association [ATA] US-based RSS criteria, in combination with activating point mutations of the RAS genes (NRAS, HRAS, and KRAS) for detection of thyroid carcinoma in cytologically indeterminate and suspicious for malignancy thyroid nodules.

METHODS

We retrospectively analyzed cytologically indeterminate and suspicious for malignancy thyroid nodules which underwent US, molecular testing and surgery between September 1, 2018, and December 31, 2023. Receiver operating characteristic (ROC) curves were generated, and the area under the curve (AUC, 95% confidence interval [CI]) was calculated.

RESULTS

A total of 100 cytologically indeterminate and 24 suspicious for malignancy thyroid nodules were analyzed. Compared to the four US-based RSSs alone, the diagnostic value of the four US-based RSSs combined with RAS mutations did not significantly improved (cytologically indeterminate, AUC [95% CI] 0.6 [0.5-0.7] and 0.6 [0.5-0.7], respectively, p = 0.70; cytologically suspicious for malignancy, AUC [95% CI] 0.7 [0.5-0.9] and 0.8 [0.6-0.9], respectively, p = 0.23).

CONCLUSIONS

The diagnostic value of the four main US-based RSSs (ACR, EU, K, and ATA) was not improved in conjunction with the evaluation of RAS mutations for preoperative risk stratification of cytologically indeterminate thyroid nodules.

CLINICAL RELEVANCE STATEMENT

In cytologically indeterminate nodules categorized according to US-based RSSs, isolated RAS positivity does not reliably distinguish between benignity and malignancy.

Implementation strategies to support ultrasound thyroid nodule risk stratification: A systematic review

BACKGROUND

Ultrasound risk stratification can improve the care of patients with thyroid nodules by providing a structured and systematic approach for the evaluation of thyroid nodule features and thyroid cancer risk. The optimal strategies to support implementation of high quality thyroid nodule risk stratification are unknown. This study seeks to summarise strategies used to support implementation of thyroid nodule ultrasound risk stratification in practice and their effects on implementation and service outcomes.

METHODS

This is a systematic review of studies evaluating implementation strategies published between January 2000 and June 2022 that were identified on Ovid MEDLINE, Ovid EMBASE, Ovid Cochrane, Scopus, or Web of Science. Screening of eligible studies, data collection and assessment for risk of bias was completed independently and in duplicate. Implementation strategies and their effects on implementation and service outcomes were evaluated and summarised.

RESULTS

We identified 2666 potentially eligible studies of which 8 were included. Most implementation strategies were directed towards radiologists. Common strategies to support the implementation of thyroid nodule risk stratification included: tools to standardise thyroid ultrasound reports, education on thyroid nodule risk stratification and use of templates/forms for reporting, and reminders at the point of care. System based strategies, local consensus or audit were less commonly described. Overall, the use of these strategies supported the implementation process of thyroid nodule risk stratification with variable effects on service outcomes.

CONCLUSIONS

Implementation of thyroid nodule risk stratification can be supported by development of standardised reporting templates, education of users on risk stratification and reminders of use at the point of care. Additional studies evaluating the value of implementation strategies in different contexts are urgently needed.

Real-Time Ultrasonography and the Evaluation of Static Images Yield Different Results in the Assessment of EU-TIRADS Categories

The studies on the effectiveness of various TIRADS in the diagnostics of thyroid nodules differ in the method of ultrasound image assessment: real time (rtUS) vs. static ultrasonography (stUS). The aim of the study was to evaluate the impact of those two methods on the categorization of nodules in EU-TIRADS. Three experienced raters assessed 842 nodules in routine rtUS and reassessed with the use of sUS. Reproducibility of the assessment of malignancy risk features and categorization of nodules with EU-TIRADS was estimated with Krippendorff's alpha coefficient (Kα). The reproducibility of EU-TIRADS categories on sUS in relation to rtUS was in range 70.9-76.5% for all raters (Kα: 0.60-0.68) with the highest reproducibility for category 3 (80.0-86.5%) and the lowest for category 5 (48.7-77.8%). There was a total disagreement of the identification of microcalcifications on sUS in relation to rtUS, a strongly variable reproducibility of marked hypoechogenicity (12.5-84.6%, Kα: 0.14-0.48) and a tendency toward more frequent identification of the non-oval shape on sUS. The percentage of agreement for each pair of raters in assigning the EU-TIRADS category on sUS was in the range 71.6-72.3% (Kα: 0.60-0.62). The method of sonographic image evaluation influences the nodule's feature assessment and, eventually, the categorization within EU-TIRADS.

Considerable interobserver variation calls for unambiguous definitions of thyroid nodule ultrasound characteristics

Objective

Thyroid nodule ultrasound characteristics are used as an indication for fine-needle aspiration cytology, usually as the basis for Thyroid Imaging Reporting and Data System (TIRADS) score calculation. Few studies on interobserver variation are available, all of which are based on analysis of preselected still ultrasound images and often lack surgical confirmation.

Methods

After the blinded online evaluation of video recordings of the ultrasound examinations of 47 consecutive malignant and 76 consecutive benign thyroid lesions, 7 experts from 7 thyroid centers answered 17 TIRADS-related questions. Surgical histology was the reference standard. Interobserver variations of each ultrasound characteristic were compared using Gwet's AC1 inter-rater coefficients; higher values mean better concordance, the maximum being 1.0.

Results

On a scale from 0.0 to 1.0, the Gwet's AC1 values were 0.34, 0.53, 0.72, and 0.79 for the four most important features in decision-making, i.e. irregular margins, microcalcifications, echogenicity, and extrathyroidal extension, respectively. The concordance in the discrimination between mildly/moderately and very hypoechogenic nodules was 0.17. The smaller the nodule size the better the agreement in echogenicity, and the larger the nodule size the better the agreement on the presence of microcalcifications. Extrathyroidal extension was correctly identified in just 45.8% of the cases.

Conclusions

Examination of video recordings, closely simulating the real-world situation, revealed substantial interobserver variation in the interpretation of each of the four most important ultrasound characteristics. In view of the importance for the management of thyroid nodules, unambiguous and widely accepted definitions of each nodule characteristic are warranted, although it remains to be investigated whether this diminishes observer variation.

Deep learning for the diagnosis of suspicious thyroid nodules based on multimodal ultrasound images

OBJECTIVES

This study aimed to differentially diagnose thyroid nodules (TNs) of Thyroid Imaging Reporting and Data System (TI-RADS) 3-5 categories using a deep learning (DL) model based on multimodal ultrasound (US) images and explore its auxiliary role for radiologists with varying degrees of experience.

METHODS

Preoperative multimodal US images of 1,138 TNs of TI-RADS 3-5 categories were randomly divided into a training set (n = 728), a validation set (n = 182), and a test set (n = 228) in a 4:1:1.25 ratio. Grayscale US (GSU), color Doppler flow imaging (CDFI), strain elastography (SE), and region of interest mask (Mask) images were acquired in both transverse and longitudinal sections, all of which were confirmed by pathology. In this study, fivefold cross-validation was used to evaluate the performance of the proposed DL model. The diagnostic performance of the mature DL model and radiologists in the test set was compared, and whether DL could assist radiologists in improving diagnostic performance was verified. Specificity, sensitivity, accuracy, positive predictive value, negative predictive value, and area under the receiver operating characteristics curves (AUC) were obtained.

RESULTS

The AUCs of DL in the differentiation of TNs were 0.858 based on (GSU + SE), 0.909 based on (GSU + CDFI), 0.906 based on (GSU + CDFI + SE), and 0.881 based (GSU + Mask), which were superior to that of 0.825-based single GSU (p = 0.014, p< 0.001, p< 0.001, and p = 0.002, respectively). The highest AUC of 0.928 was achieved by DL based on (G + C + E + M)US, the highest specificity of 89.5% was achieved by (G + C + E)US, and the highest accuracy of 86.2% and sensitivity of 86.9% were achieved by DL based on (G + C + M)US. With DL assistance, the AUC of junior radiologists increased from 0.720 to 0.796 (p< 0.001), which was slightly higher than that of senior radiologists without DL assistance (0.796 vs. 0.794, p > 0.05). Senior radiologists with DL assistance exhibited higher accuracy and comparable AUC than that of DL based on GSU (83.4% vs. 78.9%, p = 0.041; 0.822 vs. 0.825, p = 0.512). However, the AUC of DL based on multimodal US images was significantly higher than that based on visual diagnosis by radiologists (p< 0.05).

CONCLUSION

The DL models based on multimodal US images showed exceptional performance in the differential diagnosis of suspicious TNs, effectively increased the diagnostic efficacy of TN evaluations by junior radiologists, and provided an objective assessment for the clinical and surgical management phases that follow.

Ultrasound gray scale ratio for differential diagnosis of papillary thyroid microcarcinoma from benign micronodule in patients with Hashimoto's thyroiditis

BACKGROUND

To investigate the diagnostic value of ultrasound gray scale ratio (UGSR) in differentiating papillary thyroid microcarcinomas (PTMCs) from benign micronodules (BMNs) in patients with Hashimoto's thyroiditis (HT).

METHODS

The ultrasound images of 285 PTMCs (from 247 patients) and 173 BMNs (from 140 patients) in the HT group, as well as 461 PTMCs (from 417 patients) and 234 BMNs (from 197 patients) in the non-HT group were retrospectively analyzed. The diagnosis of all cases was confirmed by histopathological examinations. The gray scale values of the nodules and surrounding thyroid tissues were measured and subsequently the UGSRs were calculated. Receiver operating characteristic curve analysis was used to determine the area under the curve (AUC), optimal UGSR threshold, sensitivity and specificity in differentiating PTMCs and BMNs in the two groups.

RESULTS

The UGSR of PTMC and BMN was 0.52 ± 0.12 and 0.85 ± 0.24 in the HT group (P < 0.001), and 0.57 ± 0.13 and 0.87 ± 0.20 in the non-HT group (P < 0.001), respectively. The difference in PTMC-UGSR was significant between the two groups (P < 0.001), whereas BMN-UGSR did not differ between the two groups (P = 0.416). The AUC, optimal UGSR threshold, sensitivity and specificity of UGSR for differentiating PTMC and BMN in the HT and non-HT group were 0.890 versus 0.901, 0.68 versus 0.72, 91.23% versus 90.67%, and 77.46% versus 82.05%, respectively.

CONCLUSIONS

The USGR of the HT group was lower than that of the non-HT group. Moreover, UGSR exhibited important diagnostic value in differentiating PTMC from BMN in both HT and non-HT groups.

Clinician Agreement on the Classification of Thyroid Nodules Ultrasound Features: A Survey of 2 Endocrine Societies

CONTEXT

Thyroid nodule risk stratification allows clinicians to standardize the evaluation of thyroid cancer risk according to ultrasound features.

OBJECTIVE

To evaluate interrater agreement among clinicians assessing thyroid nodules ultrasound features and thyroid cancer risk categories.

DESIGN, SETTING, AND PARTICIPANTS

We surveyed Endocrine Society and Latin American Thyroid Society members to assess their interpretation of composition, echogenicity, shape, margins, and presence of echogenic foci of 10 thyroid nodule cases. The risk category for thyroid cancer was calculated following the American College of Radiology-Thyroid Imaging Reporting & Data System (ACR-TIRADS) framework from individual responses.

MAIN OUTCOMES AND MEASURES

We used descriptive statistics and Gwet's agreement coefficient (AC1) to assess the primary outcome of interrater agreement for ACR-TIRADS risk category. As secondary outcomes, the interrater agreement for individual features and a subgroup analysis of interrater agreement for the ACR-TIRADS category were performed (ultrasound reporting system, type of practice, and number of monthly appraisals).

RESULTS

A total of 144 participants were included, mostly endocrinologists. There was moderate level of agreement for the absence of echogenic foci (AC1 0.53, 95% CI 0.24-0.81) and composition (AC1 0.54, 95% CI 0.36-0.71). The agreement for margins (AC1 0.24, 95% CI 0.15-0.33), echogenicity (AC1 0.34, 95% CI 0.22-0.46), and shape assessment (AC1 0.42, 95% CI 0.13-0.70) was lower. The overall agreement for ACR-TIRADS assessment was AC1 0.29, (95% CI 0.13-0.45). The AC1 of ACR-TIRADS among subgroups was similar.

CONCLUSIONS

This study found high variation of judgments about ACR-TIRADS risk category and individual features, which poses a potential challenge for the widescale implementation of thyroid nodule risk stratification.

The Future of Thyroid Nodule Risk Stratification

Clinical evidence supports the association of ultrasound features with benign or malignant thyroid nodules and serves as the basis for sonographic stratification of thyroid nodules, according to an estimated thyroid cancer risk. Contemporary guidelines recommend management strategies according to thyroid cancer risk, thyroid nodule size, and the clinical scenario. Yet, reproducible and accurate thyroid nodule risk stratification requires expertise, time, and understanding of the weight different ultrasound features have on thyroid cancer risk. The application of artificial intelligence to overcome these limitations is promising and has the potential to improve the care of patients with thyroid nodules.

Systematic Review and Meta-Analysis of American College of Radiology TI-RADS Inter-Reader Reliability for Risk Stratification of Thyroid Nodules

Purpose

To investigate the inter-reader agreement of using the American College of Radiology (ACR) Thyroid Imaging Reporting and Data System (TI-RADS) for risk stratification of thyroid nodules.

Methods

A literature search of Web of Science, PubMed, Cochrane Library, EMBASE, and Google Scholar was performed to identify eligible articles published from inception until October 31, 2021. We included studies reporting inter-reader agreement of different radiologists who applied ACR TI-RADS for the classification of thyroid nodules. Quality assessment of the included studies was performed with the Quality Assessment of Diagnostic Accuracy Studies-2 tool and Guidelines for Reporting Reliability and Agreement Studies. The summary estimates of the inter-reader agreement were pooled with the random-effects model, and multiple subgroup analyses and meta-regression were performed to investigate various clinical settings.

Results

A total of 13 studies comprising 5,238 nodules were included in the current meta-analysis and systematic review. The pooled inter-reader agreement for overall ACR TI-RADS classification was moderate (κ = 0.51, 95% CI 0.42–0.59). Substantial heterogeneity was presented throughout the studies, and meta-regression analyses suggested that the malignant rate was the significant factor. Regarding the ultrasound (US) features, the best inter-reader agreement was composition (κ = 0.58, 95% CI 0.53–0.63), followed by shape (κ = 0.57, 95% CI 0.41–0.72), echogenicity (κ = 0.50, 95% CI 0.40–0.60), echogenic foci (κ = 0.44, 95% CI 0.36–0.53), and margin (κ = 0.34, 95% CI 0.24–0.44).

Conclusions

The ACR TI-RADS demonstrated moderate inter-reader agreement between radiologists for the overall classification. However, the US feature of margin only showed fair inter-reader reliability among different observers.

A Control Study on the Value of the Ultrasound Grayscale Ratio for the Differential Diagnosis of Thyroid Micropapillary Carcinoma and Micronodular Goiter in Two Medical Centers

Objective

To investigate the value of ultrasound gray-scale ratio (UGSR) for the differential diagnosis of papillary thyroid microcarcinoma (PTMC) and micronodular goiter (MNG) in two medical centers.

Methods

Ultrasound images of 881 PTMCs from 785 patients and 744 MNGs from 687 patients in center A were retrospectively analyzed and compared with 243 PTMCs from 203 patients and 251 MNGs from 198 patients in center B. All cases were confirmed by surgery and histology. The grayscale values of thyroid lesions and surrounding normal tissues were measured, and the UGSR was calculated. The optimal UGSR threshold for identifying PTMCs and MNGs in two medical centers was determined by receiver operating characteristic (ROC) curve, and the area under the curve (AUC), optimal UGSR threshold, sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were compared between the two medical centers.

Results

The UGSR values of PTMCs and MNGs in medical center A were 0.5537 (0.4699, 0.6515) and 0.8708 (0.7616, 1.0123) (Z = -27.691, P = 0), respectively, whereas those in medical center B were 0.5517 (0.4698, 0.6377) and 0.8539 (0.7366, 0.9929) (Z = -16.057, P = 0), respectively. The UGSR of PTMCs and MNGs did not differ significantly between the two medical centers (Z = -0.609, P = 0.543 and Z = -1.394, P = 0.163, respectively). The AUC, optimal UGSR threshold, sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of the two medical centers were 0.898 vs. 0.918, 0.7214 vs. 0.6911, 0.881 vs. 0.868, 0.817 vs. 0.833, 0.851 vs. 0.834, 0.853 vs. 0.867, and 0.852 vs. 0.850, respectively.

Conclusions

UGSR can quantify the echo intensity of PTMCs and MNGs and is therefore valuable for the differential diagnosis of the two diseases. The diagnostic efficacy was consistent between the two medical centers. This method should be widely promoted and applied.