Diagnostic performance of 2015 American Thyroid Association guidelines and inter-observer variability in assigning risk category

Cost-Effectiveness of the ACR TIRADS Compared to the ATA 2015 Risk Stratification Systems in the Evaluation of Incidental Thyroid Nodules

RATIONALE AND OBJECTIVES

Thyroid nodules are a common incidental imaging finding and prone to overdiagnosis. Several risk stratification systems have been developed to reduce unnecessary work-up, with two of the most utilized including the American Thyroid Association 2015 (ATA2015) and the newer American College of Radiology Thyroid Imaging, Reporting and Data System (TIRADS) guidelines. The purpose of this study is to evaluate the cost-effectiveness of the ATA2015 versus the TIRADS guidelines in the management of incidental thyroid nodules.

METHODS

A cost-utility analysis was conducted using decision tree modeling, evaluating adult patients with incidental thyroid nodules < 4 cm. Model inputs were populated using published literature, observational data, and expert opinion. Single-payer perspective, Canadian dollar currency, five-year time horizon, willingness to pay (WTP) threshold of $50,000, and discount rate of 1.5% per annum were utilized. Scenario, deterministic and probabilistic sensitivity analyses were performed. The primary outcome was the incremental cost-effectiveness ratio (ICER) expressed as incremental cost per quality-adjusted life year (QALY) gained.

RESULTS

For the base case scenario, TIRADS dominated the ATA2015 strategy by a slim margin, producing 0.005 more QALYs at $25 less cost. Results were sensitive to the malignancy rate of biopsy and the utilities of a patient with a benign nodule/subclinical malignancy or under surveillance. Probabilistic sensitivity analysis showed that TIRADS was the more cost-effective option 79.7% of the time.

CONCLUSION

The TIRADS guidelines may be the more cost-effective strategy by a small margin compared to ATA2015 in most scenarios when used to risk stratify incidental thyroid nodules.

Malignancy Risk of Thyroid Nodules That Are Not Classifiable by the American Thyroid Association Ultrasound Risk Stratification System: A Systematic Review and Meta-Analysis

Background: Sonographic evaluation is fundamental to thyroid nodule assessment. The American Thyroid Association (ATA) ultrasound risk stratification system (USRSS) is widely used, but the appearance of some nodules has been considered nonclassifiable (NC-ATA). The risk of malignancy (RoM) of NC-ATA nodules varies widely between studies, leading to uncertainty in clinical management. The aim of this study was to comprehensively evaluate the prevalence and malignancy risk of NC-ATA nodules. Methods: A systematic review was performed searching PubMed/MEDLINE and EMBASE to identify original studies of thyroid nodules classified using the ATA USRSS from 2016 to 2022 and reporting the outcome of NC-ATA nodules. Meta-analysis was conducted to obtain pooled RoM estimates and meta-regression sensitivity analyses were used to explore sources of between-study heterogeneity. Results: Of 6377 screened studies, 135 underwent full-text review, and 16 studies reporting 21,271 nodules were included. Within these, the pooled prevalence of NC-ATA nodules was 7.8% (1872 nodules; [confidence interval; CI 5.1-11.1]). The pooled RoM estimate for NC-ATA nodules was 20.3% [CI 13.0-28.7] and there was significant heterogeneity between studies (I2 = 92.8%, p < 0.001). NC-ATA nodule RoM estimates were significantly different by study type: single-center versus multicenter studies (24.8% vs. 12.3%, respectively, p = 0.031) and study design: retrospective versus prospective studies (25.1% vs. 8.5%, respectively, p = 0.003). No significant difference was observed in RoM based on inclusion of <1 cm nodules or geographic region. Meta-regression analysis showed study design and use of surgical histology for diagnostic criteria contributed significantly to differences in the reported RoM estimates. Conclusion: In this first meta-analysis comprehensively assessing the RoM of NC-ATA nodules, the malignancy risk was found to be comparable with the current ATA USRSS intermediate suspicion category. Significant heterogeneity was observed between studies and limits the interpretation of these results. In future iterations of the ATA USRSS that seek into incorporate categorization of NC-ATA nodules, these meta-analysis data may help to inform proper malignancy risk stratification. The study protocol was registered on PROSPERO, the international prospective register of systematic reviews (CRD42020182498), on July 14, 2020.

Considerable interobserver variation calls for unambiguous definitions of thyroid nodule ultrasound characteristics

Objective

Thyroid nodule ultrasound characteristics are used as an indication for fine-needle aspiration cytology, usually as the basis for Thyroid Imaging Reporting and Data System (TIRADS) score calculation. Few studies on interobserver variation are available, all of which are based on analysis of preselected still ultrasound images and often lack surgical confirmation.

Methods

After the blinded online evaluation of video recordings of the ultrasound examinations of 47 consecutive malignant and 76 consecutive benign thyroid lesions, 7 experts from 7 thyroid centers answered 17 TIRADS-related questions. Surgical histology was the reference standard. Interobserver variations of each ultrasound characteristic were compared using Gwet's AC1 inter-rater coefficients; higher values mean better concordance, the maximum being 1.0.

Results

On a scale from 0.0 to 1.0, the Gwet's AC1 values were 0.34, 0.53, 0.72, and 0.79 for the four most important features in decision-making, i.e. irregular margins, microcalcifications, echogenicity, and extrathyroidal extension, respectively. The concordance in the discrimination between mildly/moderately and very hypoechogenic nodules was 0.17. The smaller the nodule size the better the agreement in echogenicity, and the larger the nodule size the better the agreement on the presence of microcalcifications. Extrathyroidal extension was correctly identified in just 45.8% of the cases.

Conclusions

Examination of video recordings, closely simulating the real-world situation, revealed substantial interobserver variation in the interpretation of each of the four most important ultrasound characteristics. In view of the importance for the management of thyroid nodules, unambiguous and widely accepted definitions of each nodule characteristic are warranted, although it remains to be investigated whether this diminishes observer variation.

Inter-Rater Reliability of Thyroid Ultrasound Risk Criteria: A Systematic Review and Meta-Analysis

OBJECTIVE

The most commonly employed diagnostic criteria for identifying thyroid nodules include Thyroid Imaging and Reporting Data System (TI-RADS) and American Thyroid Association (ATA) guidelines. The purpose of this systematic review and meta-analysis is to determine the inter-rater reliability of thyroid ultrasound criteria.

METHODS

We performed a library search of MEDLINE (Ovid), EMBASE (Ovid), and Web of Science for full-text articles published from January 2005 to June 2022. We included full-text primary research articles that used TI-RADS and/or ATA guidelines to evaluate thyroid nodules in adults. These included studies must have calculated inter-rater reliability using any validated metric. The Quality Appraisal for Reliability Studies (QAREL) was used to assess study quality. We planned for a random-effects meta-analysis, in addition to covariate and publication bias analyses. This study was performed in accordance with Preferred Reporting Items for a Systematic Review and Meta-analysis guidelines and registered prior to conduction (International prospective register of systematic reviews-PROSPERO: CRD42021275072).

RESULTS

Of the 951 articles identified via the database search, 35 met eligibility criteria. All studies were observational. The most commonly utilized criteria were ACR Thyroid Imaging and Reporting Data System (TI-RADS) and/or ATA criteria, while the majority of studies employed Κ statistics. For ACR TI-RADS, the pooled Κ was 0.51 (95% confidence interval [CI]: 0.42, 0.57; n = 7) while for ATA, the pooled Κ was 0.52 (95% CI: 0.37, 0.67; n = 3). Due to the small number of studies, covariate or publication bias analyses were not performed.

CONCLUSION

Ultrasound criteria demonstrate moderate inter-rater reliability, but these findings are impacted by poor study quality and a lack of standardization. Laryngoscope, 133:485-493, 2023.

Thyroid cytology: The reality before and after the introduction of ultrasound classification systems for thyroid nodules

BACKGROUND

Several ultrasound-based systems for classification of thyroid nodules are available. They allow for a better triage of the nodules that require cytological assessment, and lead to standardized recommendations. Our aim was to compare patients and nodules referred to fine-needle aspiration (FNA) before and after the introduction of these systems.

METHODS

A retrospective study comparing two cohorts of patients referred for FNA was performed (386 patients and 463 nodules in 2015; 220 patients and 263 nodules in 2021).

RESULTS

The sex distribution (89.1% vs 85.9% females, p=0.243), number of nodules referred to FNA per patient (median of 1), and the distribution of the Bethesda categories (p=0.082) was similar in both years. In 2021, patients were older (53.4±14.5 years vs 57.8±13.2 years, p<0.001) and nodules over one centimetre were larger (median 17.0mm vs 19.0mm, p=0.002), especially the ones categorized as Bethesda III (median size 11mm vs 23mm, p=0.043). In 2021, at least 23.1% of the nodules referred to FNA did not have any criteria, and 38.8% of the nodules were not categorized by any system.

CONCLUSION

This analysis draws attention to the importance of systematically applying ultrasound-based classification systems. It seems that, by not being focused mainly on size thresholds, they allow for longer surveillance periods, without aggravating the cytology results when FNA becomes indicated. Nevertheless, greater efforts are needed to ensure more standardized reports, and to increase adherence to the resulting recommendations to reduce clinical uncertainty, unnecessary FNA, and overtreatment.

Clinician Agreement on the Classification of Thyroid Nodules Ultrasound Features: A Survey of 2 Endocrine Societies

CONTEXT

Thyroid nodule risk stratification allows clinicians to standardize the evaluation of thyroid cancer risk according to ultrasound features.

OBJECTIVE

To evaluate interrater agreement among clinicians assessing thyroid nodules ultrasound features and thyroid cancer risk categories.

DESIGN, SETTING, AND PARTICIPANTS

We surveyed Endocrine Society and Latin American Thyroid Society members to assess their interpretation of composition, echogenicity, shape, margins, and presence of echogenic foci of 10 thyroid nodule cases. The risk category for thyroid cancer was calculated following the American College of Radiology-Thyroid Imaging Reporting & Data System (ACR-TIRADS) framework from individual responses.

MAIN OUTCOMES AND MEASURES

We used descriptive statistics and Gwet's agreement coefficient (AC1) to assess the primary outcome of interrater agreement for ACR-TIRADS risk category. As secondary outcomes, the interrater agreement for individual features and a subgroup analysis of interrater agreement for the ACR-TIRADS category were performed (ultrasound reporting system, type of practice, and number of monthly appraisals).

RESULTS

A total of 144 participants were included, mostly endocrinologists. There was moderate level of agreement for the absence of echogenic foci (AC1 0.53, 95% CI 0.24-0.81) and composition (AC1 0.54, 95% CI 0.36-0.71). The agreement for margins (AC1 0.24, 95% CI 0.15-0.33), echogenicity (AC1 0.34, 95% CI 0.22-0.46), and shape assessment (AC1 0.42, 95% CI 0.13-0.70) was lower. The overall agreement for ACR-TIRADS assessment was AC1 0.29, (95% CI 0.13-0.45). The AC1 of ACR-TIRADS among subgroups was similar.

CONCLUSIONS

This study found high variation of judgments about ACR-TIRADS risk category and individual features, which poses a potential challenge for the widescale implementation of thyroid nodule risk stratification.

Can sonographic features of microcalcification predict thyroid nodule malignancy? a prospective observational study

Background

The aim of this study was to investigate the diagnostic accuracy of microcalcification, as well as its associated sonographic features, for prediction of thyroid nodule malignancy.

We prospectively assessed the patients with thyroid nodule, who underwent ultrasound-guided fine-needle aspiration during 2017–2020 in Babol, northern Iran. The ultrasonographic characteristics of the nodules, as well as their cytological results, were recorded. We used regression analysis to evaluate the relation between sonographic findings and nodule malignancy. A receiver operator characteristics (ROC) analysis was also used to estimate the ability of ultrasound to predict the characteristic features of malignancy, as estimated by the area under the curve (AUC).

Results

Overall, 1129 thyroid nodules were finally included in the study, of which 452 (40%) had microcalcification. A significant positive association was found between nodule malignancy and microcalcification in both univariate (OR=3.626, 95% CI 2.258–5.822) and multivariable regression analyses (OR=1.878, 95% CI 1.095–3.219). In the nodules with microcalcification, significant positive relations were seen between malignancy and hypoechogenicity (OR=3.833, 95% CI 1.032–14.238), >5 microcalcification number (OR=3.045, 95% CI 1.328–6.982), irregular margin (OR=3.341, 95% CI 1.078–10.352), and lobulated margin (OR=5.727, 95% CI 1.934–16.959). The ROC analysis indicated that AUC for hypoechogenicity, >5 microcalcification number, irregular margin, and lobulated margin were 60%, 62%, 55%, and 60%, respectively, in predicting malignant thyroid nodules.

Conclusion

The findings indicated that microcalcification can be a potential predictor of thyroid nodule malignancy. Also, the presence of irregular or lobulated margins, multiple intranodular microcalcification (>5 microcalcifications), and/or hypoechogenicity can improve the ability of microcalcification in distinguishing malignant from benign nodules.

A Preliminary Study of Quantitative Ultrasound for Cancer-Risk Assessment of Thyroid Nodules

BACKGROUND

Gray-scale, B-mode ultrasound (US) imaging is part of the standard clinical procedure for evaluating thyroid nodules (TNs). It is limited by its instrument- and operator-dependence and inter-observer variability. In addition, the accepted high-risk B-mode US TN features are more specific for detecting classic papillary thyroid cancer rather than the follicular variant of papillary thyroid cancer or follicular thyroid cancer. Quantitative ultrasound (QUS) is a technique that can non-invasively assess properties of tissue microarchitecture by exploiting information contained in raw ultrasonic radiofrequency (RF) echo signals that is discarded in conventional B-mode imaging. QUS provides quantitative parameter-value estimates that are a function of the properties of US scatterers and microarchitecture of the tissue. The purpose of this preliminary study was to assess the performance of QUS parameters in evaluating benign and malignant thyroid nodules.

METHODS

Patients from the Thyroid Health Center at the Boston Medical Center were recruited to participate. B-mode and RF data were acquired and analyzed in 225 TNs (24 malignant and 201 benign) from 208 patients. These data were acquired either before (167 nodules) or after (58 nodules) subjects underwent fine-needle biopsy (FNB). The performance of a combination of QUS parameters (CQP) was assessed and compared with the performance of B-mode risk-stratification systems.

RESULTS

CQP produced an ROC AUC value of 0.857 ± 0.033 compared to a value of 0.887 ± 0.033 (p=0.327) for the American College of Radiology Thyroid Imaging, Reporting and Data System (ACR TI-RADS) and 0.880 ± 0.041 (p=0.367) for the American Thyroid Association (ATA) risk-stratification system. Furthermore, using a CQP threshold of 0.263 would further reduce the number of unnecessary FNBs in 44% of TNs without missing any malignant TNs. When CQP used in combination with ACR TI-RADS, a potential additional reduction of 49 to 66% in unnecessary FNBs was demonstrated.

CONCLUSION

This preliminary study suggests that QUS may provide a method to classify TNs when used by itself or when combined with a conventional gray-scale US risk-stratification system and can potentially reduce the need to biopsy TNs.

Interobserver variability in quality assessment of magnetic resonance images

Background

The perceptual quality of magnetic resonance (MR) images influences diagnosis and may compromise the treatment. The purpose of this study was to evaluate how the image quality changes influence the interobserver variability of their assessment.

Methods

For the variability evaluation, a dataset containing distorted MRI images was prepared and then assessed by 31 experienced medical professionals (radiologists). Differences between observers were analyzed using the Fleiss’ kappa. However, since the kappa evaluates the agreement among radiologists taking into account aggregated decisions, a typically employed criterion of the image quality assessment (IQA) performance was used to provide a more thorough analysis. The IQA performance of radiologists was evaluated by comparing the Spearman correlation coefficients, ρ, between individual scores with the mean opinion scores (MOS) composed of the subjective opinions of the remaining professionals.

Results

The experiments show that there is a significant agreement among radiologists (κ=0.12; 95% confidence interval [CI]: 0.118, 0.121; P<0.001) on the quality of the assessed images. The resulted κ is strongly affected by the subjectivity of the assigned scores, separately presenting close scores. Therefore, the ρ was used to identify poor performance cases and to confirm the consistency of the majority of collected scores (ρ_mean = 0.5706). The results for interns (ρ_mean = 0.6868) supports the finding that the quality assessment of MR images can be successfully taught.

Conclusions

The agreement observed among radiologists from different imaging centers confirms the subjectivity of the perception of MR images. It was shown that the image content and severity of distortions affect the IQA. Furthermore, the study highlights the importance of the psychosomatic condition of the observers and their attitude.

Diagnostic Performance of Four Ultrasound Risk Stratification Systems: A Systematic Review and Meta-Analysis

Background: Several ultrasound (US)-based risk stratification systems have been increasingly used for the optimal management of thyroid nodules. However, there are considerable discrepancies across these systems. This study aimed to summarize and compare the category-based diagnostic performance in the detection of thyroid cancer of different US-based risk stratification systems from four societies: the American College of Radiology-Thyroid Imaging Reporting and Data System (ACR-TIRADS), the American Thyroid Association (ATA), the Korean Thyroid Association/Korean Society of Thyroid Radiology (KTA/KSThR; K-TIRADS), and the European Thyroid Association (EU-TIRADS). Methods: MEDLINE/PubMed and EMBASE databases were searched to identify original articles investigating the category-based diagnostic performance according to at least one of the following guidelines: ACR-TIRADS, ATA, K-TIRADS, and EU-TIRADS. Pooled sensitivity and specificity were calculated using a bivariate random-effects model. A subgroup analysis on nodules of 1 cm or larger and a meta-regression analysis to identify factors associated with the diagnostic performance were performed. Results: A total of 29 articles including 33,748 thyroid nodules met the eligibility criteria and were included in the analysis. For ACR-TIRADS, the pooled sensitivity and specificity were, respectively, 66% and 91% for category 5 and 95% and 55% for category 4 or 5. For ATA, the pooled sensitivity and specificity were, respectively, 74% and 88% for category 5 and 91% and 64% for category 4 or 5. For K-TIRADS, the pooled sensitivity and specificity were, respectively, 55% and 95% for category 5 and 89% and 64% for category 4 or 5. For EU-TIRADS, the pooled sensitivity and specificity were, respectively, 82% and 90% for category 5 and 96% and 52% for category 4 or 5. Study location, proportion of female patients and malignant nodules, and study design were associated with study heterogeneity. Conclusions: The overall diagnostic performance of the four US-based risk stratification systems was comparable.

Inter- and Intraobserver Agreement in the Assessment of Thyroid Nodule Ultrasound Features and Classification Systems: A Blinded Multicenter Study

Background: Single-center trials demonstrated moderate-substantial level of interobserver agreement in the evaluation of ultrasound (US) features of thyroid nodules. Multicenter studies on US agreement, however, are scanty, and data on intraobserver agreement are poor. Aim of the study was to assess inter- and intraobserver agreement between different thyroid centers and different specialists. Methods: A blinded analysis of 100 electronically recorded thyroid nodule US images was conducted in three large-volume thyroid centers by seven radiologists and endocrinologists. The evaluation was repeated after randomization 4 months later. The following US characteristics were evaluated: composition, echogenicity, margins, intranodular echogenic spots, vascularity, and shape. Thyroid nodules were also classified according to AACE/ACE/AME, EU-TIRADS, ATA, and ACR-TIRADS US classifications. Intra- and interobserver agreement was calculated using cross-tabulation expressed as mean Cohen's Kappa. Results: Interobserver agreement for US features: K-coefficient was 0.53 for composition, 0.47 for echogenicity, 0.46 for intranodular vascularity, and 0.33 for margins of the nodules. For echogenic foci, the K-coefficient was 0.47 for microcalcifications, 0.38 for macrocalcifications, 0.11 for the subcategory comet-tail artifacts, and 0.42 for shape. Operators resulted uncertain on hyperechoic foci definition in 16% of cases and described them as "hyperechoic foci of uncertain significance." Interobserver Cohen-K for US classification systems was 0.44 for AACE, 0.42 for ACR-TIRADS, 0.39 EU-TIRADS, and 0.34 for ATA. Intraobserver agreement: the K-coefficient for nodule US features was 0.62 for intranodular vascularity, 0.58 for composition, 0.60 for echogenicity, 0.54 for macrocalcifications, 0.55 for microcalcifications, 0.47 for comet tails, 0.39 for margins, and 0.35 for shape. Intraobserver Cohen-K for US classification systems was 0.54 for AACE, 0.49 for ACR-TIRADS, 0.38 for ATA, and 0.33 for EU-TIRADS. Conclusions: Intraobserver reproducibility for thyroid nodule US reporting and US classification systems appears fairly adequate, while the interobserver agreement between different centers is lower than that assessed in single-center trials. Reporting and rating ability of thyroid US examiners still appear not consistent. An unified lexicon of thyroid US features, a simplified method of classification, and a dedicated training in the description of thyroid US findings may increase the observers' agreement and the predictive value of US classification systems in real world practice.

Classification and Regression Tree (CART) model of sonographic signs in predicting thyroid nodules malignancy

•

Taller-than-wide thyroid nodules are more likely to require surgery.

•

A taller-than-wide, solid and hypoechoic thyroid nodule is likely a Bethesda 4 or 5.

•

Taller-than-wide sonographic sign is the strongest in predicting thyroid malignancy.

Purpose

To develop a Classification and Regression Tree (CART) model in order to recognize the most suspicious sonographic features of thyroid nodules and efficiently guide their management.

Methods

791 thyroid fine needle aspiration cytology (FNAC) performed under ultrasound guidance between January 2015 and January 2017 were reviewed. Retrieved data consisted in qualitative (patient’s gender, composition, echogenicity, shape, margins and echogenic foci of the nodule) and quantitative (patient’s age and maximal diameter of the nodule) variables as well as the Bethesda score.

Results

Patients were 48.5 ± 13.7 years old with female to male ratio of 8:2. The nodules had median size of 2.3 (1.5–3.5) cm with a majority of solid (62.5 %) and isoechoic (50.8 %) features. 700 nodules (88.5 %) had a wider-than-tall shape, 600 (75.9 %) smooth margins and 113 (14.3 %) ill-defined ones. Echogenic foci were absent in 388 nodules (49.1 %) and, when present, largely dominated by punctate foci (32.5 %). Bethesda classes 3, 4 and 5, which require surgery, represented only 10.6 % of cases. They were significantly correlated with the taller-than-wide shape and with solid or predominantly solid features. There was no significant correlation between echostructure and Bethesda scores but we did find more nodules classified Bethesda 4 and 5 in the categories hypoechoic and severely hypoechoic. In the CART model we developed, the sequence leading to most nodules classified Bethesda 4 and 5 is: taller-than-wide shape, solid composition and hypoechoic or severely hypoechoic feature.

Conclusions

Taller-than-wide, solid or predominantly solid, hypoechoic or severely hypoechoic nodules are likely to require surgery and might benefit from FNAC.