Hierarchical Temporal Attention Network for Thyroid Nodule Recognition Using Dynamic CEUS Imaging

Applying machine-learning models to differentiate benign and malignant thyroid nodules classified as C-TIRADS 4 based on 2D-ultrasound combined with five contrast-enhanced ultrasound key frames

Objectives

To apply machine learning to extract radiomics features from thyroid two-dimensional ultrasound (2D-US) combined with contrast-enhanced ultrasound (CEUS) images to classify and predict benign and malignant thyroid nodules, classified according to the Chinese version of the thyroid imaging reporting and data system (C-TIRADS) as category 4.

Materials and methods

This retrospective study included 313 pathologically diagnosed thyroid nodules (203 malignant and 110 benign). Two 2D-US images and five CEUS key frames ("2nd second after the arrival time" frame, "time to peak" frame, "2nd second after peak" frame, "first-flash" frame, and "second-flash" frame) were selected to manually label the region of interest using the "Labelme" tool. A total of 7 images of each nodule and their annotates were imported into the Darwin Research Platform for radiomics analysis. The datasets were randomly split into training and test cohorts in a 9:1 ratio. Six classifiers, namely, support vector machine, logistic regression, decision tree, random forest (RF), gradient boosting decision tree and extreme gradient boosting, were used to construct and test the models. Performance was evaluated using a receiver operating characteristic curve analysis. The area under the curve (AUC), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), accuracy (ACC), and F1-score were calculated. One junior radiologist and one senior radiologist reviewed the 2D-US image and CEUS videos of each nodule and made a diagnosis. We then compared their AUC and ACC with those of our best model.

Results

The AUC of the diagnosis of US, CEUS and US combined CEUS by junior radiologist and senior radiologist were 0.755, 0.750, 0.784, 0.800, 0.873, 0.890, respectively. The RF classifier performed better than the other five, with an AUC of 1 for the training cohort and 0.94 (95% confidence interval 0.88-1) for the test cohort. The sensitivity, specificity, accuracy, PPV, NPV, and F1-score of the RF model in the test cohort were 0.82, 0.93, 0.90, 0.85, 0.92, and 0.84, respectively. The RF model with 2D-US combined with CEUS key frames achieved equivalent performance as the senior radiologist (AUC: 0.94 vs. 0.92, P = 0.798; ACC: 0.90 vs. 0.92) and outperformed the junior radiologist (AUC: 0.94 vs. 0.80, P = 0.039, ACC: 0.90 vs. 0.81) in the test cohort.

Conclusions

Our model, based on 2D-US and CEUS key frames radiomics features, had good diagnostic efficacy for thyroid nodules, which are classified as C-TIRADS 4. It shows promising potential in assisting less experienced junior radiologists.

Shape-margin knowledge augmented network for thyroid nodule segmentation and diagnosis

BACKGROUND AND OBJECTIVE

Thyroid nodule segmentation is a crucial step in the diagnostic procedure of physicians and computer-aided diagnosis systems. However, prevailing studies often treat segmentation and diagnosis as independent tasks, overlooking the intrinsic relationship between these processes. The sequencial steps of these independent tasks in computer-aided diagnosis systems may lead to the accumulation of errors. Therefore, it is worth combining them as a whole by exploring the relationship between thyroid nodule segmentation and diagnosis. According to the diagnostic procedure of thyroid imaging reporting and data system (TI-RADS), the assessment of shape and margin characteristics is the prerequisite for radiologists to discriminate benign and malignant thyroid nodules. Inspired by TI-RADS, this study aims to integrate these tasks into a cohesive process, leveraging the insights from TI-RADS, thereby enhancing the accuracy and interpretability of thyroid nodule analysis.

METHODS

Specifically, this paper proposes a shape-margin knowledge augmented network (SkaNet) for simultaneous thyroid nodule segmentation and diagnosis. Due to the visual feature similarities between segmentation and diagnosis, SkaNet shares visual features in the feature extraction stage and then utilizes a dual-branch architecture to perform thyroid nodule segmentation and diagnosis tasks respectively. In the shared feature extraction, the combination of convolutional feature maps and self-attention maps allows to exploitation of both local information and global patterns in thyroid nodule images. To enhance effective discriminative features, an exponential mixture module is introduced, combining convolutional feature maps and self-attention maps through exponential weighting. Then, SkaNet is jointly optimized by a knowledge augmented multi-task loss function with a constraint penalty term. The constraint penalty term embeds shape and margin characteristics through numerical computations, establishing a vital relationship between thyroid nodule diagnosis results and segmentation masks.

RESULTS

We evaluate the proposed approach on a public thyroid ultrasound dataset (DDTI) and a locally collected thyroid ultrasound dataset. The experimental results reveal the value of our contributions and demonstrate that our approach can yield significant improvements compared with state-of-the-art counterparts.

CONCLUSIONS

SkaNet highlights the potential of combining thyroid nodule segmentation and diagnosis with knowledge augmented learning into a unified framework, which captures the key shape and margin characteristics for discriminating benign and malignant thyroid nodules. Our findings suggest promising insights for advancing computer-aided diagnosis joint with segmentation.

Joint Segmentation and Differential Diagnosis of Thyroid Nodule in Contrast-Enhanced Ultrasound Images

OBJECTIVE

Microvascular perfusion can be observed in real time with contrast-enhanced ultrasound (CEUS), which is a novel ultrasound technology for visualizing the dynamic patterns of parenchymal perfusion. Automatic lesion segmentation and differential diagnosis of malignant and benign based on CEUS are crucial but challenging tasks for computer-aided diagnosis of thyroid nodule.

METHODS

To tackle these two formidable challenges concurrently, we provide Trans-CEUS, a spatial-temporal transformer-based CEUS analysis model to finish the joint learning of these two challenging tasks. Specifically, the dynamic swin-transformer encoder and multi-level feature collaborative learning are combined into U-net for achieving accurate segmentation of lesions with ambiguous boundary from CEUS. In addition, variant transformer-based global spatial-temporal fusion is proposed to obtain long-distance enhancement perfusion of dynamic CEUS for promoting differential diagnosis.

RESULTS

Empirical results of clinical data showed that our Trans-CEUS model achieved not only a good lesion segmentation result with a high Dice similarity coefficient of 82.41%, but also superior diagnostic accuracy of 86.59%. Conclusion & significance: This research is novel since it is the first to incorporate the transformer into CEUS analysis, and it shows promising results on dynamic CEUS datasets for both segmentation and diagnosis tasks of the thyroid nodule.

Domain knowledge-guided adversarial adaptive fusion of hybrid breast ultrasound data

Contrast-enhanced ultrasound (CEUS), which provides more detailed microvascular information about the tumor, is always taken by radiologists in clinic diagnosis along with B-mode ultrasound (B-mode US). However, automatically analyzing breast CEUS is challenging due to the difference between the CEUS video and the natural video, e.g., sports or action videos, where the CEUS video has no positional displacements. Additionally, most existing methods rarely use the Time Intensity Curve (TIC) information of CEUS and non-imaging clinical (NIC) data. To address these issues, we propose a novel breast cancer diagnosis framework that learns the complementarity and correlation across hybrid modal data, including CEUS, B-mode US, and NIC data, by an adversarial adaptive fusion method. Furthermore, to fully exploit the CEUS information, the proposed method, inspired by the clinical processing of radiologists, first extracts the TIC parameters of CEUS. Then, we select a clip from CEUS using a frame screening strategy and finally get spatio-temporal features from these clips through a critical frame attention network. To our knowledge, this is the first AI system to use TIC parameters, NIC data, and ultrasound imaging in diagnoses. We have validated our method on a dataset collected from 554 patients. The experimental results demonstrate the excellent performance of the proposed method. The result shows that our method can achieve an accuracy of 87.73%, which is higher than that of uni-modal approaches by nearly 5%.

Low-Dose CT Denoising via Sinogram Inner-Structure Transformer

Low-Dose Computed Tomography (LDCT) technique, which reduces the radiation harm to human bodies, is now attracting increasing interest in the medical imaging field. As the image quality is degraded by low dose radiation, LDCT exams require specialized reconstruction methods or denoising algorithms. However, most of the recent effective methods overlook the inner-structure of the original projection data (sinogram) which limits their denoising ability. The inner-structure of the sinogram represents special characteristics of the data in the sinogram domain. By maintaining this structure while denoising, the noise can be obviously restrained. Therefore, we propose an LDCT denoising network namely Sinogram Inner-Structure Transformer (SIST) to reduce the noise by utilizing the inner-structure in the sinogram domain. Specifically, we study the CT imaging mechanism and statistical characteristics of sinogram to design the sinogram inner-structure loss including the global and local inner-structure for restoring high-quality CT images. Besides, we propose a sinogram transformer module to better extract sinogram features. The transformer architecture using a self-attention mechanism can exploit interrelations between projections of different view angles, which achieves an outstanding performance in sinogram denoising. Furthermore, in order to improve the performance in the image domain, we propose the image reconstruction module to complementarily denoise both in the sinogram and image domain.

A novel ultrasound image diagnostic method for thyroid nodules

The incidence of thyroid nodules is increasing year by year. Accurate determination of benign and malignant nodules is an important basis for formulating treatment plans. Ultrasonography is the most widely used methodology in the diagnosis of benign and malignant nodules, but diagnosis by doctors is highly subjective, and the rates of missed diagnosis and misdiagnosis are high. To improve the accuracy of clinical diagnosis, this paper proposes a new diagnostic model based on deep learning. The diagnostic model adopts the diagnostic strategy of localization-classification. First, the distribution laws of the nodule size and nodule aspect ratio are obtained through data statistics, a multiscale localization network structure is a priori designed, and the nodule aspect ratio is obtained from the positioning results. Then, uncropped ultrasound images and nodule area image are correspondingly input into a two-way classification network, and an improved attention mechanism is used to enhance the feature extraction performance. Finally, the deep features, the shallow features, and the nodule aspect ratio are fused, and a fully connected layer is used to complete the classification of benign and malignant nodules. The experimental dataset consists of 4021 ultrasound images, where each image has been labeled under the guidance of doctors, and the ratio of the training set, validation set, and test set sizes is close to 3:1:1. The experimental results show that the accuracy of the multiscale localization network reaches 93.74%, and that the accuracy, specificity, and sensitivity of the classification network reach 86.34%, 81.29%, and 90.48%, respectively. Compared with the champion model of the TNSCUI 2020 classification competition, the accuracy rate is 1.52 points higher. Therefore, the network model proposed in this paper can effectively diagnose benign and malignant thyroid nodules.

Classification for thyroid nodule using ViT with contrastive learning in ultrasound images

The lack of representative features between benign nodules, especially level 3 of Thyroid Imaging Reporting and Data System (TI-RADS), and malignant nodules limits diagnostic accuracy, leading to inconsistent interpretation, overdiagnosis, and unnecessary biopsies. We propose a Vision-Transformer-based (ViT) thyroid nodule classification model using contrast learning, called TC-ViT, to improve accuracy of diagnosis and specificity of biopsy recommendations. ViT can explore the global features of thyroid nodules well. Nodule images are used as ROI to enhance the local features of the ViT. Contrast learning can minimize the representation distance between nodules of the same category, enhance the representation consistency of global and local features, and achieve accurate diagnosis of TI-RADS 3 or malignant nodules. The test results achieve an accuracy of 86.9%. The evaluation metrics show that the network outperforms other classical deep learning-based networks in terms of classification performance. TC-ViT can achieve automatic classification of TI-RADS 3 and malignant nodules on ultrasound images. It can also be used as a key step in computer-aided diagnosis for comprehensive analysis and accurate diagnosis. The code will be available at https://github.com/Jiawei217/TC-ViT.

Contrast-enhanced ultrasound improves the potency of fine-needle aspiration in thyroid nodules with high inadequate risk

Background

This study aims to determine the clinical value of contrast enhanced ultrasound (CEUS) for fine-needle aspiration (FNA) of high inadequate risky thyroid nodules.

Methods

During April 2018 and April 2021, consecutive 3748 thyroid nodules underwent FNA were retrospectively analyzed. CEUS guided FNA (CEUS-FNA) was applied in 115 nodules with high inadequate risk in Lingnan Campus. Ten nodules underwent CEUS-FNA presented non-enhancing, and would be further analyzed independently. Other 105 partial or total enhancing nodules were included as CEUS-FNA group, and 210 nodules with high inadequate risk in Tianhe Campus were match as the US-FNA control group. FNA specimens were collected for liquid-based preparation. Cytological results were classified following the Bethesda System for Reporting Thyroid Cytopathology.

Results

The overall FNA specimen inadequate rate in our center was 6.6%. All of the ten non-enhancing nodules under CEUS have an inadequate result in cytopathological analyzes. The subsequent postoperative pathology and follow-up ultrasonography showed the non-enhancing nodules were benign or stable without further malignant features. Total specimen inadequate rate of high inadequate risk thyroid nodules in CEUS-FNA group was significantly lower than US-FNA group (6.7% vs. 16.7%, P = 0.014). Further stratified analyzed shown that FNA under US guidance, the inadequate rates in cystic, predominantly cystic, predominantly solid and solid sub-groups were 28.1%, 17.1%, 10.0% and 9.2% (P = 0.019). In contrast, the inadequate rates in cystic, predominantly cystic, predominantly solid and solid sub-groups were 7.4%, 6.7%, 5.6% and 6.7% (P = 0.996) in CEUS-FNA group.

Conclusions

CEUS can improve the specimen adequacy of FNA in high inadequate risk thyroid nodules by avoiding unnecessary FNAs of the non-enhancing nodules, and accurately locating the viable tissue and precise guidance in real-time. CEUS is a recommend modality for FNA guidance of high inadequate risk thyroid nodules.

Deep Neural Architectures for Contrast Enhanced Ultrasound (CEUS) Focal Liver Lesions Automated Diagnosis

Computer vision, biomedical image processing and deep learning are related fields with a tremendous impact on the interpretation of medical images today. Among biomedical image sensing modalities, ultrasound (US) is one of the most widely used in practice, since it is noninvasive, accessible, and cheap. Its main drawback, compared to other imaging modalities, like computed tomography (CT) or magnetic resonance imaging (MRI), consists of the increased dependence on the human operator. One important step toward reducing this dependence is the implementation of a computer-aided diagnosis (CAD) system for US imaging. The aim of the paper is to examine the application of contrast enhanced ultrasound imaging (CEUS) to the problem of automated focal liver lesion (FLL) diagnosis using deep neural networks (DNN). Custom DNN designs are compared with state-of-the-art architectures, either pre-trained or trained from scratch. Our work improves on and broadens previous work in the field in several aspects, e.g., a novel leave-one-patient-out evaluation procedure, which further enabled us to formulate a hard-voting classification scheme. We show the effectiveness of our models, i.e., 88% accuracy reported against a higher number of liver lesion types: hepatocellular carcinomas (HCC), hypervascular metastases (HYPERM), hypovascular metastases (HYPOM), hemangiomas (HEM), and focal nodular hyperplasia (FNH).