Sketch guided and progressive growing GAN for realistic and editable ultrasound image synthesis

Hierarchical online contrastive anomaly detection for fetal arrhythmia diagnosis in ultrasound

Arrhythmia is a major cardiac abnormality in fetuses. Therefore, early diagnosis of arrhythmia is clinically crucial. Pulsed-wave Doppler ultrasound is a commonly used diagnostic tool for fetal arrhythmia. Its key step for diagnosis involves identifying adjacent measurable cardiac cycles (MCCs). As cardiac activity is complex and the experience of sonographers is often varied, automation can improve user-independence and diagnostic-validity. However, arrhythmias pose several challenges for automation because of complex waveform variations, which can cause major localization bias and missed or false detection of MCCs. Filtering out non-MCC anomalies is difficult because of large intra-class and small inter-class variations between MCCs and non-MCCs caused by agnostic morphological waveform variations. Moreover, rare arrhythmia cases are insufficient for classification algorithms to adequately learn discriminative features. Using only normal cases for training, we propose a novel hierarchical online contrastive anomaly detection (HOCAD) framework for arrhythmia diagnosis during test time. The contribution of this study is three-fold. First, we develop a coarse-to-fine framework inspired by hierarchical diagnostic logic, which can refine localization and avoid missed detection of MCCs. Second, we propose an online learning-based contrastive anomaly detection with two new anomaly scores, which can adaptively filter out non-MCC anomalies on a single image during testing. With these complementary efforts, we precisely determine MCCs for correct measurements and diagnosis. Third, to the best of our knowledge, this is the first reported study investigating intelligent diagnosis of fetal arrhythmia on a large-scale and multi-center ultrasound dataset. Extensive experiments on 3850 cases, including 266 cases covering three typical types of arrhythmias, demonstrate the effectiveness of the proposed framework.

Fully Automated Localization and Measurement of Levator Hiatus Dimensions Using 3-D Pelvic Floor Ultrasound

OBJECTIVE

To develop an algorithm for the automated localization and measurement of levator hiatus (LH) dimensions (AI-LH) using 3-D pelvic floor ultrasound.

METHODS

The AI-LH included a 3-D plane regression model and a 2-D segmentation model, which first achieved automated localization of the minimal LH dimension plane (C-plane) and measurement of the hiatal area (HA) on maximum Valsalva on the rendered LH images, but not on the C-plane. The dataset included 600 volumetric data. We compared AI-LH with sonographer difference (ASD) as well as the inter-sonographer differences (IESD) in the testing dataset (n = 240). The assessment encompassed the mean absolute error (MAE) for the angle and center point distance of the C-plane, along with the Dice coefficient, MAE, and intra-class correlation coefficient (ICC) for HA, and included the time consumption.

RESULTS

The MAE of the C-plane of ASD was 4.81 ± 2.47° with 1.92 ± 1.54 mm. AI-LH achieved a mean Dice coefficient of 0.93 for LH segmentation. The MAE on HA of ASD (1.44 ± 1.12 mm²) was lower than that of IESD (1.63 ± 1.58 mm²). The ICC on HA of ASD (0.964) was higher than that of IESD (0.949). The average time costs of AI-LH and manual measurement were 2.00 ± 0.22 s and 59.60 ± 2.63 s (t = 18.87, p < 0.01), respectively.

CONCLUSION

AI-LH is accurate, reliable, and robust in the localization and measurement of LH dimensions, which can shorten the time cost, simplify the operation process, and have good value in clinical applications.

FetalBrainAwareNet: Bridging GANs with anatomical insight for fetal ultrasound brain plane synthesis

Over the past decade, deep-learning (DL) algorithms have become a promising tool to aid clinicians in identifying fetal head standard planes (FHSPs) during ultrasound (US) examination. However, the adoption of these algorithms in clinical settings is still hindered by the lack of large annotated datasets. To overcome this barrier, we introduce FetalBrainAwareNet, an innovative framework designed to synthesize anatomically accurate images of FHSPs. FetalBrainAwareNet introduces a cutting-edge approach that utilizes class activation maps as a prior in its conditional adversarial training process. This approach fosters the presence of the specific anatomical landmarks in the synthesized images. Additionally, we investigate specialized regularization terms within the adversarial training loss function to control the morphology of the fetal skull and foster the differentiation between the standard planes, ensuring that the synthetic images faithfully represent real US scans in both structure and overall appearance. The versatility of our FetalBrainAwareNet framework is highlighted by its ability to generate high-quality images of three predominant FHSPs using a singular, integrated framework. Quantitative (Fréchet inception distance of 88.52) and qualitative (t-SNE) results suggest that our framework generates US images with greater variability compared to state-of-the-art methods. By using the synthetic images generated with our framework, we increase the accuracy of FHSP classifiers by 3.2% compared to training the same classifiers solely with real acquisitions. These achievements suggest that using our synthetic images to increase the training set could provide benefits to enhance the performance of DL algorithms for FHSPs classification that could be integrated in real clinical scenarios.

Deep learning for the harmonization of structural MRI scans: a survey

Medical imaging datasets for research are frequently collected from multiple imaging centers using different scanners, protocols, and settings. These variations affect data consistency and compatibility across different sources. Image harmonization is a critical step to mitigate the effects of factors like inherent differences between various vendors, hardware upgrades, protocol changes, and scanner calibration drift, as well as to ensure consistent data for medical image processing techniques. Given the critical importance and widespread relevance of this issue, a vast array of image harmonization methodologies have emerged, with deep learning-based approaches driving substantial advancements in recent times. The goal of this review paper is to examine the latest deep learning techniques employed for image harmonization by analyzing cutting-edge architectural approaches in the field of medical image harmonization, evaluating both their strengths and limitations. This paper begins by providing a comprehensive fundamental overview of image harmonization strategies, covering three critical aspects: established imaging datasets, commonly used evaluation metrics, and characteristics of different scanners. Subsequently, this paper analyzes recent structural MRI (Magnetic Resonance Imaging) harmonization techniques based on network architecture, network learning algorithm, network supervision strategy, and network output. The underlying architectures include U-Net, Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), flow-based generative models, transformer-based approaches, as well as custom-designed network architectures. This paper investigates the effectiveness of Disentangled Representation Learning (DRL) as a pivotal learning algorithm in harmonization. Lastly, the review highlights the primary limitations in harmonization techniques, specifically the lack of comprehensive quantitative comparisons across different methods. The overall aim of this review is to serve as a guide for researchers and practitioners to select appropriate architectures based on their specific conditions and requirements. It also aims to foster discussions around ongoing challenges in the field and shed light on promising future research directions with the potential for significant advancements.

CBCT-based synthetic CT image generation using a diffusion model for CBCT-guided lung radiotherapy

BACKGROUND

Although cone beam computed tomography (CBCT) has lower resolution compared to planning CTs (pCT), its lower dose, higher high-contrast resolution, and shorter scanning time support its widespread use in clinical applications, especially in ensuring accurate patient positioning during the image-guided radiation therapy (IGRT) process.

PURPOSE

While CBCT is critical to IGRT, CBCT image quality can be compromised by severe stripe and scattering artifacts. Tumor movement secondary to respiratory motion also decreases CBCT resolution. In order to improve the image quality of CBCT, we propose a Lung Diffusion Model (L-DM) framework.

METHODS

Our proposed algorithm is based on a conditional diffusion model trained on pCT and deformed CBCT (dCBCT) image pairs to synthesize lung CT images from dCBCT images and benefit CBCT-based radiotherapy. dCBCT images were used as the constraint for the L-DM. The image quality and Hounsfield unit (HU) values of the synthetic CTs (sCT) images generated by the proposed L-DM were compared to three selected mainstream generation models.

RESULTS

We verified our model in both an institutional lung cancer dataset and a selected public dataset. Our L-DM showed significant improvement in the four metrics of mean absolute error (MAE), peak signal-to-noise ratio (PSNR), normalized cross-correlation (NCC), and structural similarity index measure (SSIM). In our institutional dataset, our proposed L-DM decreased the MAE from 101.47 to 37.87 HU and increased the PSNR from 24.97 to 29.89 dB, the NCC from 0.81 to 0.97, and the SSIM from 0.80 to 0.93. In the public dataset, our proposed L-DM decreased the MAE from 173.65 to 58.95 HU, while increasing the PSNR, NCC, and SSIM from 13.07 to 24.05 dB, 0.68 to 0.94, and 0.41 to 0.88, respectively.

CONCLUSIONS

The proposed L-DM significantly improved sCT image quality compared to the pre-correction CBCT and three mainstream generative models. Our model can benefit CBCT-based IGRT and other potential clinical applications as it increases the HU accuracy and decreases the artifacts from input CBCT images.

DermSynth3D: Synthesis of in-the-wild annotated dermatology images

In recent years, deep learning (DL) has shown great potential in the field of dermatological image analysis. However, existing datasets in this domain have significant limitations, including a small number of image samples, limited disease conditions, insufficient annotations, and non-standardized image acquisitions. To address these shortcomings, we propose a novel framework called DermSynth3D. DermSynth3D blends skin disease patterns onto 3D textured meshes of human subjects using a differentiable renderer and generates 2D images from various camera viewpoints under chosen lighting conditions in diverse background scenes. Our method adheres to top-down rules that constrain the blending and rendering process to create 2D images with skin conditions that mimic in-the-wild acquisitions, ensuring more meaningful results. The framework generates photo-realistic 2D dermatological images and the corresponding dense annotations for semantic segmentation of the skin, skin conditions, body parts, bounding boxes around lesions, depth maps, and other 3D scene parameters, such as camera position and lighting conditions. DermSynth3D allows for the creation of custom datasets for various dermatology tasks. We demonstrate the effectiveness of data generated using DermSynth3D by training DL models on synthetic data and evaluating them on various dermatology tasks using real 2D dermatological images. We make our code publicly available at https://github.com/sfu-mial/DermSynth3D.

A novel artificial intelligence model for measuring fetal intracranial markers during the first trimester based on two-dimensional ultrasound image

OBJECTIVE

To establish reference ranges of fetal intracranial markers during the first trimester and develop the first novel artificial intelligence (AI) model to measure key markers automatically.

METHODS

This retrospective study used two-dimensional (2D) ultrasound images from 4233 singleton normal fetuses scanned at 11+0-13+6 weeks of gestation at the Affiliated Suzhou Hospital of Nanjing Medical University from January 2018 to July 2022. We analyzed 10 key markers in three important planes of the fetal head. Based on these, reference ranges of 10 fetal intracranial markers were established and an AI model was developed for automated marker measurement. AI and manual measurements were compared to evaluate differences, correlations, consistency, and time consumption based on mean error, Pearson correlation analysis, intraclass correlation coefficients (ICCs), and average measurement time.

RESULTS

The results of AI and manual methods had strong consistency and correlation (all ICC values >0.75, all r values >0.75, and all P values <0.001). The average absolute error of both only ranged from 0.124 to 0.178 mm. AI achieved a 100% detection rate for abnormal cases. Additionally, the average measurement time of AI was only 0.49 s, which was more than 65 times faster than the manual measurement method.

CONCLUSION

The present study first established the normal standard reference ranges of fetal intracranial markers based on a large Chinese population data set. Furthermore, the proposed AI model demonstrated its capability to measure multiple fetal intracranial markers automatically, serving as a highly effective tool to streamline sonographer tasks and mitigate manual measurement errors, which can be generalized to first-trimester scanning.

Synthetic Optical Coherence Tomography Angiographs for Detailed Retinal Vessel Segmentation Without Human Annotations

Optical coherence tomography angiography (OCTA) is a non-invasive imaging modality that can acquire high-resolution volumes of the retinal vasculature and aid the diagnosis of ocular, neurological and cardiac diseases. Segmenting the visible blood vessels is a common first step when extracting quantitative biomarkers from these images. Classical segmentation algorithms based on thresholding are strongly affected by image artifacts and limited signal-to-noise ratio. The use of modern, deep learning-based segmentation methods has been inhibited by a lack of large datasets with detailed annotations of the blood vessels. To address this issue, recent work has employed transfer learning, where a segmentation network is trained on synthetic OCTA images and is then applied to real data. However, the previously proposed simulations fail to faithfully model the retinal vasculature and do not provide effective domain adaptation. Because of this, current methods are unable to fully segment the retinal vasculature, in particular the smallest capillaries. In this work, we present a lightweight simulation of the retinal vascular network based on space colonization for faster and more realistic OCTA synthesis. We then introduce three contrast adaptation pipelines to decrease the domain gap between real and artificial images. We demonstrate the superior segmentation performance of our approach in extensive quantitative and qualitative experiments on three public datasets that compare our method to traditional computer vision algorithms and supervised training using human annotations. Finally, we make our entire pipeline publicly available, including the source code, pretrained models, and a large dataset of synthetic OCTA images.

An intelligent quantification system for fetal heart rhythm assessment: A multicenter prospective study

BACKGROUND

The motion relationship and time intervals of the pulsed-wave Doppler (PWD) spectrum are essential for diagnosing fetal arrhythmia. However, few technologies currently are available to automatically calculate fetal cardiac time intervals (CTIs).

OBJECTIVE

The purpose of this study was to develop a fetal heart rhythm intelligent quantification system (HR-IQS) for the automatic extraction of CTIs and establish the normal reference range for fetal CTIs.

METHODS

A total of 6498 PWD spectrums of 2630 fetuses over the junction between the left ventricular inflow and outflow tracts were recorded across 14 centers. E, A, and V waves were manually labeled by 3 experienced fetal cardiologists, with 17 CTIs extracted. Five-fold cross-validation was performed for training and testing of the deep learning model. Agreement between the manual and HR-IQS-based values was evaluated using the intraclass correlation coefficient and Spearman's rank correlation coefficient. The Jarque-Bera test was applied to evaluate the normality of CTIs' distributions, and the normal reference range of 17 CTIs was established with quantile regression. Arrhythmia subset was compared with the non-arrhythmia subset using the Mann-Whitney U test.

RESULTS

Significant positive correlation (P <.001) and moderate-to-excellent consistency (P <.001) between the manual and HR-IQS automated measurements of CTIs was found. The distribution of CTIs was non-normal (P <.001). The normal range (2.5th to 97.5th percentiles) was successfully established for the 17 CTIs.

CONCLUSIONS

Using our HR-IQS is feasible for the automated calculation of CTIs in practice and thus could provide a promising tool for the assessment of fetal rhythm and function.

SEG-LUS: A novel ultrasound segmentation method for liver and its accessory structures based on muti-head self-attention

Although liver ultrasound (US) is quick and convenient, it presents challenges due to patient variations. Previous research has predominantly focused on computer-aided diagnosis (CAD), particularly for disease analysis. However, characterizing liver US images is complex due to structural diversity and a limited number of samples. Normal liver US images are crucial, especially for standard section diagnosis. This study explicitly addresses Liver US standard sections (LUSS) and involves detailed labeling of eight anatomical structures. We propose SEG-LUS, a US image segmentation model for the liver and its accessory structures. In SEG-LUS, we have adopted the shifted windows feature encoder combined with the cross-attention mechanism to adapt to capturing image information at different scales and resolutions and address context mismatch and sample imbalance in the segmentation task. By introducing the UUF module, we achieve the perfect fusion of shallow and deep information, making the information retained by the network in the feature extraction process more comprehensive. We have improved the Focal Loss to tackle the imbalance of pixel-level distribution. The results show that the SEG-LUS model exhibits significant performance improvement, with mPA, mDice, mIOU, and mASD reaching 85.05%, 82.60%, 74.92%, and 0.31, respectively. Compared with seven state-of-the-art semantic segmentation methods, the mPA improves by 5.32%. SEG-LUS is positioned to serve as a crucial reference for research in computer-aided modeling using liver US images, thereby advancing the field of US medicine research.

US2Mask: Image-to-mask generation learning via a conditional GAN for cardiac ultrasound image segmentation

Cardiac ultrasound (US) image segmentation is vital for evaluating clinical indices, but it often demands a large dataset and expert annotations, resulting in high costs for deep learning algorithms. To address this, our study presents a framework utilizing artificial intelligence generation technology to produce multi-class RGB masks for cardiac US image segmentation. The proposed approach directly performs semantic segmentation of the heart's main structures in US images from various scanning modes. Additionally, we introduce a novel learning approach based on conditional generative adversarial networks (CGAN) for cardiac US image segmentation, incorporating a conditional input and paired RGB masks. Experimental results from three cardiac US image datasets with diverse scan modes demonstrate that our approach outperforms several state-of-the-art models, showcasing improvements in five commonly used segmentation metrics, with lower noise sensitivity. Source code is available at https://github.com/energy588/US2mask.

LumVertCancNet: A novel 3D lumbar vertebral body cancellous bone location and segmentation method based on hybrid Swin-transformer

Lumbar vertebral body cancellous bone location and segmentation is crucial in an automated lumbar spine processing pipeline. Accurate and reliable analysis of lumbar spine image is expected to advantage practical medical diagnosis and population-based analysis of bone strength. However, the design of automated algorithms for lumbar spine processing is demanding due to significant anatomical variations and scarcity of publicly available data. In recent years, convolutional neural network (CNN) and vision transformers (Vits) have been the de facto standard in medical image segmentation. Although adept at capturing global features, the inherent bias of locality and weight sharing of CNN constrains its capacity to model long-range dependency. In contrast, Vits excel at long-range dependency modeling, but they may not generalize well with limited datasets due to the lack of inductive biases inherent to CNN. In this paper, we propose a deep learning-based two-stage coarse-to-fine solution to address the problem of automatic location and segmentation of lumbar vertebral body cancellous bone. Specifically, in the first stage, a Swin-transformer based model is applied to predict the heatmap of lumbar vertebral body centroids. Considering the characteristic anatomical structure of lumbar spine, we propose a novel loss function called LumAnatomy loss, which enforces the order and bend of the predicted vertebral body centroids. To inherit the excellence of CNN and Vits while preventing their respective limitations, in the second stage, we propose an encoder-decoder network to segment the identified lumbar vertebral body cancellous bone, which consists of two parallel encoders, i.e., a Swin-transformer encoder and a CNN encoder. To enhance the combination of CNNs and Vits, we propose a novel multi-scale attention feature fusion module (MSA-FFM), which address issues that arise when fusing features given at different encoders. To tackle the issue of lack of data, we raise the first large-scale lumbar vertebral body cancellous bone segmentation dataset called LumVBCanSeg containing a total of 185 CT scans annotated at voxel level by 3 physicians. Extensive experimental results on the LumVBCanSeg dataset demonstrate the proposed algorithm outperform other state-of-the-art medical image segmentation methods. The data is publicly available at: https://zenodo.org/record/8181250. The implementation of the proposed method is available at: https://github.com/sia405yd/LumVertCancNet.

Artificial intelligence assistance for fetal development: evaluation of an automated software for biometry measurements in the mid-trimester

BACKGROUND

This study presents CUPID, an advanced automated measurement software based on Artificial Intelligence (AI), designed to evaluate nine fetal biometric parameters in the mid-trimester. Our primary objective was to assess and compare the CUPID performance of experienced senior and junior radiologists.

MATERIALS AND METHODS

This prospective cross-sectional study was conducted at Shenzhen University General Hospital between September 2022 and June 2023, and focused on mid-trimester fetuses. All ultrasound images of the six standard planes, that enabled the evaluation of nine biometric measurements, were included to compare the performance of CUPID through subjective and objective assessments.

RESULTS

There were 642 fetuses with a mean (±SD) age of 22 ± 2.82 weeks at enrollment. In the subjective quality assessment, out of 642 images representing nine biometric measurements, 617-635 images (90.65-96.11%) of CUPID caliper placements were determined to be accurately placed and did not require any adjustments. Whereas, for the junior category, 447-691 images (69.63-92.06%) were determined to be accurately placed and did not require any adjustments. In the objective measurement indicators, across all nine biometric parameters and estimated fetal weight (EFW), the intra-class correlation coefficients (ICC) (0.843-0.990) and Pearson correlation coefficients (PCC) (0.765-0.978) between the senior radiologist and CUPID reflected good reliability compared with the ICC (0.306-0.937) and PCC (0.566-0.947) between the senior and junior radiologists. Additionally, the mean absolute error (MAE), percentage error (PE), and average error in days of gestation were lower between the senior and CUPID compared to the difference between the senior and junior radiologists. The specific differences are as follows: MAE (0.36-2.53 mm, 14.67 g) compared to (0.64- 8.13 mm, 38.05 g), PE (0.94-9.38%) compared to (1.58-16.04%), and average error in days (3.99-7.92 days) compared to (4.35-11.06 days). In the time-consuming task, CUPID only takes 0.05-0.07 s to measure nine biometric parameters, while senior and junior radiologists require 4.79-11.68 s and 4.95-13.44 s, respectively.

CONCLUSIONS

CUPID has proven to be highly accurate and efficient software for automatically measuring fetal biometry, gestational age, and fetal weight, providing a precise and fast tool for assessing fetal growth and development.

Segment anything model for medical images?

The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: (1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. (2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. (3) SAM performed better with manual hints, especially box, than the Everything mode. (4) SAM could help human annotation with high labeling quality and less time. (5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. (6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. (7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. (8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. Codes and models are available at: https://github.com/yuhoo0302/Segment-Anything-Model-for-Medical-Images. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.

MRI radiomics-based decision support tool for a personalized classification of cervical disc degeneration: a two-center study

Objectives: To develop and validate an MRI radiomics-based decision support tool for the automated grading of cervical disc degeneration. Methods: The retrospective study included 2,610 cervical disc samples of 435 patients from two hospitals. The cervical magnetic resonance imaging (MRI) analysis of patients confirmed cervical disc degeneration grades using the Pfirrmann grading system. A training set (1,830 samples of 305 patients) and an independent test set (780 samples of 130 patients) were divided for the construction and validation of the machine learning model, respectively. We provided a fine-tuned MedSAM model for automated cervical disc segmentation. Then, we extracted 924 radiomic features from each segmented disc in T1 and T2 MRI modalities. All features were processed and selected using minimum redundancy maximum relevance (mRMR) and multiple machine learning algorithms. Meanwhile, the radiomics models of various machine learning algorithms and MRI images were constructed and compared. Finally, the combined radiomics model was constructed in the training set and validated in the test set. Radiomic feature mapping was provided for auxiliary diagnosis. Results: Of the 2,610 cervical disc samples, 794 (30.4%) were classified as low grade and 1,816 (69.6%) were classified as high grade. The fine-tuned MedSAM model achieved good segmentation performance, with the mean Dice coefficient of 0.93. Higher-order texture features contributed to the dominant force in the diagnostic task (80%). Among various machine learning models, random forest performed better than the other algorithms (p < 0.01), and the T2 MRI radiomics model showed better results than T1 MRI in the diagnostic performance (p < 0.05). The final combined radiomics model had an area under the receiver operating characteristic curve (AUC) of 0.95, an accuracy of 89.51%, a precision of 87.07%, a recall of 98.83%, and an F1 score of 0.93 in the test set, which were all better than those of other models (p < 0.05). Conclusion: The radiomics-based decision support tool using T1 and T2 MRI modalities can be used for cervical disc degeneration grading, facilitating individualized management.

Approaches and Limitations of Machine Learning for Synthetic Ultrasound Generation: A Scoping Review

This scoping review examines the emerging field of synthetic ultrasound generation using machine learning (ML) models in radiology. Nineteen studies were analyzed, revealing three primary methodological strategies: unconditional generation, conditional generation, and domain translation. Synthetic ultrasound is mainly used to augment training datasets and as training material for radiologists. Blind expert assessment and Fréchet Inception Distance are common evaluation methods. Current limitations include the need for large training datasets, manual annotations for controllable generation, and insufficient research on incorporating new domain knowledge. While generative ultrasound models show promise, further work is required for clinical implementation.

Application and prospects of AI-based radiomics in ultrasound diagnosis

Artificial intelligence (AI)-based radiomics has attracted considerable research attention in the field of medical imaging, including ultrasound diagnosis. Ultrasound imaging has unique advantages such as high temporal resolution, low cost, and no radiation exposure. This renders it a preferred imaging modality for several clinical scenarios. This review includes a detailed introduction to imaging modalities, including Brightness-mode ultrasound, color Doppler flow imaging, ultrasound elastography, contrast-enhanced ultrasound, and multi-modal fusion analysis. It provides an overview of the current status and prospects of AI-based radiomics in ultrasound diagnosis, highlighting the application of AI-based radiomics to static ultrasound images, dynamic ultrasound videos, and multi-modal ultrasound fusion analysis.

A novel artificial intelligence model for fetal facial profile marker measurement during the first trimester

BACKGROUND

To study the validity of an artificial intelligence (AI) model for measuring fetal facial profile markers, and to evaluate the clinical value of the AI model for identifying fetal abnormalities during the first trimester.

METHODS

This retrospective study used two-dimensional mid-sagittal fetal profile images taken during singleton pregnancies at 11-13+ 6 weeks of gestation. We measured the facial profile markers, including inferior facial angle (IFA), maxilla-nasion-mandible (MNM) angle, facial-maxillary angle (FMA), frontal space (FS) distance, and profile line (PL) distance using AI and manual measurements. Semantic segmentation and landmark localization were used to develop an AI model to measure the selected markers and evaluate the diagnostic value for fetal abnormalities. The consistency between AI and manual measurements was compared using intraclass correlation coefficients (ICC). The diagnostic value of facial markers measured using the AI model during fetal abnormality screening was evaluated using receiver operating characteristic (ROC) curves.

RESULTS

A total of 2372 normal fetuses and 37 with abnormalities were observed, including 18 with trisomy 21, 7 with trisomy 18, and 12 with CLP. Among them, 1872 normal fetuses were used for AI model training and validation, and the remaining 500 normal fetuses and all fetuses with abnormalities were used for clinical testing. The ICCs (95%CI) of the IFA, MNM angle, FMA, FS distance, and PL distance between the AI and manual measurement for the 500 normal fetuses were 0.812 (0.780-0.840), 0.760 (0.720-0.795), 0.766 (0.727-0.800), 0.807 (0.775-0.836), and 0.798 (0.764-0.828), respectively. IFA clinically significantly identified trisomy 21 and trisomy 18, with areas under the ROC curve (AUC) of 0.686 (95%CI, 0.585-0.788) and 0.729 (95%CI, 0.621-0.837), respectively. FMA effectively predicted trisomy 18, with an AUC of 0.904 (95%CI, 0.842-0.966). MNM angle and FS distance exhibited good predictive value in CLP, with AUCs of 0.738 (95%CI, 0.573-0.902) and 0.677 (95%CI, 0.494-0.859), respectively.

CONCLUSIONS

The consistency of fetal facial profile marker measurements between the AI and manual measurement was good during the first trimester. The AI model is a convenient and effective tool for the early screen for fetal trisomy 21, trisomy 18, and CLP, which can be generalized to first-trimester scanning (FTS).

Anatomy preserving GAN for realistic simulation of intraoperative liver ultrasound images

In ultrasound-guided liver surgery, the lack of large-scale intraoperative ultrasound images with important anatomical structures remains an obstacle hindering the successful application of AI to ultrasound guidance. In this case, intraoperative ultrasound (iUS) simulation should be conducted from preoperative magnetic resonance (pMR), which not only helps doctors understand the characteristics of iUS in advance, but also expands the iUS dataset from various imaging positions, thereby promoting the automatic iUS analysis in ultrasound guidance. Herein, a novel anatomy preserving generative adversarial network (ApGAN) framework was proposed to generate simulated intraoperative ultrasound (Sim-iUS) of liver with precise structure information from pMR. Specifically, the low-rank factors based bimodal fusion was first established focusing on the effective information of hepatic parenchyma. Then, a deformation field based correction module was introduced to learn and correct the slight structural distortion from surgical operations. Meanwhile, the multiple loss functions were designed to constrain the simulation of the content, structures, and style. Empirical results of clinical data showed that the proposed ApGAN obtained higher Structural Similarity (SSIM) of 0.74 and Fr´echet Inception Distance (FID) of 35.54 compared to existing methods. Furthermore, the average Hausdorff Distance (HD) error of the liver capsule structure was less than 0.25 mm, and the average relative (Euclidean Distance) ED error for polyps was 0.12 mm, indicating the high-level precision of this ApGAN in simulating the anatomical structures and focal areas.

Less Is More: Unsupervised Mask-Guided Annotated CT Image Synthesis With Minimum Manual Segmentations

As a pragmatic data augmentation tool, data synthesis has generally returned dividends in performance for deep learning based medical image analysis. However, generating corresponding segmentation masks for synthetic medical images is laborious and subjective. To obtain paired synthetic medical images and segmentations, conditional generative models that use segmentation masks as synthesis conditions were proposed. However, these segmentation mask-conditioned generative models still relied on large, varied, and labeled training datasets, and they could only provide limited constraints on human anatomical structures, leading to unrealistic image features. Moreover, the invariant pixel-level conditions could reduce the variety of synthetic lesions and thus reduce the efficacy of data augmentation. To address these issues, in this work, we propose a novel strategy for medical image synthesis, namely Unsupervised Mask (UM)-guided synthesis, to obtain both synthetic images and segmentations using limited manual segmentation labels. We first develop a superpixel based algorithm to generate unsupervised structural guidance and then design a conditional generative model to synthesize images and annotations simultaneously from those unsupervised masks in a semi-supervised multi-task setting. In addition, we devise a multi-scale multi-task Fréchet Inception Distance (MM-FID) and multi-scale multi-task standard deviation (MM-STD) to harness both fidelity and variety evaluations of synthetic CT images. With multiple analyses on different scales, we could produce stable image quality measurements with high reproducibility. Compared with the segmentation mask guided synthesis, our UM-guided synthesis provided high-quality synthetic images with significantly higher fidelity, variety, and utility ( by Wilcoxon Signed Ranked test).

Locating Multiple Standard Planes in First-Trimester Ultrasound Videos via the Detection and Scoring of Key Anatomical Structures

OBJECTIVE

This study was aimed at developing a first-trimester standard plane detection (FTSPD) system that can automatically locate nine standard planes in ultrasound videos and investigating its utility in clinical practice.

METHODS

The FTSPD system, based on the YOLOv3 network, was developed to detect structures and evaluate the quality of plane images by using a pre-defined scoring system. A total of 220 videos from two different ultrasound scanners were collected to compare detection performance between our FTSPD system and sonographers with different levels of experience. The quality of the detected standard planes was quantitatively rated by an expert according to a scoring protocol. Kolmogorov-Smirnov analysis was used to compare the distributions of scores across all nine standard planes.

RESULTS

The expert-rated scores indicated that the quality of the standard planes detected by the FTSPD system was on par with that of the planes detected by senior sonographers. There were no significant differences in the distributions of the scores across all nine standard planes. The FTSPD system performed significantly better than junior sonographers in five standard plane types.

CONCLUSION

The results of this study suggest that our FTSPD system has significant potential for detecting standard planes in first-trimester ultrasound screening, which may help to improve the accuracy of fetal ultrasound screening and facilitate early diagnosis of abnormalities. The quality of the standard planes selected by junior sonographers can be significantly improved with the assistance of our FTSPD system.

Test-time bi-directional adaptation between image and model for robust segmentation

BACKGROUND AND OBJECTIVE

Deep learning models often suffer from performance degradations when deployed in real clinical environments due to appearance shifts between training and testing images. Most extant methods use training-time adaptation, which almost require target domain samples in the training phase. However, these solutions are limited by the training process and cannot guarantee the accurate prediction of test samples with unforeseen appearance shifts. Further, it is impractical to collect target samples in advance. In this paper, we provide a general method of making existing segmentation models robust to samples with unknown appearance shifts when deployed in daily clinical practice.

METHODS

Our proposed test-time bi-directional adaptation framework combines two complementary strategies. First, our image-to-model (I2M) adaptation strategy adapts appearance-agnostic test images to the learned segmentation model using a novel plug-and-play statistical alignment style transfer module during testing. Second, our model-to-image (M2I) adaptation strategy adapts the learned segmentation model to test images with unknown appearance shifts. This strategy applies an augmented self-supervised learning module to fine-tune the learned model with proxy labels that it generates. This innovative procedure can be adaptively constrained using our novel proxy consistency criterion. This complementary I2M and M2I framework demonstrably achieves robust segmentation against unknown appearance shifts using existing deep-learning models.

RESULTS

Extensive experiments on 10 datasets containing fetal ultrasound, chest X-ray, and retinal fundus images demonstrate that our proposed method achieves promising robustness and efficiency in segmenting images with unknown appearance shifts.

CONCLUSIONS

To address the appearance shift problem in clinically acquired medical images, we provide robust segmentation by using two complementary strategies. Our solution is general and amenable for deployment in clinical settings.

Deep learning based on carotid transverse B-mode scan videos for the diagnosis of carotid plaque: a prospective multicenter study

OBJECTIVES

Accurate detection of carotid plaque using ultrasound (US) is essential for preventing stroke. However, the diagnostic performance of junior radiologists (with approximately 1 year of experience in carotid US evaluation) is relatively poor. We thus aim to develop a deep learning (DL) model based on US videos to improve junior radiologists' performance in plaque detection.

METHODS

This multicenter prospective study was conducted at five hospitals. CaroNet-Dynamic automatically detected carotid plaque from carotid transverse US videos allowing clinical detection. Model performance was evaluated using expert annotations (with more than 10 years of experience in carotid US evaluation) as the ground truth. Model robustness was investigated on different plaque characteristics and US scanning systems. Furthermore, its clinical applicability was evaluated by comparing the junior radiologists' diagnoses with and without DL-model assistance.

RESULTS

A total of 1647 videos from 825 patients were evaluated. The DL model yielded high performance with sensitivities of 87.03% and 94.17%, specificities of 82.07% and 74.04%, and areas under the receiver operating characteristic curve of 0.845 and 0.841 on the internal and multicenter external test sets, respectively. Moreover, no significant difference in performance was noted among different plaque characteristics and scanning systems. Using the DL model, the performance of the junior radiologists improved significantly, especially in terms of sensitivity (largest increase from 46.3 to 94.44%).

CONCLUSIONS

The DL model based on US videos corresponding to real examinations showed robust performance for plaque detection and significantly improved the diagnostic performance of junior radiologists.

KEY POINTS

• The deep learning model based on US videos conforming to real examinations showed robust performance for plaque detection. • Computer-aided diagnosis can significantly improve the diagnostic performance of junior radiologists in clinical practice.

A comparative study of gastric histopathology sub-size image classification: From linear regression to visual transformer

Introduction

Gastric cancer is the fifth most common cancer in the world. At the same time, it is also the fourth most deadly cancer. Early detection of cancer exists as a guide for the treatment of gastric cancer. Nowadays, computer technology has advanced rapidly to assist physicians in the diagnosis of pathological pictures of gastric cancer. Ensemble learning is a way to improve the accuracy of algorithms, and finding multiple learning models with complementarity types is the basis of ensemble learning. Therefore, this paper compares the performance of multiple algorithms in anticipation of applying ensemble learning to a practical gastric cancer classification problem.

Methods

The complementarity of sub-size pathology image classifiers when machine performance is insufficient is explored in this experimental platform. We choose seven classical machine learning classifiers and four deep learning classifiers for classification experiments on the GasHisSDB database. Among them, classical machine learning algorithms extract five different image virtual features to match multiple classifier algorithms. For deep learning, we choose three convolutional neural network classifiers. In addition, we also choose a novel Transformer-based classifier.

Results

The experimental platform, in which a large number of classical machine learning and deep learning methods are performed, demonstrates that there are differences in the performance of different classifiers on GasHisSDB. Classical machine learning models exist for classifiers that classify Abnormal categories very well, while classifiers that excel in classifying Normal categories also exist. Deep learning models also exist with multiple models that can be complementarity.

Discussion

Suitable classifiers are selected for ensemble learning, when machine performance is insufficient. This experimental platform demonstrates that multiple classifiers are indeed complementarity and can improve the efficiency of ensemble learning. This can better assist doctors in diagnosis, improve the detection of gastric cancer, and increase the cure rate.

Multimodal ultrasound fusion network for differentiating between benign and malignant solid renal tumors

Objective: We aim to establish a deep learning model called multimodal ultrasound fusion network (MUF-Net) based on gray-scale and contrast-enhanced ultrasound (CEUS) images for classifying benign and malignant solid renal tumors automatically and to compare the model’s performance with the assessments by radiologists with different levels of experience.

Methods: A retrospective study included the CEUS videos of 181 patients with solid renal tumors (81 benign and 100 malignant tumors) from June 2012 to June 2021. A total of 9794 B-mode and CEUS-mode images were cropped from the CEUS videos. The MUF-Net was proposed to combine gray-scale and CEUS images to differentiate benign and malignant solid renal tumors. In this network, two independent branches were designed to extract features from each of the two modalities, and the features were fused using adaptive weights. Finally, the network output a classification score based on the fused features. The model’s performance was evaluated using five-fold cross-validation and compared with the assessments of the two groups of radiologists with different levels of experience.

Results: For the discrimination between benign and malignant solid renal tumors, the junior radiologist group, senior radiologist group, and MUF-Net achieved accuracy of 70.6%, 75.7%, and 80.0%, sensitivity of 89.3%, 95.9%, and 80.4%, specificity of 58.7%, 62.9%, and 79.1%, and area under the receiver operating characteristic curve of 0.740 (95% confidence internal (CI): 0.70–0.75), 0.794 (95% CI: 0.72–0.83), and 0.877 (95% CI: 0.83–0.93), respectively.

Conclusion: The MUF-Net model can accurately classify benign and malignant solid renal tumors and achieve better performance than senior radiologists.

Key points: The CEUS video data contain the entire tumor microcirculation perfusion characteristics. The proposed MUF-Net based on B-mode and CEUS-mode images can accurately distinguish between benign and malignant solid renal tumors with an area under the receiver operating characteristic curve of 0.877, which surpasses senior radiologists’ assessments by a large margin.