Low-Memory CNNs Enabling Real-Time Ultrasound Segmentation Towards Mobile Deployment

A collaborative inference strategy for medical image diagnosis in mobile edge computing environment

The popularity and convenience of mobile medical image analysis and diagnosis in mobile edge computing (MEC) environments have greatly improved the efficiency and quality of healthcare services, necessitating the use of deep neural networks (DNNs) for image analysis. However, DNNs face performance and energy constraints when operating on the mobile side, and are limited by communication costs and privacy issues when operating on the edge side, and previous edge-end collaborative approaches have shown unstable performance and low search efficiency when exploring classification strategies. To address these issues, we propose a DNN edge-optimized collaborative inference strategy (MOCI) for medical image diagnosis, which optimizes data transfer and computation allocation by combining compression techniques and multi-agent reinforcement learning (MARL) methods. The MOCI strategy first uses coding and quantization-based compression methods to reduce the redundancy of image data during transmission at the edge, and then dynamically segments the DNN model through MARL and executes it collaboratively between the edge and the mobile device. To improve policy stability and adaptability, MOCI introduces the optimal transmission distance (Wasserstein) to optimize the policy update process, and uses the long short-term memory (LSTM) network to improve the model's adaptability to dynamic task complexity. The experimental results show that the MOCI strategy can effectively solve the collaborative inference task of medical image diagnosis and significantly reduce the latency and energy consumption with less than a 2% loss in classification accuracy, with a maximum reduction of 38.5% in processing latency and 71% in energy consumption compared to other inference strategies. In real-world MEC scenarios, MOCI has a wide range of potential applications that can effectively promote the development and application of intelligent healthcare.

LSMD: Long-Short Memory-Based Detection Network for Carotid Artery Detection in B-Mode Ultrasound Video Streams

Carotid atherosclerotic plaques are a major complication associated with type II diabetes, and carotid ultrasound is commonly used for diagnosing carotid vascular disease. In primary hospitals, less experienced ultrasound physicians often struggle to consistently capture standard carotid images and identify plaques. To address this issue, we propose a novel approach, the long-short memory-based detection (LSMD) network, for carotid artery detection in ultrasound video streams, facilitating the identification and localization of critical anatomical structures and plaques. This approach models short- and long-distance spatiotemporal features through short-term temporal aggregation (STA) and long-term temporal aggregation (LTA) modules, effectively expanding the temporal receptive field with minimal delay and enhancing the detection efficiency of carotid anatomy and plaques. Specifically, we introduce memory buffers with a dynamic updating strategy to ensure extensive temporal receptive field coverage while minimizing memory and computation costs. The proposed model was trained on 80 carotid ultrasound videos and evaluated on 50, with all videos annotated by physicians for carotid anatomies and plaques. The trained LSMD was evaluated for performance on the validation and test sets using the single-frame image-based single shot multibox detector (SSD) algorithm as a baseline. The results show that the precision, recall, average precision (AP) at ( ), and mean AP (mAP) are 6.83%, 12.29%, 11.23%, and 13.21% higher than the baseline ( ), respectively, while the model's inference latency reaches 6.97 ms on a desktop-level GPU (NVIDIA RTX 3090Ti) and 29.69 ms on an edge computing device (Jetson Orin Nano). These findings demonstrate that LSMD can accurately localize carotid anatomy and plaques with real-time inference, indicating its potential for enhancing diagnostic accuracy in clinical practice.

Enhancing Surgical Guidance: Deep Learning-Based Liver Vessel Segmentation in Real-Time Ultrasound Video Frames

BACKGROUND/OBJECTIVES

In the field of surgical medicine, the planning and execution of liver resection procedures present formidable challenges, primarily attributable to the intricate and highly individualized nature of liver vascular anatomy. In the current surgical milieu, intraoperative ultrasonography (IOUS) has become indispensable; however, traditional 2D ultrasound imaging's interpretability is hindered by noise and speckle artifacts. Accurate identification of critical structures for preservation during hepatectomy requires advanced surgical skills.

METHODS

An AI-based model that can help detect and recognize vessels including the inferior vena cava (IVC); the right (RHV), middle (MHV), and left (LVH) hepatic veins; the portal vein (PV) and its major first and second order branches the left portal vein (LPV), right portal vein (RPV), and right anterior (RAPV) and posterior (RPPV) portal veins, for real-time IOUS navigation can be of immense value in liver surgery. This research aims to advance the capabilities of IOUS-guided interventions by applying an innovative AI-based approach named the "2D-weigthed U-Net model" for the segmentation of multiple blood vessels in real-time IOUS video frames.

RESULTS

Our proposed deep learning (DL) model achieved a mean Dice score of 0.92 for IVC, 0.90 for RHV, 0.89 for MHV, 0.86 for LHV, 0.95 for PV, 0.93 for LPV, 0.84 for RPV, 0.85 for RAPV, and 0.96 for RPPV.

CONCLUSION

In the future, this research will be extended for real-time multi-label segmentation of extended vasculature in the liver, followed by the translation of our model into the surgical suite.

MEF-UNet: An end-to-end ultrasound image segmentation algorithm based on multi-scale feature extraction and fusion

Ultrasound image segmentation is a challenging task due to the complexity of lesion types, fuzzy boundaries, and low-contrast images along with the presence of noises and artifacts. To address these issues, we propose an end-to-end multi-scale feature extraction and fusion network (MEF-UNet) for the automatic segmentation of ultrasound images. Specifically, we first design a selective feature extraction encoder, including detail extraction stage and structure extraction stage, to precisely capture the edge details and overall shape features of the lesions. In order to enhance the representation capacity of contextual information, we develop a context information storage module in the skip-connection section, responsible for integrating information from adjacent two-layer feature maps. In addition, we design a multi-scale feature fusion module in the decoder section to merge feature maps with different scales. Experimental results indicate that our MEF-UNet can significantly improve the segmentation results in both quantitative analysis and visual effects.

Automated analysis of pectoralis major thickness in pec-fly exercises: evolving from manual measurement to deep learning techniques

This study addresses a limitation of prior research on pectoralis major (PMaj) thickness changes during the pectoralis fly exercise using a wearable ultrasound imaging setup. Although previous studies used manual measurement and subjective evaluation, it is important to acknowledge the subsequent limitations of automating widespread applications. We then employed a deep learning model for image segmentation and automated measurement to solve the problem and study the additional quantitative supplementary information that could be provided. Our results revealed increased PMaj thickness changes in the coronal plane within the probe detection region when real-time ultrasound imaging (RUSI) visual biofeedback was incorporated, regardless of load intensity (50% or 80% of one-repetition maximum). Additionally, participants showed uniform thickness changes in the PMaj in response to enhanced RUSI biofeedback. Notably, the differences in PMaj thickness changes between load intensities were reduced by RUSI biofeedback, suggesting altered muscle activation strategies. We identified the optimal measurement location for the maximal PMaj thickness close to the rib end and emphasized the lightweight applicability of our model for fitness training and muscle assessment. Further studies can refine load intensities, investigate diverse parameters, and employ different network models to enhance accuracy. This study contributes to our understanding of the effects of muscle physiology and exercise training.

Dense-PSP-UNet: A neural network for fast inference liver ultrasound segmentation

Liver Ultrasound (US) or sonography is popularly used because of its real-time output, low-cost, ease-of-use, portability, and non-invasive nature. Segmentation of real-time liver US is essential for diagnosing and analyzing liver conditions (e.g., hepatocellular carcinoma (HCC)), assisting the surgeons/radiologists in therapeutic procedures. In this paper, we propose a method using a modified Pyramid Scene Parsing (PSP) module in tuned neural network backbones to achieve real-time segmentation without compromising the segmentation accuracy. Considering widespread noise in US data and its impact on outcomes, we study the impact of pre-processing and the influence of loss functions on segmentation performance. We have tested our method after annotating a publicly available US dataset containing 2400 images of 8 healthy volunteers (link to the annotated dataset is provided); the results show that the Dense-PSP-UNet model achieves a high Dice coefficient of 0.913±0.024 while delivering a real-time performance of 37 frames per second (FPS).

Computation and memory optimized spectral domain convolutional neural network for throughput and energy-efficient inference

Conventional convolutional neural networks (CNNs) present a high computational workload and memory access cost (CMC). Spectral domain CNNs (SpCNNs) offer a computationally efficient approach to compute CNN training and inference. This paper investigates CMC of SpCNNs and its contributing components analytically and then proposes a methodology to optimize CMC, under three strategies, to enhance inference performance. In this methodology, output feature map (OFM) size, OFM depth or both are progressively reduced under an accuracy constraint to compute performance-optimized CNN inference. Before conducting training or testing, it can provide designers guidelines and preliminary insights regarding techniques for optimum performance, least degradation in accuracy and a balanced performance-accuracy trade-off. This methodology was evaluated on MNIST and Fashion MNIST datasets using LeNet-5 and AlexNet architectures. When compared to state-of-the-art SpCNN models, LeNet-5 achieves up to 4.2× (batch inference) and 4.1× (single-image inference) higher throughputs and 10.5× (batch inference) and 4.2× (single-image inference) greater energy efficiency at a maximum loss of 3% in test accuracy. When compared to the baseline model used in this study, AlexNet delivers 11.6× (batch inference) and 5× (single-image inference) increased throughput and 25× (batch inference) and 8.8× (single-image inference) more energy-efficient inference with just 4.4% reduction in accuracy.

A lightweight neural network with multiscale feature enhancement for liver CT segmentation

Segmentation of abdominal Computed Tomography (CT) scan is essential for analyzing, diagnosing, and treating visceral organ diseases (e.g., hepatocellular carcinoma). This paper proposes a novel neural network (Res-PAC-UNet) that employs a fixed-width residual UNet backbone and Pyramid Atrous Convolutions, providing a low disk utilization method for precise liver CT segmentation. The proposed network is trained on medical segmentation decathlon dataset using a modified surface loss function. Additionally, we evaluate its quantitative and qualitative performance; the Res16-PAC-UNet achieves a Dice coefficient of 0.950 ± 0.019 with less than half a million parameters. Alternatively, the Res32-PAC-UNet obtains a Dice coefficient of 0.958 ± 0.015 with an acceptable parameter count of approximately 1.2 million.

STAMP: Simultaneous Training and Model Pruning for low data regimes in medical image segmentation

Acquisition of high quality manual annotations is vital for the development of segmentation algorithms. However, to create them we require a substantial amount of expert time and knowledge. Large numbers of labels are required to train convolutional neural networks due to the vast number of parameters that must be learned in the optimisation process. Here, we develop the STAMP algorithm to allow the simultaneous training and pruning of a UNet architecture for medical image segmentation with targeted channelwise dropout to make the network robust to the pruning. We demonstrate the technique across segmentation tasks and imaging modalities. It is then shown that, through online pruning, we are able to train networks to have much higher performance than the equivalent standard UNet models while reducing their size by more than 85% in terms of parameters. This has the potential to allow networks to be directly trained on datasets where very low numbers of labels are available.

Hardware Architectures for Real-Time Medical Imaging

Medical imaging is considered one of the most important advances in the history of medicine and has become an essential part of the diagnosis and treatment of patients. Earlier prediction and treatment have been driving the acquisition of higher image resolutions as well as the fusion of different modalities, raising the need for sophisticated hardware and software systems for medical image registration, storage, analysis, and processing. In this scenario and given the new clinical pipelines and the huge clinical burden of hospitals, these systems are often required to provide both highly accurate and real-time processing of large amounts of imaging data. Additionally, lowering the prices of each part of imaging equipment, as well as its development and implementation, and increasing their lifespan is crucial to minimize the cost and lead to more accessible healthcare. This paper focuses on the evolution and the application of different hardware architectures (namely, CPU, GPU, DSP, FPGA, and ASIC) in medical imaging through various specific examples and discussing different options depending on the specific application. The main purpose is to provide a general introduction to hardware acceleration techniques for medical imaging researchers and developers who need to accelerate their implementations.