A learning-based approach for fast and robust vessel tracking in long ultrasound sequences

BAG-Net: a boundary detection and multiple attention-guided network for liver ultrasound image automatic segmentation in ultrasound guided surgery

Objective.Automated segmentation of targets in ultrasound (US) images during US-guided liver surgery holds the potential to assist physicians in fast locating critical areas such as blood vessels and lesions. However, this remains a challenging task primarily due to the image quality issues associated with US, including blurred edges and low contrast. In addition, studies specifically targeting liver segmentation are relatively scarce possibly since studying deep abdominal organs under US is difficult. In this paper, we proposed a network named BAG-Net to address these challenges and achieve accurate segmentation of liver targets with varying morphologies, including lesions and blood vessels.Approach.The BAG-Net was designed with a boundary detection module together with a position module to locate the target, and multiple attention-guided modules combined with the depth supervision strategy to enhance detailed segmentation of the target area.Main Results.Our method was compared to other approaches and demonstrated superior performance on two liver US datasets. Specifically, the method achieved 93.9% precision, 91.2% recall, 92.4% Dice coefficient, and 86.2% IoU to segment the liver tumor. Additionally, we evaluated the capability of our network to segment tumors on the breast US dataset (BUSI), where it also achieved excellent results.Significance.Our proposed method was validated to effectively segment liver targets with diverse morphologies, providing suspicious areas for clinicians to identify lesions or other characteristics. In the clinic, the method is anticipated to improve surgical efficiency during US-guided surgery.

Temporal contexts for motion tracking in ultrasound sequences with information bottleneck

BACKGROUND

Recently, deep convolutional neural networks (CNNs) have been widely adopted for ultrasound sequence tracking and shown to perform satisfactorily. However, existing trackers ignore the rich temporal contexts that exists between consecutive frames, making it difficult for these trackers to perceive information about the motion of the target.

PURPOSE

In this paper, we propose a sophisticated method to fully utilize temporal contexts for ultrasound sequences tracking with information bottleneck. This method determines the temporal contexts between consecutive frames to perform both feature extraction and similarity graph refinement, and information bottleneck is integrated into the feature refinement process.

METHODS

The proposed tracker combined three models. First, online temporal adaptive convolutional neural network (TAdaCNN) is proposed to focus on feature extraction and enhance spatial features using temporal information. Second, information bottleneck (IB) is incorporated to achieve more accurate target tracking by maximally limiting the amount of information in the network and discarding irrelevant information. Finally, we propose temporal adaptive transformer (TA-Trans) that efficiently encodes temporal knowledge by decoding it for similarity graph refinement. The tracker was trained on 2015 MICCAI Challenge on Liver Ultrasound Tracking (CLUST) dataset to evaluate the performance of the proposed method by calculating the tracking error (TE) between the predicted landmarks and the ground truth landmarks for each frame. The experimental results are compared with 13 state-of-the-art methods, and ablation studies are conducted.

RESULTS

On CLUST 2015 dataset, our proposed model achieves a mean TE of 0.81 ± 0.74 mm and a maximum TE of 1.93 mm for 85 point-landmarks across 39 ultrasound sequences in the 2D sequences. Tracking speed ranged from 41 to 63 frames per second (fps).

CONCLUSIONS

This study demonstrates a new integrated workflow for ultrasound sequences motion tracking. The results show that the model has excellent accuracy and robustness. Reliable and accurate motion estimation is provided for applications requiring real-time motion estimation in the context of ultrasound-guided radiation therapy.

An Upgraded Siamese Neural Network for Motion Tracking in Ultrasound Image Sequences

Deep learning is heavily being borrowed to solve problems in medical imaging applications, and Siamese neural networks are the front runners of motion tracking. In this article, we propose to upgrade one such Siamese architecture-based neural network for robust and accurate landmark tracking in ultrasound images to improve the quality of image-guided radiation therapy. Although several researchers have improved the Siamese architecture-based networks with sophisticated detection modules and by incorporating transfer learning, the inherent assumptions of the constant position model and the missing motion model remain unaddressed limitations. In our proposed model, we overcome these limitations by introducing two modules into the original architecture. We employ a reference template update to resolve the constant position model and a linear Kalman filter (LKF) to address the missing motion model. Moreover, we demonstrate that the proposed architecture provides promising results without transfer learning. The proposed model was submitted to an open challenge organized by MICCAI and was evaluated exhaustively on the Liver US Tracking (CLUST) 2D dataset. Experimental results proved that the proposed model tracked the landmarks with promising accuracy. Furthermore, we also induced synthetic occlusions to perform a qualitative analysis of the proposed approach. The evaluations were performed on the training set of the CLUST 2D dataset. The proposed method outperformed the original Siamese architecture by a significant margin.

Investigation of tumor and vessel motion correlation in the liver

Intrafraction imaging‐based motion management systems for external beam radiotherapy can rely on internal surrogate structures when the target is not easily visualized. This work evaluated the validity of using liver vessels as internal surrogates for the estimation of liver tumor motion. Vessel and tumor motion were assessed using ten two‐dimensional sagittal MR cine datasets collected on the ViewRay MRIdian. For each case, a liver tumor and at least one vessel were tracked for 175 s. A tracking approach utilizing block matching and multiple simultaneous templates was applied. Accuracy of the tracked motion was calculated from the error between the tracked centroid position and manually defined ground truth annotations. The patient’s abdomen surface and diaphragm were manually annotated in all frames. The Pearson correlation coefficient (CC) was used to compare the motion of the features and tumor in the anterior–posterior (AP) and superior–inferior (SI) directions. The distance between the centroids of the features and the tumors was calculated to assess if feature proximity affects relative correlation, and the tumor range of motion was determined. Intra‐ and interfraction motion amplitude variabilities were evaluated to further assess the relationship between tumor and feature motion. The mean CC between the motion of the vessel and the tumor were 0.85 ± 0.11 (AP) and 0.92 ± 0.04 (SI), 0.83 ± 0.11 (AP) and −0.89 ± 0.06 (SI) for the surface and tumor, and 0.80 ± 0.17 (AP) and 0.94 ± 0.03 (SI) for the diaphragm and tumor. For intrafraction analysis, the average amplitude variability was 2.47 ± 0.77 mm (AP) and 3.14 ± 1.49 mm (SI) for the vessels, 2.70 ± 1.08 mm (AP) and 3.43 ± 1.73 mm (SI) for the surface, and 2.76 ± 1.41 mm (AP) and 2.91 ± 1.38 mm (SI) for the diaphragm. No relationship between distance and motion correlation was observed. The motion of liver tumors and liver vessels was well correlated, making vessels a suitable surrogate for tumor motion in the liver.

2D ultrasound imaging based intra-fraction respiratory motion tracking for abdominal radiation therapy using machine learning

We have previously developed a robotic ultrasound imaging system for motion monitoring in abdominal radiation therapy. Owing to the slow speed of ultrasound image processing, our previous system could only track abdominal motions under breath-hold. To overcome this limitation, a novel 2D-based image processing method for tracking intra-fraction respiratory motion is proposed. Fifty-seven different anatomical features acquired from 27 sets of 2D ultrasound sequences were used in this study. Three 2D ultrasound sequences were acquired with the robotic ultrasound system from three healthy volunteers. The remaining datasets were provided by the 2015 MICCAI Challenge on Liver Ultrasound Tracking. All datasets were preprocessed to extract the feature point, and a patient-specific motion pattern was extracted by principal component analysis and slow feature analysis (SFA). The tracking finds the most similar frame (or indexed frame) by a k-dimensional-tree-based nearest neighbor search for estimating the tracked object location. A template image was updated dynamically through the indexed frame to perform a fast template matching (TM) within a learned smaller search region on the incoming frame. The mean tracking error between manually annotated landmarks and the location extracted from the indexed training frame is 1.80  ±  1.42 mm. Adding a fast TM procedure within a small search region reduces the mean tracking error to 1.14  ±  1.16 mm. The tracking time per frame is 15 ms, which is well below the frame acquisition time. Furthermore, the anatomical reproducibility was measured by analyzing the location's anatomical landmark relative to the probe; the position-controlled probe has better reproducibility and yields a smaller mean error across all three volunteer cases, compared to the force-controlled probe (2.69 versus 11.20 mm in the superior-inferior direction and 1.19 versus 8.21 mm in the anterior-posterior direction). Our method reduces the processing time for tracking respiratory motion significantly, which can reduce the delivery uncertainty.

Improving realism in patient-specific abdominal ultrasound simulation using CycleGANs

PURPOSE

In this paper, we propose to apply generative adversarial neural networks trained with a cycle consistency loss, or CycleGANs, to improve realism in ultrasound (US) simulation from computed tomography (CT) scans.

METHODS

A ray-casting US simulation approach is used to generate intermediate synthetic images from abdominal CT scans. Then, an unpaired set of these synthetic and real US images is used to train CycleGANs with two alternative architectures for the generator, a U-Net and a ResNet. These networks are finally used to translate ray-casting based simulations into more realistic synthetic US images.

RESULTS

Our approach was evaluated both qualitatively and quantitatively. A user study performed by 21 experts in US imaging shows that both networks significantly improve realism with respect to the original ray-casting algorithm ([Formula: see text]), with the ResNet model performing better than the U-Net ([Formula: see text]).

CONCLUSION

Applying CycleGANs allows to obtain better synthetic US images of the abdomen. These results can contribute to reduce the gap between artificially generated and real US scans, which might positively impact in applications such as semi-supervised training of machine learning algorithms and low-cost training of medical doctors and radiologists in US image interpretation.

Evaluation of 2D and 3D ultrasound tracking algorithms and impact on ultrasound-guided liver radiotherapy margins

PURPOSE

Compensation for respiratory motion is important during abdominal cancer treatments. In this work we report the results of the 2015 MICCAI Challenge on Liver Ultrasound Tracking and extend the 2D results to relate them to clinical relevance in form of reducing treatment margins and hence sparing healthy tissues, while maintaining full duty cycle.

METHODS

We describe methodologies for estimating and temporally predicting respiratory liver motion from continuous ultrasound imaging, used during ultrasound-guided radiation therapy. Furthermore, we investigated the trade-off between tracking accuracy and runtime in combination with temporal prediction strategies and their impact on treatment margins.

RESULTS

Based on 2D ultrasound sequences from 39 volunteers, a mean tracking accuracy of 0.9 mm was achieved when combining the results from the 4 challenge submissions (1.2 to 3.3 mm). The two submissions for the 3D sequences from 14 volunteers provided mean accuracies of 1.7 and 1.8 mm. In combination with temporal prediction, using the faster (41 vs 228 ms) but less accurate (1.4 vs 0.9 mm) tracking method resulted in substantially reduced treatment margins (70% vs 39%) in contrast to mid-ventilation margins, as it avoided non-linear temporal prediction by keeping the treatment system latency low (150 vs 400 ms). Acceleration of the best tracking method would improve the margin reduction to 75%.

CONCLUSIONS

Liver motion estimation and prediction during free-breathing from 2D ultrasound images can substantially reduce the in-plane motion uncertainty and hence treatment margins. Employing an accurate tracking method while avoiding non-linear temporal prediction would be favorable. This approach has the potential to shorten treatment time compared to breath-hold and gated approaches, and increase treatment efficiency and safety.

Ultrasound-based liver tracking utilizing a hybrid template/optical flow approach

PURPOSE

With the ongoing shift toward reduced invasiveness in many surgical procedures, methods for tracking moving targets within the body become vital. Non-invasive treatment methods such as stereotactic radiation therapy and high intensity focused ultrasound, in particular, rely on the accurate localization of targets throughout treatment to ensure optimal treatment provision. This work aims at developing a robust, accurate and fast method for target tracking based on ultrasound images.

METHODS

A method for tracking of targets in real-time ultrasound image data was developed, based on the combination of template matching, dense optical flow and image intensity information. A weighting map is generated from each of these approaches which are then normalized, weighted and combined, with the weighted mean position then calculated to predict the current position. The approach was evaluated on the Challenge for Liver Ultrasound Tracking 2015 dataset, consisting of a total of 24 training and 39 test datasets with a total of 53 and 85 annotated targets throughout the liver, respectively.

RESULTS

The proposed method was implemented in MATLAB and achieved an accuracy of [Formula: see text] (95%: 1.91) mm and [Formula: see text] (95%: 1.85) mm on the training and test data, respectively. Tracking frequencies of between 8 and 36 fps (mean of 22 fps) were observed, largely dependent on the size of the region of interest. The achieved results represent an improvement in mean accuracy of approximately 0.3 mm over the reported methods in existing literature.

CONCLUSIONS

This work describes an accurate and robust method for the tracking of points of interest within 2D ultrasound data, based on a combination of multi-template matching, dense optical flow and relative image intensity information.

A block matching based approach with multiple simultaneous templates for the real-time 2D ultrasound tracking of liver vessels

PURPOSE

The implementation of motion management techniques in radiation therapy can aid in mitigating uncertainties and reducing margins. For motion management to be effective, it is necessary to track key structures both accurately and at a real-time speed. Therefore, the focus of this work was to develop a 2D algorithm for the real-time tracking of ultrasound features to aid in radiation therapy motion management.

MATERIALS AND METHODS

The developed algorithm utilized a similarity measure-based block matching algorithm incorporating training methods and multiple simultaneous templates. The algorithm is broken down into three primary components, all of which use normalized cross-correlation (NCC) as a similarity metric. First, a global feature shift to account for gross displacements from the previous frame is determined using large block sizes which encompass the entirety of the feature. Second, the most similar reference frame is chosen from a series of training images that are accumulated during the first K frames of tracking to aid in contour consistency and provide a starting point for the localized template initialization. Finally, localized block matching is performed through the simultaneous use of both a training frame and the previous frame. The localized block matching utilizes a series of templates positioned at the boundary points of the training and previous contours. The weighted final boundary points from both the previous and the training frame are ultimately combined and used to determine an affine transformation from the previous frame to the current frame.

RESULTS

A mean tracking error of 0.72 ± 1.25 mm was observed for 85 point-landmarks across 39 ultrasound sequences relative to manual ground truth annotations. The image processing speed per landmark with the GPU implementation was between 41 and 165 frames per second (fps) during the training set accumulation, and between 73 and 234 fps after training set accumulation. Relative to a comparable multithreaded CPU approach using OpenMP, the GPU implementation resulted in speedups between -30% and 355% during training set accumulation, and between -37% and 639% postaccumulation.

CONCLUSIONS

Initial implementations indicated an accuracy that was comparable to or exceeding those achieved by alternative 2D tracking methods, with a computational speed that is more than sufficient for real-time applications in a radiation therapy environment. While the overall performance reached levels suitable for implementation in radiation therapy, the observed increase in failures for smaller features, as well as the algorithm's inability to be applied to nonconvex features warrants additional investigation to address the shortcomings observed.

Review of ultrasound image guidance in external beam radiotherapy part II: intra-fraction motion management and novel applications

Imaging has become an essential tool in modern radiotherapy (RT), being used to plan dose delivery prior to treatment and verify target position before and during treatment. Ultrasound (US) imaging is cost-effective in providing excellent contrast at high resolution for depicting soft tissue targets apart from those shielded by the lungs or cranium. As a result, it is increasingly used in RT setup verification for the measurement of inter-fraction motion, the subject of Part I of this review (Fontanarosa et al 2015 Phys. Med. Biol. 60 R77-114). The combination of rapid imaging and zero ionising radiation dose makes US highly suitable for estimating intra-fraction motion. The current paper (Part II of the review) covers this topic. The basic technology for US motion estimation, and its current clinical application to the prostate, is described here, along with recent developments in robust motion-estimation algorithms, and three dimensional (3D) imaging. Together, these are likely to drive an increase in the number of future clinical studies and the range of cancer sites in which US motion management is applied. Also reviewed are selections of existing and proposed novel applications of US imaging to RT. These are driven by exciting developments in structural, functional and molecular US imaging and analytical techniques such as backscatter tissue analysis, elastography, photoacoustography, contrast-specific imaging, dynamic contrast analysis, microvascular and super-resolution imaging, and targeted microbubbles. Such techniques show promise for predicting and measuring the outcome of RT, quantifying normal tissue toxicity, improving tumour definition and defining a biological target volume that describes radiation sensitive regions of the tumour. US offers easy, low cost and efficient integration of these techniques into the RT workflow. US contrast technology also has potential to be used actively to assist RT by manipulating the tumour cell environment and by improving the delivery of radiosensitising agents. Finally, US imaging offers various ways to measure dose in 3D. If technical problems can be overcome, these hold potential for wide-dissemination of cost-effective pre-treatment dose verification and in vivo dose monitoring methods. It is concluded that US imaging could eventually contribute to all aspects of the RT workflow.

Temporal regularization of ultrasound-based liver motion estimation for image-guided radiation therapy

PURPOSE

Ultrasound-based motion estimation is an expanding subfield of image-guided radiation therapy. Although ultrasound can detect tissue motion that is a fraction of a millimeter, its accuracy is variable. For controlling linear accelerator tracking and gating, ultrasound motion estimates must remain highly accurate throughout the imaging sequence. This study presents a temporal regularization method for correlation-based template matching which aims to improve the accuracy of motion estimates.

METHODS

Liver ultrasound sequences (15-23 Hz imaging rate, 2.5-5.5 min length) from ten healthy volunteers under free breathing were used. Anatomical features (blood vessels) in each sequence were manually annotated for comparison with normalized cross-correlation based template matching. Five sequences from a Siemens Acuson™ scanner were used for algorithm development (training set). Results from incremental tracking (IT) were compared with a temporal regularization method, which included a highly specific similarity metric and state observer, known as the α-β filter/similarity threshold (ABST). A further five sequences from an Elekta Clarity™ system were used for validation, without alteration of the tracking algorithm (validation set).

RESULTS

Overall, the ABST method produced marked improvements in vessel tracking accuracy. For the training set, the mean and 95th percentile (95%) errors (defined as the difference from manual annotations) were 1.6 and 1.4 mm, respectively (compared to 6.2 and 9.1 mm, respectively, for IT). For each sequence, the use of the state observer leads to improvement in the 95% error. For the validation set, the mean and 95% errors for the ABST method were 0.8 and 1.5 mm, respectively.

CONCLUSIONS

Ultrasound-based motion estimation has potential to monitor liver translation over long time periods with high accuracy. Nonrigid motion (strain) and the quality of the ultrasound data are likely to have an impact on tracking performance. A future study will investigate spatial uniformity of motion and its effect on the motion estimation errors.

4D Ultrasound Tracking of Liver and its Verification for TIPS Guidance

In this work we describe a 4D registration method for on the fly stabilization of ultrasound volumes for improving image guidance for transjugular intrahepatic portosystemic shunt (TIPS) interventions. The purpose of the method is to enable a continuous visualization of the relevant anatomical planes (determined in a planning stage) in a free breathing patient during the intervention. This requires registration of the planning information to the interventional images, which is achieved in two steps. In the first step tracking is performed across the streaming input. An approximate transformation between the reference image and the incoming image is estimated by composing the intermediate transformations obtained from the tracking. In the second step a subsequent registration is performed between the reference image and the approximately transformed incoming image to account for the accumulation of error. The two step approach helps in reducing the search range and is robust under rotation. We additionally present an approach to initialize and verify the registration. Verification is required when the reference image (containing planning information) is acquired in the past and is not part of the (interventional) 4D ultrasound sequence. The verification score will help in invalidating the registration outcome, for instance, in the case of insufficient overlap or information between the registering images due to probe motion or loss of contact, respectively. We evaluate the method over thirteen 4D US sequences acquired from eight subjects. A graphics processing unit implementation runs the 4D tracking at 9 Hz with a mean registration error of 1.7 mm.

Estimation of Large-Scale Organ Motion in B-Mode Ultrasound Image Sequences: A Survey

Reviewed here are methods developed for following (i.e., tracking) structures in medical B-mode ultrasound time sequences during large-scale motion. The resulting motion estimation problem and its key components are defined. The main tracking approaches are described, and their strengths and weaknesses are discussed. Existing motion estimation methods, tested on multiple in vivo sequences, are categorized with respect to their clinical applications, namely, cardiac, respiratory and muscular motion. A large number of works in this field had to be discarded as thorough validation of the results was missing. The remaining relevant works identified indicate the possibility of reaching an average tracking accuracy up to 1-2 mm. Real-time performance can be achieved using several methods. Yet only very few of these have progressed to clinical practice. The latest trends include incorporation of complementary and prior information. Advances are expected from common evaluation databases and 4-D ultrasound scanning technologies.

First Steps Toward Ultrasound-Based Motion Compensation for Imaging and Therapy: Calibration with an Optical System and 4D PET Imaging

Target motion, particularly in the abdomen, due to respiration or patient movement is still a challenge in many diagnostic and therapeutic processes. Hence, methods to detect and compensate this motion are required. Diagnostic ultrasound (US) represents a non-invasive and dose-free alternative to fluoroscopy, providing more information about internal target motion than respiration belt or optical tracking. The goal of this project is to develop an US-based motion tracking for real-time motion correction in radiation therapy and diagnostic imaging, notably in 4D positron emission tomography (PET). In this work, a workflow is established to enable the transformation of US tracking data to the coordinates of the treatment delivery or imaging system – even if the US probe is moving due to respiration. It is shown that the US tracking signal is equally adequate for 4D PET image reconstruction as the clinically used respiration belt and provides additional opportunities in this concern. Furthermore, it is demonstrated that the US probe being within the PET field of view generally has no relevant influence on the image quality. The accuracy and precision of all the steps in the calibration workflow for US tracking-based 4D PET imaging are found to be in an acceptable range for clinical implementation. Eventually, we show in vitro that an US-based motion tracking in absolute room coordinates with a moving US transducer is feasible.

The 2014 liver ultrasound tracking benchmark

The Challenge on Liver Ultrasound Tracking (CLUST) was held in conjunction with the MICCAI 2014 conference to enable direct comparison of tracking methods for this application. This paper reports the outcome of this challenge, including setup, methods, results and experiences. The database included 54 2D and 3D sequences of the liver of healthy volunteers and tumor patients under free breathing. Participants had to provide the tracking results of 90% of the data (test set) for pre-defined point-landmarks (healthy volunteers) or for tumor segmentations (patient data). In this paper we compare the best six methods which participated in the challenge. Quantitative evaluation was performed by the organizers with respect to manual annotations. Results of all methods showed a mean tracking error ranging between 1.4 mm and 2.1 mm for 2D points, and between 2.6 mm and 4.6 mm for 3D points. Fusing all automatic results by considering the median tracking results, improved the mean error to 1.2 mm (2D) and 2.5 mm (3D). For all methods, the performance is still not comparable to human inter-rater variability, with a mean tracking error of 0.5–0.6 mm (2D) and 1.2–1.8 mm (3D). The segmentation task was fulfilled only by one participant, resulting in a Dice coefficient ranging from 76.7% to 92.3%. The CLUST database continues to be available and the online leader-board will be updated as an ongoing challenge.