SUPRA: open-source software-defined ultrasound processing for real-time applications : A 2D and 3D pipeline from beamforming to B-mode

Needle tracking in low-resolution ultrasound volumes using deep learning

PURPOSE

Clinical needle insertion into tissue, commonly assisted by 2D ultrasound imaging for real-time navigation, faces the challenge of precise needle and probe alignment to reduce out-of-plane movement. Recent studies investigate 3D ultrasound imaging together with deep learning to overcome this problem, focusing on acquiring high-resolution images to create optimal conditions for needle tip detection. However, high-resolution also requires a lot of time for image acquisition and processing, which limits the real-time capability. Therefore, we aim to maximize the US volume rate with the trade-off of low image resolution. We propose a deep learning approach to directly extract the 3D needle tip position from sparsely sampled US volumes.

METHODS

We design an experimental setup with a robot inserting a needle into water and chicken liver tissue. In contrast to manual annotation, we assess the needle tip position from the known robot pose. During insertion, we acquire a large data set of low-resolution volumes using a 16  ×  16 element matrix transducer with a volume rate of 4 Hz. We compare the performance of our deep learning approach with conventional needle segmentation.

RESULTS

Our experiments in water and liver show that deep learning outperforms the conventional approach while achieving sub-millimeter accuracy. We achieve mean position errors of 0.54 mm in water and 1.54 mm in liver for deep learning.

CONCLUSION

Our study underlines the strengths of deep learning to predict the 3D needle positions from low-resolution ultrasound volumes. This is an important milestone for real-time needle navigation, simplifying the alignment of needle and ultrasound probe and enabling a 3D motion analysis.

Real-Time Motion Analysis With 4D Deep Learning for Ultrasound-Guided Radiotherapy

Motion compensation in radiation therapy is a challenging scenario that requires estimating and forecasting motion of tissue structures to deliver the target dose. Ultrasound offers direct imaging of tissue in real-time and is considered for image guidance in radiation therapy. Recently, fast volumetric ultrasound has gained traction, but motion analysis with such high-dimensional data remains difficult. While deep learning could bring many advantages, such as fast data processing and high performance, it remains unclear how to process sequences of hundreds of image volumes efficiently and effectively. We present a 4D deep learning approach for real-time motion estimation and forecasting using long-term 4D ultrasound data. Using motion traces acquired during radiation therapy combined with various tissue types, our results demonstrate that long-term motion estimation can be performed markerless with a tracking error of 0.35±0.2 mm and with an inference time of less than 5 ms. Also, we demonstrate forecasting directly from the image data up to 900 ms into the future. Overall, our findings highlight that 4D deep learning is a promising approach for motion analysis during radiotherapy.

Modular robotic platform for precision neurosurgery with a bio-inspired needle: System overview and first in-vivo deployment

Over the past 10 years, minimally invasive surgery (MIS) has shown significant benefits compared to conventional surgical techniques, with reduced trauma, shorter hospital stays, and shorter patient recovery times. In neurosurgical MIS procedures, inserting a straight tool (e.g. catheter) is common practice in applications ranging from biopsy and laser ablation, to drug delivery and fluid evacuation. How to handle tissue deformation, target migration and access to deep-seated anatomical structures remain an open challenge, affecting both the preoperative planning phase and eventual surgical intervention. Here, we present the first neurosurgical platform in the literature, able to deliver an implantable steerable needle for a range of diagnostic and therapeutic applications, with a short-term focus on localised drug delivery. This work presents the system's architecture and first in vivo deployment with an optimised surgical workflow designed for pre-clinical trials with the ovine model, which demonstrate appropriate function and safe implantation.

Systematic analysis of volumetric ultrasound parameters for markerless 4D motion tracking

OBJECTIVES

Motion compensation is an interesting approach to improve treatments of moving structures. For example, target motion can substantially affect dose delivery in radiation therapy, where methods to detect and mitigate the motion are widely used. Recent advances in fast, volumetric ultrasound have rekindled the interest in ultrasound for motion tracking. We present a setup to evaluate ultrasound based motion tracking and we study the effect of imaging rate and motion artifacts on its performance.

METHODS

We describe an experimental setup to acquire markerless 4D ultrasound data with precise ground truth from a robot and evaluate different real-world trajectories and system settings toward accurate motion estimation. We analyze motion artifacts in continuously acquired data by comparing to data recorded in a step-and-shoot fashion. Furthermore, we investigate the trade-off between the imaging frequency and resolution.

RESULTS

The mean tracking errors show that continuously acquired data leads to similar results as data acquired in a step-and-shoot fashion. We report mean tracking errors up to 2.01 mm and 1.36 mm on the continuous data for the lower and higher resolution, respectively, while step-and-shoot data leads to mean tracking errors of 2.52 mm and 0.98 mm.

CONCLUSIONS

We perform a quantitative analysis of different system settings for motion tracking with 4D ultrasound. We can show that precise tracking is feasible and additional motion in continuously acquired data does not impair the tracking. Moreover, the analysis of the frequency resolution trade-off shows that a high imaging resolution is beneficial in ultrasound tracking.

Insights into Infusion-Based Targeted Drug Delivery in the Brain: Perspectives, Challenges and Opportunities

Targeted drug delivery in the brain is instrumental in the treatment of lethal brain diseases, such as glioblastoma multiforme, the most aggressive primary central nervous system tumour in adults. Infusion-based drug delivery techniques, which directly administer to the tissue for local treatment, as in convection-enhanced delivery (CED), provide an important opportunity; however, poor understanding of the pressure-driven drug transport mechanisms in the brain has hindered its ultimate success in clinical applications. In this review, we focus on the biomechanical and biochemical aspects of infusion-based targeted drug delivery in the brain and look into the underlying molecular level mechanisms. We discuss recent advances and challenges in the complementary field of medical robotics and its use in targeted drug delivery in the brain. A critical overview of current research in these areas and their clinical implications is provided. This review delivers new ideas and perspectives for further studies of targeted drug delivery in the brain.

An unsupervised learning approach to ultrasound strain elastography with spatio-temporal consistency

Quasi-static ultrasound elastography (USE) is an imaging modality that measures deformation (i.e. strain) of tissue in response to an applied mechanical force. In USE, the strain modulus is traditionally obtained by deriving the displacement field estimated between a pair of radio-frequency data. In this work we propose a recurrent network architecture with convolutional long-short-term memory decoder blocks to improve displacement estimation and spatio-temporal continuity between time series ultrasound frames. The network is trained in an unsupervised way, by optimising a similarity metric between the reference and compressed image. Our training loss is also composed of a regularisation term that preserves displacement continuity by directly optimising the strain smoothness, and a temporal continuity term that enforces consistency between successive strain predictions. In addition, we propose an open-access in vivo database for quasi-static USE, which consists of radio-frequency data sequences captured on the arm of a human volunteer. Our results from numerical simulation and in vivo data suggest that our recurrent neural network can account for larger deformations, as compared with two other feed-forward neural networks. In all experiments, our recurrent network outperformed the state-of-the-art for both learning-based and optimisation-based methods, in terms of elastographic signal-to-noise ratio, strain consistency, and image similarity. Finally, our open-source code provides a 3D-slicer visualisation module that can be used to process ultrasound RF frames in real-time, at a rate of up to 20 frames per second, using a standard GPU.

Real-Time Volumetric Synthetic Aperture Software Beamforming of Row-Column Probe Data

Two delay-and-sum beamformers for 3-D synthetic aperture imaging with row-column addressed arrays are presented. Both beamformers are software implementations for graphics processing unit (GPU) execution with dynamic apodizations and third-order polynomial subsample interpolation. The first beamformer was written in the MATLAB programming language and the second was written in C/C++ with the compute unified device architecture (CUDA) extensions by NVIDIA. Performance was measured as volume rate and sample throughput on three different GPUs: a 1050 Ti, a 1080 Ti, and a TITAN V. The beamformers were evaluated across 112 combinations of output geometry, depth range, transducer array size, number of virtual sources, floating-point precision, and Nyquist rate or in-phase/quadrature beamforming using analytic signals. Real-time imaging defined as more than 30 volumes per second was attained by the CUDA beamformer on the three GPUs for 13, 27, and 43 setups, respectively. The MATLAB beamformer did not attain real-time imaging for any setup. The median, single-precision sample throughput of the CUDA beamformer was 4.9, 20.8, and 33.5 Gsamples/s on the three GPUs, respectively. The throughput of CUDA beamformer was an order of magnitude higher than that of the MATLAB beamformer.

A Real-Time, GPU-Based Implementation of Aperture Domain Model Image REconstruction

Multipath and off-axis scattering are two of the primary mechanisms for ultrasound image degradation. To address their impact, we have proposed Aperture Domain Model Image REconstruction (ADMIRE). This algorithm utilizes a model-based approach in order to identify and suppress sources of acoustic clutter. The ability of ADMIRE to suppress clutter and improve image quality has been demonstrated in previous works, but its use for real-time imaging has been infeasible due to its significant computational requirements. However, in recent years, the use of graphics processing units (GPUs) for general-purpose computing has enabled the significant acceleration of compute-intensive algorithms. This is because many modern GPUs have thousands of computational cores that can be utilized to perform massively parallel processing. Therefore, in this work, we have developed a GPU-based implementation of ADMIRE. The implementation on a single GPU provides a speedup of two orders of magnitude when compared to a serial CPU implementation, and additional speedup is achieved when the computations are distributed across two GPUs. In addition, we demonstrate the feasibility of the GPU implementation to be used for real-time imaging by interfacing it with a Verasonics Vantage 128 ultrasound research system. Moreover, we show that other beamforming techniques, such as delay-and-sum (DAS) and short-lag spatial coherence (SLSC), can be computed and simultaneously displayed with ADMIRE. The frame rate depends upon various parameters, and this is exhibited in the multiple imaging cases that are presented. An open-source code repository containing CPU and GPU implementations of ADMIRE is also provided.

Computer Assisted Planning for Curved Laser Interstitial Thermal Therapy

Laser interstitial thermal therapy (LiTT) is a minimally invasive alternative to conventional open surgery for drug-resistant focal mesial temporal lobe epilepsy (MTLE). Recent studies suggest that higher seizure freedom rates are correlated with maximal ablation of the mesial hippocampal head, whilst sparing of the parahippocampal gyrus (PHG) may reduce neuropsychological sequelae. Current commercially available laser catheters are inserted following manually planned straight-line trajectories, which cannot conform to curved brain structures, such as the hippocampus, without causing collateral damage or requiring multiple insertions. The clinical feasibility and potential of curved LiTT trajectories through steerable needles has yet to be investigated. This is the focus of our work. We propose a GPU-accelerated computer-assisted planning (CAP) algorithm for steerable needle insertions that generates optimized curved 3D trajectories with maximal ablation of the amygdalohippocampal complex and minimal collateral damage to nearby structures, while accounting for a variable ablation diameter ( 5-15mm). Simulated trajectories and ablations were performed on 5 patients with mesial temporal sclerosis (MTS), which were identified from a prospectively managed database. The algorithm generated obstacle-free paths with significantly greater target area ablation coverage and lower PHG ablation variance compared to straight line trajectories. The presented CAP algorithm returns increased ablation of the amygdalohippocampal complex, with lower patient risk scores compared to straight-line trajectories. This is the first clinical application of preoperative planning for steerable needle based LiTT. This study suggests that steerable needles have the potential to improve LiTT procedure efficacy whilst improving the safety and should thus be investigated further.

Application of machine learning in ophthalmic imaging modalities

In clinical ophthalmology, a variety of image-related diagnostic techniques have begun to offer unprecedented insights into eye diseases based on morphological datasets with millions of data points. Artificial intelligence (AI), inspired by the human multilayered neuronal system, has shown astonishing success within some visual and auditory recognition tasks. In these tasks, AI can analyze digital data in a comprehensive, rapid and non-invasive manner. Bioinformatics has become a focus particularly in the field of medical imaging, where it is driven by enhanced computing power and cloud storage, as well as utilization of novel algorithms and generation of data in massive quantities. Machine learning (ML) is an important branch in the field of AI. The overall potential of ML to automatically pinpoint, identify and grade pathological features in ocular diseases will empower ophthalmologists to provide high-quality diagnosis and facilitate personalized health care in the near future. This review offers perspectives on the origin, development, and applications of ML technology, particularly regarding its applications in ophthalmic imaging modalities.

Evaluation of a novel tomographic ultrasound device for abdominal examinations

Conventional ultrasound (US) is the first-line imaging method for abdominal pathologies, but its diagnostic accuracy is operator-dependent, and data storage is usually limited to two-dimensional images. A novel tomographic US system (Curefab CS, Munich, Germany) processes imaging data combined with three-dimensional spatial information using a magnetic field tracking. This enables standardized image presentation in axial planes and a review of the entire examination. The applicability and diagnostic performance of this tomographic US approach was analyzed in an abdominal setting using conventional US as reference. Tomographic US data were successfully compiled in all subjects of a training cohort (20 healthy volunteers) and in 50 patients with abdominal lesions. Image quality (35% and 79% for training and patient cohort respectively) and completeness of organ visualization (45% and 44%) were frequently impaired in tomographic US compared to conventional US. Conventional and tomographic US showed good agreement for measurement of organ sizes in the training cohort (right liver lobe and both kidneys with a median deviation of 5%). In the patient cohort, tomographic US identified 57 of 74 hepatic or renal lesions detected by conventional ultrasound (sensitivity 77%). In conclusion, this study illustrates the diagnostic potential of abdominal tomographic US, but current significant limitations of the tomographic ultrasound device demand further technical improvements before this and comparable approaches can be implemented in clinical practice.

Improved Visualization in Difficult-to-Image Stress Echocardiography Patients Using Real-Time Harmonic Spatial Coherence Imaging

Stress echocardiography is used to detect myocardial ischemia by evaluating cardiovascular function both at rest and at elevated heart rates. Stress echocardiography requires excellent visualization of the left ventricle (LV) throughout the cardiac cycle. However, LV endocardial border visualization is often negatively impacted by high levels of clutter associated with patient obesity, which has risen dramatically worldwide in recent decades. Short-lag spatial coherence (SLSC) imaging has demonstrated reduced clutter in several applications. In this work, a computationally efficient formulation of SLSC was implemented into an object-oriented graphics processing unit-based software beamformer, enabling real-time (>30 frames per second) SLSC echocardiography on a research ultrasound scanner. The system was then used to image 15 difficult-to-image stress echocardiography patients in a comparison study of tissue harmonic imaging (THI) and harmonic spatial coherence imaging (HSCI). Video clips of four standard stress echocardiography views acquired with either THI or HSCI were provided in random shuffled order to three experienced readers. Each reader rated the visibility of 17 LV segments as "invisible," "suboptimally visualized," or "well visualized," with the first two categories indicating a need for contrast agent. In a symmetry test unadjusted for patientwise clustering, HSCI demonstrated a clear superiority over THI ( ). When measured on a per-patient basis, the median total score significantly favored HSCI with . When collapsing the ratings to a two-level scale ("needs contrast" versus "well visualized"), HSCI once again showed an overall superiority over THI, with by McNemar test adjusted for clustering.

Embedded deep learning in ophthalmology: making ophthalmic imaging smarter

Deep learning has recently gained high interest in ophthalmology due to its ability to detect clinically significant features for diagnosis and prognosis. Despite these significant advances, little is known about the ability of various deep learning systems to be embedded within ophthalmic imaging devices, allowing automated image acquisition. In this work, we will review the existing and future directions for 'active acquisition'-embedded deep learning, leading to as high-quality images with little intervention by the human operator. In clinical practice, the improved image quality should translate into more robust deep learning-based clinical diagnostics. Embedded deep learning will be enabled by the constantly improving hardware performance with low cost. We will briefly review possible computation methods in larger clinical systems. Briefly, they can be included in a three-layer framework composed of edge, fog, and cloud layers, the former being performed at a device level. Improved egde-layer performance via 'active acquisition' serves as an automatic data curation operator translating to better quality data in electronic health records, as well as on the cloud layer, for improved deep learning-based clinical data mining.