Model-based correction of tissue compression for tracked ultrasound in soft tissue image-guided surgery

A review on organ deformation modeling approaches for reliable surgical navigation using augmented reality

Augmented Reality (AR) holds the potential to revolutionize surgical procedures by allowing surgeons to visualize critical structures within the patient's body. This is achieved through superimposing preoperative organ models onto the actual anatomy. Challenges arise from dynamic deformations of organs during surgery, making preoperative models inadequate for faithfully representing intraoperative anatomy. To enable reliable navigation in augmented surgery, modeling of intraoperative deformation to obtain an accurate alignment of the preoperative organ model with the intraoperative anatomy is indispensable. Despite the existence of various methods proposed to model intraoperative organ deformation, there are still few literature reviews that systematically categorize and summarize these approaches. This review aims to fill this gap by providing a comprehensive and technical-oriented overview of modeling methods for intraoperative organ deformation in augmented reality in surgery. Through a systematic search and screening process, 112 closely relevant papers were included in this review. By presenting the current status of organ deformation modeling methods and their clinical applications, this review seeks to enhance the understanding of organ deformation modeling in AR-guided surgery, and discuss the potential topics for future advancements.

Intraoperative patient-specific volumetric reconstruction and 3D visualization for laparoscopic liver surgery

Despite the benefits of minimally invasive surgery, interventions such as laparoscopic liver surgery present unique challenges, like the significant anatomical differences between preoperative images and intraoperative scenes due to pneumoperitoneum, patient pose, and organ manipulation by surgical instruments. To address these challenges, a method for intraoperative three-dimensional reconstruction of the surgical scene, including vessels and tumors, without altering the surgical workflow, is proposed. The technique combines neural radiance field reconstructions from tracked laparoscopic videos with ultrasound three-dimensional compounding. The accuracy of our reconstructions on a clinical laparoscopic liver ablation dataset, consisting of laparoscope and patient reference posed from optical tracking, laparoscopic and ultrasound videos, as well as preoperative and intraoperative computed tomographies, is evaluated. The authors propose a solution to compensate for liver deformations due to pressure applied during ultrasound acquisitions, improving the overall accuracy of the three-dimensional reconstructions compared to the ground truth intraoperative computed tomography with pneumoperitoneum. A unified neural radiance field from the ultrasound and laparoscope data, which allows real-time view synthesis providing surgeons with comprehensive intraoperative visual information for laparoscopic liver surgery, is trained.

Quantitative validation of two model-based methods for the correction of probe pressure deformation in ultrasound

PURPOSE

The acquisition of good quality ultrasound (US) images requires good acoustic coupling between the ultrasound probe and the patient's skin. In practice, this good coupling is achieved by the operator applying a force to the skin through the probe. This force induces a deformation of the tissues underlying the probe. The distorted images deteriorate the quality of the reconstructed 3D US image.

METHODS

In this work, we propose two methods to correct these deformations. These methods are based on the construction of a biomechanical model to predict the mechanical behavior of the imaged soft tissues. The originality of the methods is that they do not use external information (force or position value from sensors, or elasticity value from the literature). The model is parameterized thanks to the information contained in the image. This is allowed thanks to the optimization of two key parameters for the model which are the indentation d and the elasticity ratio α.

RESULTS

The validation is performed on real images acquired on a gelatin-based phantom using an ultrasound probe inducing an increasing vertical indentation using a step motor. The results showed a good correction of the two methods for indentations less than 4 mm. For larger indentations, one of the two methods (guided by the similarity score) provides a better quality of correction, presenting a Euclidean distance between the contours of the reference image and the corrected image of 0.71 mm.

CONCLUSION

The proposed methods ensured the correction of the deformed images induced by a linear probe pressure without using any information coming from sensors (force or position), or generic information about the mechanical parameters. The corrected images can be used to obtain a corrected 3D US image.

Computational Imaging to Compensate for Soft-Tissue Deformations in Image-Guided Breast Conserving Surgery

OBJECTIVE

During breast conserving surgery (BCS), magnetic resonance (MR) images aligned to accurately display intraoperative lesion locations can offer improved understanding of tumor extent and position relative to breast anatomy. Unfortunately, even under consistent supine conditions, soft tissue deformation compromises image-to-physical alignment and results in positional errors.

METHODS

A finite element inverse modeling technique has been developed to nonrigidly register preoperative supine MR imaging data to the surgical scene for improved localization accuracy during surgery. Registration is driven using sparse data compatible with acquisition during BCS, including corresponding surface fiducials, sparse chest wall contours, and the intra-fiducial skin surface. Deformation predictions were evaluated at surface fiducial locations and subsurface tissue features that were expertly identified and tracked. Among n = 7 different human subjects, an average of 22 ± 3 distributed subsurface targets were analyzed in each breast volume.

RESULTS

The average target registration error (TRE) decreased significantly when comparing rigid registration to this nonrigid approach (10.4 ± 2.3 mm vs 6.3 ± 1.4 mm TRE, respectively). When including a single subsurface feature as additional input data, the TRE significantly improved further (4.2 ± 1.0 mm TRE), and in a region of interest within 15 mm of a mock biopsy clip TRE was 3.9 ± 0.9 mm.

CONCLUSION

These results demonstrate accurate breast deformation estimates based on sparse-data-driven model predictions.

SIGNIFICANCE

The data suggest that a computational imaging approach can account for image-to-surgery shape changes to enhance surgical guidance during BCS.

Value of ultrasound elastography combined with contrast-enhanced ultrasound and micro-flow imaging in differential diagnosis of benign and malignant breast lesions

OBJECTIVE

Breast cancer is one of the most common malignant tumors in women and shows a rising incidence at younger ages. Therefore, early diagnosis is of great significance for treatment and prognosis. This study aimed to compare the value of ultrasound elastography (UE) combined with contrast-enhanced ultrasound (CEUS) and micro-flow imaging (MFI) in differential diagnosis of benign and malignant lesions of the breast.

METHODS

The sonographic characteristics of UE and CEUS as well as the vascular characteristics of MFI of 109 breast lesions categorized as Breast Imaging Reporting and Data System (BI-RADS) category 4, confirmed by surgical or biopsy pathology were retrospectively analyzed. Receiver operating characteristic (ROC) curves were used to compare the diagnostic efficacy of the three examination modalities, either alone or in combination.

RESULTS

Of the 109 breast lesions, 78 lesions were pathologically diagnosed as malignant and 31 as benign. At diagnosis, the area under the ROC curve (AUC), sensitivity, specificity, and accuracy of UE were 0.8495, 65.38%, 83.87% and 83.34%, respectively. The AUC, sensitivity, specificity and accuracy of MFI were 86.29%, 70.51%, 87.10% and 85.56%, respectively. The AUC, sensitivity, specificity and accuracy of CEUS were 90.84%, 88.46%, 74.19% and 89.16%, respectively. The AUC, sensitivity, specificity and accuracy of the combined diagnosis of UE, MFI, and CEUS were 93.90%, 85.90%, 90.32%, and 92.07%, respectively.

CONCLUSIONS

The combination of UE, CEUS and MFI has the highest specificity and accuracy in the differential diagnosis of benign and malignant breast lesions compared to any one used singly.

Biomechanical modelling of probe to tissue interaction during ultrasound scanning

PURPOSE

Biomechanical simulation of anatomical deformations caused by ultrasound probe pressure is of outstanding importance for several applications, from the testing of robotic acquisition systems to multi-modal image fusion and development of ultrasound training platforms. Different approaches can be exploited for modelling the probe-tissue interaction, each achieving different trade-offs among accuracy, computation time and stability.

METHODS

We assess the performances of different strategies based on the finite element method for modelling the interaction between the rigid probe and soft tissues. Probe-tissue contact is modelled using (i) penalty forces, (ii) constraint forces, and (iii) by prescribing the displacement of the mesh surface nodes. These methods are tested in the challenging context of ultrasound scanning of the breast, an organ undergoing large nonlinear deformations during the procedure.

RESULTS

The obtained results are evaluated against those of a non-physically based method. While all methods achieve similar accuracy, performance in terms of stability and speed shows high variability, especially for those methods modelling the contacts explicitly. Overall, prescribing surface displacements is the approach with best performances, but it requires prior knowledge of the contact area and probe trajectory.

CONCLUSIONS

In this work, we present different strategies for modelling probe-tissue interaction, each able to achieve different compromises among accuracy, speed and stability. The choice of the preferred approach highly depends on the requirements of the specific clinical application. Since the presented methodologies can be applied to describe general tool-tissue interactions, this work can be seen as a reference for researchers seeking the most appropriate strategy to model anatomical deformation induced by the interaction with medical tools.

Non-rigid registration of 3D ultrasound for neurosurgery using automatic feature detection and matching

PURPOSE

The brain undergoes significant structural change over the course of neurosurgery, including highly nonlinear deformation and resection. It can be informative to recover the spatial mapping between structures identified in preoperative surgical planning and the intraoperative state of the brain. We present a novel feature-based method for achieving robust, fully automatic deformable registration of intraoperative neurosurgical ultrasound images.

METHODS

A sparse set of local image feature correspondences is first estimated between ultrasound image pairs, after which rigid, affine and thin-plate spline models are used to estimate dense mappings throughout the image. Correspondences are derived from 3D features, distinctive generic image patterns that are automatically extracted from 3D ultrasound images and characterized in terms of their geometry (i.e., location, scale, and orientation) and a descriptor of local image appearance. Feature correspondences between ultrasound images are achieved based on a nearest-neighbor descriptor matching and probabilistic voting model similar to the Hough transform.

RESULTS

Experiments demonstrate our method on intraoperative ultrasound images acquired before and after opening of the dura mater, during resection and after resection in nine clinical cases. A total of 1620 automatically extracted 3D feature correspondences were manually validated by eleven experts and used to guide the registration. Then, using manually labeled corresponding landmarks in the pre- and post-resection ultrasound images, we show that our feature-based registration reduces the mean target registration error from an initial value of 3.3 to 1.5 mm.

CONCLUSIONS

This result demonstrates that the 3D features promise to offer a robust and accurate solution for 3D ultrasound registration and to correct for brain shift in image-guided neurosurgery.

Signal dropout correction-based ultrasound segmentation for diastolic mitral valve modeling

Three-dimensional ultrasound segmentation of mitral valve (MV) at diastole is helpful for duplicating geometry and pathology in a patient-specific dynamic phantom. The major challenge is the signal dropout at leaflet regions in transesophageal echocardiography image data. Conventional segmentation approaches suffer from missing sonographic data leading to inaccurate MV modeling at leaflet regions. This paper proposes a signal dropout correction-based ultrasound segmentation method for diastolic MV modeling. The proposed method combines signal dropout correction, image fusion, continuous max-flow segmentation, and active contour segmentation techniques. The signal dropout correction approach is developed to recover the missing segmentation information. Once the signal dropout regions of TEE image data are recovered, the MV model can be accurately duplicated. Compared with other methods in current literature, the proposed algorithm exhibits lower computational cost. The experimental results show that the proposed algorithm gives competitive results for diastolic MV modeling compared with conventional segmentation algorithms, evaluated in terms of accuracy and efficiency.

Evaluation of model-based deformation correction in image-guided liver surgery via tracked intraoperative ultrasound

Soft-tissue deformation represents a significant error source in current surgical navigation systems used for open hepatic procedures. While numerous algorithms have been proposed to rectify the tissue deformation that is encountered during open liver surgery, clinical validation of the proposed methods has been limited to surface-based metrics, and subsurface validation has largely been performed via phantom experiments. The proposed method involves the analysis of two deformation-correction algorithms for open hepatic image-guided surgery systems via subsurface targets digitized with tracked intraoperative ultrasound (iUS). Intraoperative surface digitizations were acquired via a laser range scanner and an optically tracked stylus for the purposes of computing the physical-to-image space registration and for use in retrospective deformation-correction algorithms. Upon completion of surface digitization, the organ was interrogated with a tracked iUS transducer where the iUS images and corresponding tracked locations were recorded. Mean closest-point distances between the feature contours delineated in the iUS images and corresponding three-dimensional anatomical model generated from preoperative tomograms were computed to quantify the extent to which the deformation-correction algorithms improved registration accuracy. The results for six patients, including eight anatomical targets, indicate that deformation correction can facilitate reduction in target error of [Formula: see text].

Continuous ultrasound speckle tracking with Gaussian mixtures

Speckle tracking echocardiography (STE) is now widely used for measuring strain, deformations, and motion in cardiology. STE involves three successive steps: acquisition of individual frames, speckle detection, and image registration using speckles as landmarks. This work proposes to avoid explicit detection and registration by representing dynamic ultrasound images as sparse collections of moving Gaussian elements in the continuous joint space-time space. Individual speckles or local clusters of speckles are approximated by a single multivariate Gaussian kernel with associated linear trajectory over a short time span. A hierarchical tree-structured model is fitted to sampled input data such that predicted image estimates can be retrieved by regression after reconstruction, allowing a (bias-variance) trade-off between model complexity and image resolution. The inverse image reconstruction problem is solved with an online Bayesian statistical estimation algorithm. Experiments on clinical data could estimate subtle sub-pixel accurate motion that is difficult to capture with frame-to-frame elastic image registration techniques.

Toward a generic real-time compression correction framework for tracked ultrasound

PURPOSE

Tissue compression during ultrasound imaging leads to error in the location and geometry of subsurface targets during soft tissue interventions. We present a novel compression correction method, which models a generic block of tissue and its subsurface tissue displacements resulting from application of a probe to the tissue surface. The advantages of the new method are that it can be realized independent of preoperative imaging data and is capable of near-video framerate compression compensation for real-time guidance.

METHODS

The block model is calibrated to the tip of any tracked ultrasound probe. Intraoperative digitization of the tissue surface is used to measure the depth of compression and provide boundary conditions to the biomechanical model of the tissue. The tissue displacement field solution of the model is inverted to nonrigidly transform the ultrasound images to an estimation of the tissue geometry prior to compression. This method was compared to a previously developed method using a patient-specific model and within the context of simulation, phantom, and clinical data.

RESULTS

Experimental results with gel phantoms demonstrated that the proposed generic method reduced the mock tumor margin modified Hausdorff distance (MHD) from 5.0 ± 1.6 to 2.1 ± 0.7 mm and reduced mock tumor centroid alignment error from 7.6 ± 2.6 to 2.6 ± 1.1mm. The method was applied to a clinical case and reduced the in vivo tumor margin MHD error from 5.4 ± 0.1 to 2.9 ± 0.1mm, and the centroid alignment error from 7.2 ± 0.2 to 3.8 ± 0.4 mm.

CONCLUSIONS

The correction method was found to effectively improve alignment of ultrasound and tomographic images and was more efficient compared to a previously proposed correction.

How Sensor, Signal, and Imaging Informatics May Impact Patient Centered Care and Care Coordination

OBJECTIVE

This synopsis presents a selection for the IMIA (International Medical Informatics Association) Yearbook 2015 of excellent research in the broad field of Sensor, Signal, and Imaging Informatics published in the year 2014, with a focus on patient centered care coordination.

METHODS

The two section editors performed a systematic initial selection and a double blind peer review process to select a list of candidate best papers in the domain published in 2014, from the PubMed and Web of Science databases. A set of MeSH keywords provided by experts was used. This selection was peer-reviewed by external reviewers.

RESULTS

The review process highlighted articles illustrating two current trends related to care coordination and patient centered care: the enhanced capacity to predict the evolution of a disease based on patient-specific information can impact care coordination; similarly, better perception of the patient and his treatment could lead to enhanced personalized care with a potential impact on care coordination.

CONCLUSIONS

This review shows the multiplicity of angles from which the question of patient-centered care can be addressed, with consequences on care coordination that will need to be confirmed and demonstrated in the future.

Realization of a biomechanical model-assisted image guidance system for breast cancer surgery using supine MRI

PURPOSE

Unfortunately, the current re-excision rates for breast conserving surgeries due to positive margins average 20-40 %. The high re-excision rates arise from difficulty in localizing tumor boundaries intraoperatively and lack of real-time information on the presence of residual disease. The work presented here introduces the use of supine magnetic resonance (MR) images, digitization technology, and biomechanical models to investigate the capability of using an image guidance system to localize tumors intraoperatively.

METHODS

Preoperative supine MR images were used to create patient-specific biomechanical models of the breast tissue, chest wall, and tumor. In a mock intraoperative setup, a laser range scanner was used to digitize the breast surface and tracked ultrasound was used to digitize the chest wall and tumor. Rigid registration combined with a novel nonrigid registration routine was used to align the preoperative and intraoperative patient breast and tumor. The registration framework is driven by breast surface data (laser range scan of visible surface), ultrasound chest wall surface, and MR-visible fiducials. Tumor localizations by tracked ultrasound were only used to evaluate the fidelity of aligning preoperative MR tumor contours to physical patient space. The use of tracked ultrasound to digitize subsurface features to constrain our nonrigid registration approach and to assess the fidelity of our framework makes this work unique. Two patient subjects were analyzed as a preliminary investigation toward the realization of this supine image-guided approach.

RESULTS

An initial rigid registration was performed using adhesive MR-visible fiducial markers for two patients scheduled for a lumpectomy. For patient 1, the rigid registration resulted in a root-mean-square fiducial registration error (FRE) of 7.5 mm and the difference between the intraoperative tumor centroid as visualized with tracked ultrasound imaging and the registered preoperative MR counterpart was 6.5 mm. Nonrigid correction resulted in a decrease in FRE to 2.9 mm and tumor centroid difference to 5.5 mm. For patient 2, rigid registration resulted in a FRE of 8.8 mm and a 3D tumor centroid difference of 12.5 mm. Following nonrigid correction for patient 2, the FRE was reduced to 7.4 mm and the 3D tumor centroid difference was reduced to 5.3 mm.

CONCLUSION

Using our prototype image-guided surgery platform, we were able to align intraoperative data with preoperative patient-specific models with clinically relevant accuracy; i.e., tumor centroid localizations of approximately 5.3-5.5 mm.