Automatic multimodal 2D/3D breast image registration using biomechanical FEM models and intensity-based optimization

Estimation of the biomechanical mammographic deformation of the breast using machine learning models

BACKGROUND

A typical problem in the registration of MRI and X-ray mammography is the nonlinear deformation applied to the breast during mammography. We have developed a method for virtual deformation of the breast using a biomechanical model automatically constructed from MRI. The virtual deformation is applied in two steps: unloaded state estimation and compression simulation. The finite element method is used to solve the deformation process. However, the extensive computational cost prevents its usage in clinical routine.

METHODS

We propose three machine learning models to overcome this problem: an extremely randomized tree (first model), extreme gradient boosting (second model), and deep learning-based bidirectional long short-term memory with an attention layer (third model) to predict the deformation of a biomechanical model. We evaluated our methods with 516 breasts with realistic compression ratios up to 76%.

FINDINGS

We first applied one-fold validation, in which the second and third models performed better than the first model. We then applied ten-fold validation. For the unloaded state estimation, the median RMSE for the second and third models is 0.8 mm and 1.2 mm, respectively. For the compression, the median RMSE is 3.4 mm for both models. We evaluated correlations between model accuracy and characteristics of the clinical datasets such as compression ratio, breast volume, and tissue types.

INTERPRETATION

Using the proposed models, we achieved accurate results comparable to the finite element model, with a speedup of factor 240 using the extreme gradient boosting model. These proposed models can replace the finite element model simulation, enabling clinically relevant real-time application.

A review of bioengineering techniques applied to breast tissue: Mechanical properties, tissue engineering and finite element analysis

Female breast cancer was the most prevalent cancer worldwide in 2020, according to the Global Cancer Observatory. As a prophylactic measure or as a treatment, mastectomy and lumpectomy are often performed at women. Following these surgeries, women normally do a breast reconstruction to minimize the impact on their physical appearance and, hence, on their mental health, associated with self-image issues. Nowadays, breast reconstruction is based on autologous tissues or implants, which both have disadvantages, such as volume loss over time or capsular contracture, respectively. Tissue engineering and regenerative medicine can bring better solutions and overcome these current limitations. Even though more knowledge needs to be acquired, the combination of biomaterial scaffolds and autologous cells appears to be a promising approach for breast reconstruction. With the growth and improvement of additive manufacturing, three dimensional (3D) printing has been demonstrating a lot of potential to produce complex scaffolds with high resolution. Natural and synthetic materials have been studied in this context and seeded mainly with adipose derived stem cells (ADSCs) since they have a high capability of differentiation. The scaffold must mimic the environment of the extracellular matrix (ECM) of the native tissue, being a structural support for cells to adhere, proliferate and migrate. Hydrogels (e.g., gelatin, alginate, collagen, and fibrin) have been a biomaterial widely studied for this purpose since their matrix resembles the natural ECM of the native tissues. A powerful tool that can be used in parallel with experimental techniques is finite element (FE) modeling, which can aid the measurement of mechanical properties of either breast tissues or scaffolds. FE models may help in the simulation of the whole breast or scaffold under different conditions, predicting what might happen in real life. Therefore, this review gives an overall summary concerning the human breast, specifically its mechanical properties using experimental and FE analysis, and the tissue engineering approaches to regenerate this particular tissue, along with FE models.

Incorporating heterogeneity and anisotropy for surgical applications in breast deformation modeling

BACKGROUND

Simulating soft-tissue breast deformations is of interest for many applications including image fusion, longitudinal registration, and image-guided surgery. For the surgical use case, positional changes cause breast deformations that compromise the use of preoperative imaging to inform tumor excision. Even when acquiring imaging in the supine position, which better reflects surgical presentation, deformations still occur due to arm motion and orientation changes. A biomechanical modeling approach to simulate supine breast deformations for surgical applications must be both accurate and compatible with the clinical workflow.

METHODS

A supine MR breast imaging dataset from n = 11 healthy volunteers was used to simulate surgical deformations by acquiring images in arm-down and arm-up positions. Three linear-elastic modeling approaches with varying levels of complexity were used to predict deformations caused by this arm motion: a homogeneous isotropic model, a heterogeneous isotropic model, and a heterogeneous anisotropic model using a transverse-isotropic constitutive model.

FINDINGS

The average target registration errors for subsurface anatomical features were 5.4 ± 1.5 mm for the homogeneous isotropic model, 5.3 ± 1.5 mm for the heterogeneous isotropic model, and 4.7 ± 1.4 mm for the heterogeneous anisotropic model. A statistically significant improvement in target registration error was observed between the heterogeneous anisotropic model and both the homogeneous and the heterogeneous isotropic models (P < 0.01).

INTERPRETATION

While a model that fully incorporates all constitutive complexities of anatomical structure likely achieves the best accuracy, a computationally tractable heterogeneous anisotropic model provided significant improvement and may be applicable for image-guided breast surgeries.

Geometric Mapping Evaluation for Real-Time Local Sensor Simulation

Medical augmented reality and simulated test environments struggle in accurately simulating local sensor measurements across large spatial domains while maintaining the proper resolution of information required and real time capability. Here, a simple method for real-time simulation of intraoperative sensors is presented to aid with medical sensor development and professional training. During a surgical intervention, the interaction between medical sensor systems and tissue leads to mechanical deformation of the tissue. Through the inclusion of detailed finite element simulations in a real-time augmented reality system the method presented will allow for more accurate simulation of intraoperative sensor measurements that are independent of the mechanical state of the tissue. This concept uses a coarse, macro-level deformation mesh to maintain both computational speed and the illusion of reality and a simple geometric point mapping method to include detailed fine mesh information. The resulting system allows for flexible simulation of different types of localized sensor measurement techniques. Preliminary simulation results are provided using a real-time capable simulation environment and prove the feasibility of the method.

Diagnosis of Breast Cancer Using Radiomics Models Built Based on Dynamic Contrast Enhanced MRI Combined With Mammography

Objective

To build radiomics models using features extracted from DCE-MRI and mammography for diagnosis of breast cancer.

Materials and Methods

266 patients receiving MRI and mammography, who had well-enhanced lesions on MRI and histologically confirmed diagnosis were analyzed. Training dataset had 146 malignant and 56 benign, and testing dataset had 48 malignant and 18 benign lesions. Fuzzy-C-means clustering algorithm was used to segment the enhanced lesion on subtraction MRI maps. Two radiologists manually outlined the corresponding lesion on mammography by consensus, with the guidance of MRI maximum intensity projection. Features were extracted using PyRadiomics from three DCE-MRI parametric maps, and from the lesion and a 2-cm bandshell margin on mammography. The support vector machine (SVM) was applied for feature selection and model building, using 5 datasets: DCE-MRI, mammography lesion-ROI, mammography margin-ROI, mammography lesion+margin, and all combined.

Results

In the training dataset evaluated using 10-fold cross-validation, the diagnostic accuracy of the individual model was 83.2% for DCE-MRI, 75.7% for mammography lesion, 64.4% for mammography margin, and 77.2% for lesion+margin. When all features were combined, the accuracy was improved to 89.6%. By adding mammography features to MRI, the specificity was significantly improved from 69.6% (39/56) to 82.1% (46/56), p<0.01. When the developed models were applied to the independent testing dataset, the accuracy was 78.8% for DCE-MRI and 83.3% for combined MRI+Mammography.

Conclusion

The radiomics model built from the combined MRI and mammography has the potential to provide a machine learning-based diagnostic tool and decrease the false positive diagnosis of contrast-enhanced benign lesions on MRI.

Multimodal Patient-Specific Registration for Breast Imaging Using Biomechanical Modeling with Reference to AI Evaluation of Breast Tumor Change

Background: The strategy to combat the problem associated with large deformations in the breast due to the difference in the medical imaging of patient posture plays a vital role in multimodal medical image registration with artificial intelligence (AI) initiatives. How to build a breast biomechanical model simulating the large-scale deformation of soft tissue remains a challenge but is highly desirable. Methods: This study proposed a hybrid individual-specific registration model of the breast combining finite element analysis, property optimization, and affine transformation to register breast images. During the registration process, the mechanical properties of the breast tissues were individually assigned using an optimization process, which allowed the model to become patient specific. Evaluation and results: The proposed method has been extensively tested on two datasets collected from two independent institutions, one from America and another from Hong Kong. Conclusions: Our method can accurately predict the deformation of breasts from the supine to prone position for both the Hong Kong and American samples, with a small target registration error of lesions.

Data assimilation of structural deformation using finite element simulations and single-perspective projection data based on the example of X-ray imaging

Gaining insight in the complex deformation processes inside a vehicle during the process of crashing is to date only possible via numerical simulations. To validate these simulations experimentally, high speed, high-energy X-ray imaging is currently developed. So far, X-ray images have only been compared qualitatively to simulations as quantitative analysis proves challenging: Three dimensional (3D) information has to be reconstructed from two dimensional (2D) X-ray images showing overlapping parts which undergo complex deformation. We propose a novel analysis method which adapts ideas from data assimilation to reconstruct three dimensional motion and deformation from two dimensional measurements using high speed X-ray video data and finite element (FE) simulations. Furthermore, we analyze the accuracy of the approach based on an observing system simulation experiment.

Deformable Mapping Method to Relate Lesions in Dedicated Breast CT Images to Those in Automated Breast Ultrasound and Digital Breast Tomosynthesis Images

This work demonstrates the potential for using a deformable mapping method to register lesions between dedicated breast computed tomography (bCT) and both automated breast ultrasound (ABUS) and digital breast tomosynthesis (DBT) images (craniocaudal [CC] and mediolateral oblique [MLO] views). Two multi-modality breast phantoms with external fiducial markers attached were imaged by the three modalities. The DBT MLO view was excluded for the second phantom. The automated deformable mapping algorithm uses biomechanical modeling to determine corresponding lesions based on distances between their centers of mass (d_COM) in the deformed bCT model and the reference model (DBT or ABUS). For bCT to ABUS, the mean d_COM was 5.2 ± 2.6 mm. For bCT to DBT (CC), the mean d_COM was 5.1 ± 2.4 mm. For bCT to DBT (MLO), the mean d_COM was 4.7 ± 2.5 mm. This application could help improve a radiologist's efficiency and accuracy in breast lesion characterization, using multiple imaging modalities.

Finite Element Breast Simulation for Sonography Image Registration

In case of female breast cancer, a breast conserving excision is often necessary. For this purpose, information from multiple medical imaging techniques have to be combined. Sonography imaging is essential for dense breast tissue and the only medical imaging technique available during surgery. During sonography of the outer breast quadrants the woman is usually in contralateral posterior oblique position, being in supine orientation while holding her ipsilateral arm over the head. Thus, these images cannot be directly registered with MRI or mammography images because these imaging technologies are performed in other patient positions with hands on the side of the body. Thus, we present a novel Finite Element approach how to enable a sonography image registration by showing the first time how to transfer the supine position with the arm straight on side into a supine position with the ipsilateral arm over the head which can be used to include information from MRI or mammography images. This approach is shown and validated with 3D scanner breast surface data as proof of concept. When comparing the simulation result with a 3D surface scan in supine orientation with the arm over the head, a mean surface distance error of 1.57 mm is achieved.

Towards CNN-Based Registration of Craniocaudal and Mediolateral Oblique 2-D X-ray Mammographic Images

We investigate methodologies for the automated registration of pairs of 2-D X-ray mammographic images, taken from the two standard mammographic angles. We present two exploratory techniques, based on Convolutional Neural Networks, to examine their potential for co-registration of findings on the two standard mammographic views. To test algorithm performance, our analysis uses a synthetic, surrogate data set for performing controlled experiments, as well as real 2-D X-ray mammogram imagery. The preliminary results are promising, and provide insights into how the proposed techniques may support multi-view X-ray mammography image registration currently and as technology evolves in the future.

Different Medical Image Registration Techniques: A Comparative Analysis

Background:

Image registration provides major role in real world applications and classic digital image processing. Image registration is carried out for more than one image and this image was captured from a different location, different sensors, different time and different viewpoints.

Discussion:

This paper deals with the comparative analysis of various registration techniques and here six registration techniques depending upon intensity, phase correlation, image feature, area, control points and mutual information are compared. Comparative analysis for different methodologies shows the advantages of one method over the other methods. The foremost objective of this paper is to deliver a complete reference source for the scholars interested in registration, irrespective of specific application extents.

Conclusion:

Finally performance analyses are evaluated for the medical datasets and comparison is graphically shown with the MATLAB simulation tool.

A novel finite element model-based navigation system-supported workflow for breast tumor excision

In the case of female breast cancer, a breast-conserving excision is often desirable. This surgery is based on preoperatively gathered MRI, mammography, and sonography images. These images are recorded in multiple patient positions, e. g., 2D mammography images in standing position with a compressed breast and 3D MRI images in prone position. In contrast, the surgery happens in supine or beach chair position. Due to these different perspectives and the flexible, thus challenging, breast tissue, the excision puts high demands on the physician. Therefore, this publication presents a novel eight-step excision support workflow that can be used to include information captured preoperatively through medical imaging based on a finite element (FE) model. In addition, an indoor positioning system is integrated in the workflow in order to track surgical devices and the sonography transducer during surgery. The preoperative part of the navigation system-supported workflow is outlined exemplarily based on first experimental results including 3D scans of a patient in different patient positions and her MRI images. Graphical Abstract Finite Element model based navigation system supported workflow for breast tumor excision is based on eight steps and allows inclusion of information from medical images recorded in multiple patient positions.

Deformable mapping technique to correlate lesions in digital breast tomosynthesis and automated breast ultrasound images

PURPOSE

To develop a deformable mapping technique to match corresponding lesions between digital breast tomosynthesis (DBT) and automated breast ultrasound (ABUS) images.

METHODS

External fiducial markers were attached to the surface of two CIRS multi-modality compressible breast phantoms (A and B) containing multiple simulated lesions. Both phantoms were imaged with DBT (upright positioning with cranial-caudal compression) and ABUS (supine positioning with anterior-to-chest wall compression). The lesions and markers were manually segmented by three different readers. Reader segmentation similarity and reader reproducibility were assessed using Dice similarity coefficients (DSC) and distances between centers of mass (d_COM ). For deformable mapping between the modalities each reader's segmented dataset was processed with an automated deformable mapping algorithm as follows: First, Morfeus, a finite element (FE) based multi-organ deformable image registration platform, converted segmentations into triangular surface meshes. Second, Altair HyperMesh, a FE pre-processor, created base FE models for the ABUS and DBT data sets. All deformation is performed on the DBT image data; the ABUS image sets remain fixed throughout the process. Deformation was performed on the external skin contour (DBT image set) to match the external skin contour on the ABUS set, and the locations of the external markers were used to morph the skin contours to be within a user-defined distance. Third, the base DBT-FE model was deformed with the FE analysis solver, Optistruct. Deformed DBT lesions were correlated with matching lesions in the base ABUS FE model. Performance (lesion correlation) was assessed with d_COM for all corresponding lesions and lesion overlap. Analysis was performed to determine the minimum number of external fiducial markers needed to create the desired correlation and the improvement of correlation with the use of external markers.

RESULTS

Average DSC for reader similarity ranged from 0.88 to 0.91 (ABUS) and 0.57 to 0.83 (DBT). Corresponding d_COM ranged from 0.20 to 0.36 mm (ABUS) and 0.11 to 1.16 mm (DBT). Lesion correlation is maximized when all corresponding markers are within a maximum distance of 5 mm. For deformable mapping of phantom A, without the use of external markers, only two of six correlated lesions showed overlap with an average lesion d_COM of 6.8 ± 2.8 mm. With use of three external fiducial markers, five of six lesions overlapped and average d_COM improved to 4.9 ± 2.4 mm. For deformable mapping of Phantom B without external markers analysis, four lesions were correlated of seven with overlap between only one of seven lesions, and an average lesion d_COM of 9.7 ± 3.5 mm. With three external markers, all seven possible lesions were correlated with overlap between four of seven lesions. The average d_COM was 8.5 ± 4.0 mm.

CONCLUSION

This work demonstrates the potential for a deformable mapping technique to relate corresponding lesions in DBT and ABUS images by showing improved lesion correspondence and reduced lesion registration errors with the use of external fiducial markers. The technique should improve radiologists' characterization of breast lesions which can reduce patient callbacks, misdiagnoses and unnecessary biopsies.

Multimodal Breast Parenchymal Patterns Correlation Using a Patient-Specific Biomechanical Model

In this paper, we aim to produce a realistic 2-D projection of the breast parenchymal distribution from a 3-D breast magnetic resonance image (MRI). To evaluate the accuracy of our simulation, we compare our results with the local breast density (i.e., density map) obtained from the complementary full-field digital mammogram. To achieve this goal, we have developed a fully automatic framework, which registers MRI volumes to X-ray mammograms using a subject-specific biomechanical model of the breast. The optimization step modifies the position, orientation, and elastic parameters of the breast model to perform the alignment between the images. When the model reaches an optimal solution, the MRI glandular tissue is projected and compared with the one obtained from the corresponding mammograms. To reduce the loss of information during the ray-casting, we introduce a new approach that avoids resampling the MRI volume. In the results, we focus our efforts on evaluating the agreement of the distributions of glandular tissue, the degree of structural similarity, and the correlation between the real and synthetic density maps. Our approach obtained a high-structural agreement regardless the glandularity of the breast, whilst the similarity of the glandular tissue distributions and correlation between both images increase in denser breasts. Furthermore, the synthetic images show continuity with respect to large structures in the density maps.

A system for automatic monitoring of surgical instruments and dynamic, non-rigid surface deformations in breast cancer surgery

When negative tumor margins are achieved at the time of resection, breast conserving therapy (lumpectomy followed with radiation therapy) offers patients improved cosmetic outcomes and quality of life with equivalent survival outcomes to mastectomy. However, high reoperation rates ranging 10-59% continue to challenge adoption and suggest that improved intraoperative tumor localization is a pressing need. We propose to couple an optical tracker and stereo camera system for automated monitoring of surgical instruments and non-rigid breast surface deformations. A bracket was designed to rigidly pair an optical tracker with a stereo camera, optimizing overlap volume. Utilizing both devices allowed for precise instrument tracking of multiple objects with reliable, workflow friendly tracking of dynamic breast movements. Computer vision techniques were employed to automatically track fiducials, requiring one-time initialization with bounding boxes in stereo camera images. Point based rigid registration was performed between fiducial locations triangulated from stereo camera images and fiducial locations recorded with an optically tracked stylus. We measured fiducial registration error (FRE) and target registration error (TRE) with two different stereo camera devices using a phantom breast with five fiducials. Average FREs of 2.7 ± 0.4 mm and 2.4 ± 0.6 mm with each stereo-camera device demonstrate considerable promise for this approach in monitoring the surgical field. Automated tracking was shown to reduce error when compared to manually selected fiducial locations in stereo camera image-based localization. The proposed instrumentation framework demonstrated potential for the continuous measurement of surgical instruments in relation to the dynamic deformations of a breast during lumpectomy.

A Regression Model for Predicting Shape Deformation after Breast Conserving Surgery

Breast cancer treatments can have a negative impact on breast aesthetics, in case when surgery is intended to intersect tumor. For many years mastectomy was the only surgical option, but more recently breast conserving surgery (BCS) has been promoted as a liable alternative to treat cancer while preserving most part of the breast. However, there is still a significant number of BCS intervened patients who are unpleasant with the result of the treatment, which leads to self-image issues and emotional overloads. Surgeons recognize the value of a tool to predict the breast shape after BCS to facilitate surgeon/patient communication and allow more educated decisions; however, no such tool is available that is suited for clinical usage. These tools could serve as a way of visually sensing the aesthetic consequences of the treatment. In this research, it is intended to propose a methodology for predict the deformation after BCS by using machine learning techniques. Nonetheless, there is no appropriate dataset containing breast data before and after surgery in order to train a learning model. Therefore, an in-house semi-synthetic dataset is proposed to fulfill the requirement of this research. Using the proposed dataset, several learning methodologies were investigated, and promising outcomes are obtained.

Morphometric study of adipocytes on breast cancer by means of photonic microscopy and image analysis

Worldwide, breast cancer (BrCa) is currently the leading cause of deaths associated to malignant lesions in adult women. Given that some studies have mentioned that peritumoral adipocytes may contribute to breast carcinogenesis, present work sought to quantitative evaluate the morphometry of these cells in a group of adult women. Three thousand six hundred sixty four breast adipocytes, that came from biopsies of a group of adult females with different types of breast carcinomas (ductal, lobular, and mixed) and one with normal tissues, were evaluated through an image analysis (IA) process regarding six morphometric descriptors: area (A), perimeter (P), Feret diameter (F_D ), aspect ratio (AR), roundness factor (RF), and fractal dimension of cellular contour (FD_C ). Data showed that the adipocytes of the normal tissues group were bigger (A: 3398 ± 2331 µm² , P: 239 ± 83 µm, and F_D : 79.9 ± 24.5 µm) than those from BrCa samples (A: 2860 ± 1933 µm² , P: 214 ± 66 µm, and F_D : 73.2 ± 22.5 µm), and presented a more irregular contour (FD_C of 1.370 ± 0.037 for normal group and of 1.335 ± 0.049 for the oncologic one). Moreover, it could be accounted that adipocytes of mixed carcinomas were largest (F_D : 75.1 ± 22.4 µm) than those of lobular lesions (F_D : 61.6 ± 22.6 µm), while the adipocytes of ductal carcinomas were the most oval (AR: 1.421 ± 0.524) and roughest (FD_C : 1.324 ± 0.050) cells. IA results suggest that BrCa lesions can be categorized through a quantitative morphometric evaluation of peritumoral adipocytes. These findings could let the development of an analytical tool to help the Pathologist to enhance the accuracy of the oncologic diagnose.

Towards an in-plane methodology to track breast lesions using mammograms and patient-specific finite-element simulations

In breast cancer screening or diagnosis, it is usual to combine different images in order to locate a lesion as accurately as possible. These images are generated using a single or several imaging techniques. As x-ray-based mammography is widely used, a breast lesion is located in the same plane of the image (mammogram), but tracking it across mammograms corresponding to different views is a challenging task for medical physicians. Accordingly, simulation tools and methodologies that use patient-specific numerical models can facilitate the task of fusing information from different images. Additionally, these tools need to be as straightforward as possible to facilitate their translation to the clinical area. This paper presents a patient-specific, finite-element-based and semi-automated simulation methodology to track breast lesions across mammograms. A realistic three-dimensional computer model of a patient's breast was generated from magnetic resonance imaging to simulate mammographic compressions in cranio-caudal (CC, head-to-toe) and medio-lateral oblique (MLO, shoulder-to-opposite hip) directions. For each compression being simulated, a virtual mammogram was obtained and posteriorly superimposed to the corresponding real mammogram, by sharing the nipple as a common feature. Two-dimensional rigid-body transformations were applied, and the error distance measured between the centroids of the tumors previously located on each image was 3.84 mm and 2.41 mm for CC and MLO compression, respectively. Considering that the scope of this work is to conceive a methodology translatable to clinical practice, the results indicate that it could be helpful in supporting the tracking of breast lesions.

A finite element-based machine learning approach for modeling the mechanical behavior of the breast tissues under compression in real-time

This work presents a data-driven method to simulate, in real-time, the biomechanical behavior of the breast tissues in some image-guided interventions such as biopsies or radiotherapy dose delivery as well as to speed up multimodal registration algorithms. Ten real breasts were used for this work. Their deformation due to the displacement of two compression plates was simulated off-line using the finite element (FE) method. Three machine learning models were trained with the data from those simulations. Then, they were used to predict in real-time the deformation of the breast tissues during the compression. The models were a decision tree and two tree-based ensemble methods (extremely randomized trees and random forest). Two different experimental setups were designed to validate and study the performance of these models under different conditions. The mean 3D Euclidean distance between nodes predicted by the models and those extracted from the FE simulations was calculated to assess the performance of the models in the validation set. The experiments proved that extremely randomized trees performed better than the other two models. The mean error committed by the three models in the prediction of the nodal displacements was under 2 mm, a threshold usually set for clinical applications. The time needed for breast compression prediction is sufficiently short to allow its use in real-time (<0.2 s).

Efficient Segmentation of a Breast in B-Mode Ultrasound Tomography Using Three-Dimensional GrabCut (GC3D)

As an emerging modality for whole breast imaging, ultrasound tomography (UST), has been adopted for diagnostic purposes. Efficient segmentation of an entire breast in UST images plays an important role in quantitative tissue analysis and cancer diagnosis, while major existing methods suffer from considerable time consumption and intensive user interaction. This paper explores three-dimensional GrabCut (GC3D) for breast isolation in thirty reflection (B-mode) UST volumetric images. The algorithm can be conveniently initialized by localizing points to form a polygon, which covers the potential breast region. Moreover, two other variations of GrabCut and an active contour method were compared. Algorithm performance was evaluated from volume overlap ratios ( T O , target overlap; M O , mean overlap; F P , false positive; F N , false negative) and time consumption. Experimental results indicate that GC3D considerably reduced the work load and achieved good performance ( T O = 0.84; M O = 0.91; F P = 0.006; F N = 0.16) within an average of 1.2 min per volume. Furthermore, GC3D is not only user friendly, but also robust to various inputs, suggesting its great potential to facilitate clinical applications during whole-breast UST imaging. In the near future, the implemented GC3D can be easily automated to tackle B-mode UST volumetric images acquired from the updated imaging system.

Personalized heterogeneous deformable model for fast volumetric registration

BACKGROUND

Biomechanical deformable volumetric registration can help improve safety of surgical interventions by ensuring the operations are extremely precise. However, this technique has been limited by the accuracy and the computational efficiency of patient-specific modeling.

METHODS

This study presents a tissue-tissue coupling strategy based on penalty method to model the heterogeneous behavior of deformable body, and estimate the personalized tissue-tissue coupling parameters in a data-driven way. Moreover, considering that the computational efficiency of biomechanical model is highly dependent on the mechanical resolution, a practical coarse-to-fine scheme is proposed to increase runtime efficiency. Particularly, a detail enrichment database is established in an offline fashion to represent the mapping relationship between the deformation results of high-resolution hexahedral mesh extracted from the raw medical data and a newly constructed low-resolution hexahedral mesh. At runtime, the mechanical behavior of human organ under interactions is simulated with this low-resolution hexahedral mesh, then the microstructures are synthesized in virtue of the detail enrichment database.

RESULTS

The proposed method is validated by volumetric registration in an abdominal phantom compression experiments. Our personalized heterogeneous deformable model can well describe the coupling effects between different tissues of the phantom. Compared with high-resolution heterogeneous deformable model, the low-resolution deformable model with our detail enrichment database can achieve 9.4× faster, and the average target registration error is 3.42 mm, which demonstrates that the proposed method shows better volumetric registration performance than state-of-the-art.

CONCLUSIONS

Our framework can well balance the precision and efficiency, and has great potential to be adopted in the practical augmented reality image-guided robotic systems.

Finite-element modeling of compression and gravity on a population of breast phantoms for multimodality imaging simulation

PURPOSE

The authors are developing a series of computational breast phantoms based on breast CT data for imaging research. In this work, the authors develop a program that will allow a user to alter the phantoms to simulate the effect of gravity and compression of the breast (craniocaudal or mediolateral oblique) making the phantoms applicable to multimodality imaging.

METHODS

This application utilizes a template finite-element (FE) breast model that can be applied to their presegmented voxelized breast phantoms. The FE model is automatically fit to the geometry of a given breast phantom, and the material properties of each element are set based on the segmented voxels contained within the element. The loading and boundary conditions, which include gravity, are then assigned based on a user-defined position and compression. The effect of applying these loads to the breast is computed using a multistage contact analysis in FEBio, a freely available and well-validated FE software package specifically designed for biomedical applications. The resulting deformation of the breast is then applied to a boundary mesh representation of the phantom that can be used for simulating medical images. An efficient script performs the above actions seamlessly. The user only needs to specify which voxelized breast phantom to use, the compressed thickness, and orientation of the breast.

RESULTS

The authors utilized their FE application to simulate compressed states of the breast indicative of mammography and tomosynthesis. Gravity and compression were simulated on example phantoms and used to generate mammograms in the craniocaudal or mediolateral oblique views. The simulated mammograms show a high degree of realism illustrating the utility of the FE method in simulating imaging data of repositioned and compressed breasts.

CONCLUSIONS

The breast phantoms and the compression software can become a useful resource to the breast imaging research community. These phantoms can then be used to evaluate and compare imaging modalities that involve different positioning and compression of the breast.

Population of 224 realistic human subject-based computational breast phantoms

PURPOSE

To create a database of highly realistic and anatomically variable 3D virtual breast phantoms based on dedicated breast computed tomography (bCT) data.

METHODS

A tissue classification and segmentation algorithm was used to create realistic and detailed 3D computational breast phantoms based on 230 + dedicated bCT datasets from normal human subjects. The breast volume was identified using a coarse three-class fuzzy C-means segmentation algorithm which accounted for and removed motion blur at the breast periphery. Noise in the bCT data was reduced through application of a postreconstruction 3D bilateral filter. A 3D adipose nonuniformity (bias field) correction was then applied followed by glandular segmentation using a 3D bias-corrected fuzzy C-means algorithm. Multiple tissue classes were defined including skin, adipose, and several fractional glandular densities. Following segmentation, a skin mask was produced which preserved the interdigitated skin, adipose, and glandular boundaries of the skin interior. Finally, surface modeling was used to produce digital phantoms with methods complementary to the XCAT suite of digital human phantoms.

RESULTS

After rejecting some datasets due to artifacts, 224 virtual breast phantoms were created which emulate the complex breast parenchyma of actual human subjects. The volume breast density (with skin) ranged from 5.5% to 66.3% with a mean value of 25.3% ± 13.2%. Breast volumes ranged from 25.0 to 2099.6 ml with a mean value of 716.3 ± 386.5 ml. Three breast phantoms were selected for imaging with digital compression (using finite element modeling) and simple ray-tracing, and the results show promise in their potential to produce realistic simulated mammograms.

CONCLUSIONS

This work provides a new population of 224 breast phantoms based on in vivo bCT data for imaging research. Compared to previous studies based on only a few prototype cases, this dataset provides a rich source of new cases spanning a wide range of breast types, volumes, densities, and parenchymal patterns.

Towards Personalized Statistical Deformable Model and Hybrid Point Matching for Robust MR-TRUS Registration

Registration and fusion of magnetic resonance (MR) and 3D transrectal ultrasound (TRUS) images of the prostate gland can provide high-quality guidance for prostate interventions. However, accurate MR-TRUS registration remains a challenging task, due to the great intensity variation between two modalities, the lack of intrinsic fiducials within the prostate, the large gland deformation caused by the TRUS probe insertion, and distinctive biomechanical properties in patients and prostate zones. To address these challenges, a personalized model-to-surface registration approach is proposed in this study. The main contributions of this paper can be threefold. First, a new personalized statistical deformable model (PSDM) is proposed with the finite element analysis and the patient-specific tissue parameters measured from the ultrasound elastography. Second, a hybrid point matching method is developed by introducing the modality independent neighborhood descriptor (MIND) to weight the Euclidean distance between points to establish reliable surface point correspondence. Third, the hybrid point matching is further guided by the PSDM for more physically plausible deformation estimation. Eighteen sets of patient data are included to test the efficacy of the proposed method. The experimental results demonstrate that our approach provides more accurate and robust MR-TRUS registration than state-of-the-art methods do. The averaged target registration error is 1.44 mm, which meets the clinical requirement of 1.9 mm for the accurate tumor volume detection. It can be concluded that the presented method can effectively fuse the heterogeneous image information in the elastography, MR, and TRUS to attain satisfactory image alignment performance.

A neural network-based 2D/3D image registration quality evaluator for pediatric patient setup in external beam radiotherapy

Our purpose was to develop a neural network‐based registration quality evaluator (RQE) that can improve the 2D/3D image registration robustness for pediatric patient setup in external beam radiotherapy. Orthogonal daily setup X‐ray images of six pediatric patients with brain tumors receiving proton therapy treatments were retrospectively registered with their treatment planning computed tomography (CT) images. A neural network‐based pattern classifier was used to determine whether a registration solution was successful based on geometric features of the similarity measure values near the point‐of‐solution. Supervised training and test datasets were generated by rigidly registering a pair of orthogonal daily setup X‐ray images to the treatment planning CT. The best solution for each registration task was selected from 50 optimizing attempts that differed only by the randomly generated initial transformation parameters. The distance from each individual solution to the best solution in the normalized parametrical space was compared to a user‐defined error tolerance to determine whether that solution was acceptable. A supervised training was then used to train the RQE. Performance of the RQE was evaluated using test dataset consisting of registration results that were not used in training. The RQE was integrated with our in‐house 2D/3D registration system and its performance was evaluated using the same patient dataset. With an optimized sampling step size (i.e., 5 mm) in the feature space, the RQE has the sensitivity and the specificity in the ranges of 0.865–0.964 and 0.797–0.990, respectively, when used to detect registration error with mean voxel displacement (MVD) greater than 1 mm. The trial‐to‐acceptance ratio of the integrated 2D/3D registration system, for all patients, is equal to 1.48. The final acceptance ratio is 92.4%. The proposed RQE can potentially be used in a 2D/3D rigid image registration system to improve the overall robustness by rejecting unsuccessful registration solutions. The RQE is not patient‐specific, so a single RQE can be constructed and used for a particular application (e.g., the registration for images acquired on the same anatomical site). Implementation of the RQE in a 2D/3D registration system is clinically feasible.

PACS numbers: 87.57.nj, 87.85.dq, 87.55.Qr

An Anthropometric-Based Subject-Specific Finite Element Model of the Human Breast for Predicting Large Deformations

The large deformation of the human breast threatens proper nodules tracking when the subject mammograms are used as pre-planning data for biopsy. However, techniques capable of accurately supporting the surgeons during biopsy are missing. Finite element (FE) models are at the basis of currently investigated methodologies to track nodules displacement. Nonetheless, the impact of breast material modeling on the mechanical response of its tissues (e.g., tumors) is not clear. This study proposes a subject-specific FE model of the breast, obtained by anthropometric measurements, to predict breast large deformation. A healthy breast subject-specific FE parametric model was developed and validated by Cranio-caudal (CC) and Medio-Lateral Oblique (MLO) mammograms. The model was successively modified, including nodules, and utilized to investigate the effect of nodules size, typology, and material modeling on nodules shift under the effect of CC, MLO, and gravity loads. Results show that a Mooney–Rivlin material model can estimate healthy breast large deformation. For a pathological breast, under CC compression, the nodules displacement is very close to zero when a linear elastic material model is used. Finally, when nodules are modeled, including tumor material properties, under CC, or MLO or gravity loads, nodules shift shows ~15% average relative difference.

A review of biomechanically informed breast image registration

Breast radiology encompasses the full range of imaging modalities from routine imaging via x-ray mammography, magnetic resonance imaging and ultrasound (both two- and three-dimensional), to more recent technologies such as digital breast tomosynthesis, and dedicated breast imaging systems for positron emission mammography and ultrasound tomography. In addition new and experimental modalities, such as Photoacoustics, Near Infrared Spectroscopy and Electrical Impedance Tomography etc, are emerging. The breast is a highly deformable structure however, and this greatly complicates visual comparison of imaging modalities for the purposes of breast screening, cancer diagnosis (including image guided biopsy), tumour staging, treatment monitoring, surgical planning and simulation of the effects of surgery and wound healing etc. Due primarily to the challenges posed by these gross, non-rigid deformations, development of automated methods which enable registration, and hence fusion, of information within and across breast imaging modalities, and between the images and the physical space of the breast during interventions, remains an active research field which has yet to translate suitable methods into clinical practice. This review describes current research in the field of breast biomechanical modelling and identifies relevant publications where the resulting models have been incorporated into breast image registration and simulation algorithms. Despite these developments there remain a number of issues that limit clinical application of biomechanical modelling. These include the accuracy of constitutive modelling, implementation of representative boundary conditions, failure to meet clinically acceptable levels of computational cost, challenges associated with automating patient-specific model generation (i.e. robust image segmentation and mesh generation) and the complexity of applying biomechanical modelling methods in routine clinical practice.

Methodology based on genetic heuristics for in-vivo characterizing the patient-specific biomechanical behavior of the breast tissues

This paper presents a novel methodology to in-vivo estimate the elastic constants of a constitutive model proposed to characterize the mechanical behavior of the breast tissues. An iterative search algorithm based on genetic heuristics was constructed to in-vivo estimate these parameters using only medical images, thus avoiding invasive measurements of the mechanical response of the breast tissues. For the first time, a combination of overlap and distance coefficients were used for the evaluation of the similarity between a deformed MRI of the breast and a simulation of that deformation. The methodology was validated using breast software phantoms for virtual clinical trials, compressed to mimic MRI-guided biopsies. The biomechanical model chosen to characterize the breast tissues was an anisotropic neo-Hookean hyperelastic model. Results from this analysis showed that the algorithm is able to find the elastic constants of the constitutive equations of the proposed model with a mean relative error of about 10%. Furthermore, the overlap between the reference deformation and the simulated deformation was of around 95% showing the good performance of the proposed methodology. This methodology can be easily extended to characterize the real biomechanical behavior of the breast tissues, which means a great novelty in the field of the simulation of the breast behavior for applications such as surgical planing, surgical guidance or cancer diagnosis. This reveals the impact and relevance of the presented work.

Numerical simulation of pressure therapy glove by using Finite Element Method

Pressure therapy garments apply pressure to suppress the growth and flatten hypertrophic scars caused by serious burns. The amount of pressure given by the pressure garments is critical to the treatment adherence and outcomes. In the present study, a biomechanical model for simulating the pressure magnitudes and distribution over hand dorsum given by a pressure glove was developed by using finite element method. In this model, the shape geometry of the hand, the mechanical properties of the glove and human body tissues were incorporated in the numerical stress analyses. The geometry of the hand was obtained by a 3D laser scanner. The material properties of two warp knitted fabrics were considered in the glove fabric model that developed from the glove production pattern with 10% size reduction in circumferential dimensions. The glove was regarded an isotropic elastic shell and the hand was assumed to be a homogeneous, isotropic and linearly elastic body. A glove wearing process was carried in the finite element analysis and the surface-to-surface contact pressure between hand and glove fabric was hence obtained. Through validation, the simulated contact pressure showed a good agreement with the experimental interface pressure measurement. The simulation model can be used to predict and visualise the pressure distribution exerted by a pressure therapy glove onto hand dorsum. It can provide information for optimising the material mechanical properties in pressure garment design and development, give a clue to understand the mechanisms of pressure action on hypertrophic scars and ultimately improve the medical functions of pressure garment.

Identification of Breast Cancer Using Integrated Information from MRI and Mammography

Objectives

Integration of information from corresponding regions between the breast MRI and an X-ray mammogram could benefit the detection of breast cancer in clinical diagnosis. We aimed to provide a framework of registration from breast MRI to mammography and to evaluate the diagnosis using the combined information.

Materials and Methods

43 patients with 46 lesions underwent both MRI and mammography scans, and the interval between the two examinations was around one month. The distribution of malignant to benign lesions was 31/46 based on histological results. Maximum intensity projection and thin-plate spline methods were applied for image registration for MRI to mammography. The diagnosis using integrated information was evaluated using results of histology as the reference. The assessment of annotations and statistical analysis were performed by the two radiologists.

Results

For the cranio-caudal view, the mean post-registration error between MRI and mammography was 2.2±1.9 mm. For the medio-lateral oblique view, the proposed approach performed even better with a mean error of 3.0±2.4 mm. In the diagnosis using MRI assessment with information of mammography, the sensitivity was 91.9±2.3% (29/31, 28/31), specificity 70.0±4.7% (11/15, 10/15), accuracy 84.8±3.1% (40/46, 38/46), positive predictive value 86.4±2.1% (29/33, 28/33) and negative predictive value 80.8±5.4% (11/13, 10/13).

Conclusion

MRI with the aid of mammography shows potential improvements of sensitivity, specificity, accuracy, PPV and NPV in clinical breast cancer diagnosis compared to the use of MRI alone.

Patient-specific biomechanical model as whole-body CT image registration tool

Whole-body computed tomography (CT) image registration is important for cancer diagnosis, therapy planning and treatment. Such registration requires accounting for large differences between source and target images caused by deformations of soft organs/tissues and articulated motion of skeletal structures. The registration algorithms relying solely on image processing methods exhibit deficiencies in accounting for such deformations and motion. We propose to predict the deformations and movements of body organs/tissues and skeletal structures for whole-body CT image registration using patient-specific non-linear biomechanical modelling. Unlike the conventional biomechanical modelling, our approach for building the biomechanical models does not require time-consuming segmentation of CT scans to divide the whole body into non-overlapping constituents with different material properties. Instead, a Fuzzy C-Means (FCM) algorithm is used for tissue classification to assign the constitutive properties automatically at integration points of the computation grid. We use only very simple segmentation of the spine when determining vertebrae displacements to define loading for biomechanical models. We demonstrate the feasibility and accuracy of our approach on CT images of seven patients suffering from cancer and aortic disease. The results confirm that accurate whole-body CT image registration can be achieved using a patient-specific non-linear biomechanical model constructed without time-consuming segmentation of the whole-body images.

Image fusion of Ultrasound Computer Tomography volumes with X-ray mammograms using a biomechanical model based 2D/3D registration

Ultrasound Computer Tomography (USCT) is a promising breast imaging modality under development. Comparison to a standard method like mammography is essential for further development. Due to significant differences in image dimensionality and compression state of the breast, correlating USCT images and X-ray mammograms is challenging. In this paper we present a 2D/3D registration method to improve the spatial correspondence and allow direct comparison of the images. It is based on biomechanical modeling of the breast and simulation of the mammographic compression. We investigate the effect of including patient-specific material parameters estimated automatically from USCT images. The method was systematically evaluated using numerical phantoms and in-vivo data. The average registration accuracy using the automated registration was 11.9mm. Based on the registered images a method for analysis of the diagnostic value of the USCT images was developed and initially applied to analyze sound speed and attenuation images based on X-ray mammograms as ground truth. Combining sound speed and attenuation allows differentiating lesions from surrounding tissue. Overlaying this information on mammograms, combines quantitative and morphological information for multimodal diagnosis.

Simulation of mammographic breast compression in 3D MR images using ICP-based B-spline deformation for multimodality breast cancer diagnosis

PURPOSE

Multimodality mammography using conventional 2D mammography and dynamic contrast-enhanced 3D magnetic resonance imaging (DCE-MRI) is frequently performed for breast cancer detection and diagnosis. Combination of both imaging modalities requires superimposition of corresponding structures in mammograms and MR images. This task is challenging due to large differences in (1) dimensionality and spatial resolution, (2) variations in tissue contrast, as well as (3) differences in breast orientation and deformation during the image acquisition. A new method for multimodality breast image registration was developed and tested.

METHODS

Combined diagnosis of mammograms and MRI datasets was achieved by simulation of mammographic breast compression to overcome large differences in breast deformation. Surface information was extracted from the 3D MR image, and back-projection of the 2D breast contour in the mammogram was done. B-spline-based 3D/3D surface-based registration was then used to approximate mammographic breast compression. This breast deformation simulation was performed on 14 MRI datasets with 19 corresponding mammograms. The results were evaluated by comparison with distances between corresponding structures identified by an expert observer.

RESULTS

The evaluation revealed an average distance of 6.46 mm between corresponding structures, when an optimized initial alignment between both image datasets is performed. Without the optimization, the accuracy is 9.12 mm.

CONCLUSION

A new surface-based method that approximates the mammographic deformation due to breast compression without using a specific complex model needed for finite-element-based methods was developed and tested with favorable results. The simulated compression can serve as foundation for a point-to-line correspondence between 2D mammograms and 3D MR image data.

Robust 3D-2D image registration: application to spine interventions and vertebral labeling in the presence of anatomical deformation

We present a framework for robustly estimating registration between a 3D volume image and a 2D projection image and evaluate its precision and robustness in spine interventions for vertebral localization in the presence of anatomical deformation. The framework employs a normalized gradient information similarity metric and multi-start covariance matrix adaptation evolution strategy optimization with local-restarts, which provided improved robustness against deformation and content mismatch. The parallelized implementation allowed orders-of-magnitude acceleration in computation time and improved the robustness of registration via multi-start global optimization. Experiments involved a cadaver specimen and two CT datasets (supine and prone) and 36 C-arm fluoroscopy images acquired with the specimen in four positions (supine, prone, supine with lordosis, prone with kyphosis), three regions (thoracic, abdominal, and lumbar), and three levels of geometric magnification (1.7, 2.0, 2.4). Registration accuracy was evaluated in terms of projection distance error (PDE) between the estimated and true target points in the projection image, including 14 400 random trials (200 trials on the 72 registration scenarios) with initialization error up to ±200 mm and ±10°. The resulting median PDE was better than 0.1 mm in all cases, depending somewhat on the resolution of input CT and fluoroscopy images. The cadaver experiments illustrated the tradeoff between robustness and computation time, yielding a success rate of 99.993% in vertebral labeling (with 'success' defined as PDE <5 mm) using 1,718 664 ± 96 582 function evaluations computed in 54.0 ± 3.5 s on a mid-range GPU (nVidia, GeForce GTX690). Parameters yielding a faster search (e.g., fewer multi-starts) reduced robustness under conditions of large deformation and poor initialization (99.535% success for the same data registered in 13.1 s), but given good initialization (e.g., ±5 mm, assuming a robust initial run) the same registration could be solved with 99.993% success in 6.3 s. The ability to register CT to fluoroscopy in a manner robust to patient deformation could be valuable in applications such as radiation therapy, interventional radiology, and an assistant to target localization (e.g., vertebral labeling) in image-guided spine surgery.

Breast lesion co-localisation between X-ray and MR images using finite element modelling

This paper presents a novel X-ray and MR image registration technique based on individual-specific biomechanical finite element (FE) models of the breasts. Information from 3D magnetic resonance (MR) images was registered to X-ray mammographic images using non-linear FE models subject to contact mechanics constraints to simulate the large compressive deformations between the two imaging modalities. A physics-based perspective ray-casting algorithm was used to generate 2D pseudo-X-ray projections of the FE-warped 3D MR images. Unknown input parameters to the FE models, such as the location and orientation of the compression plates, were optimised to provide the best match between the pseudo and clinical X-ray images. The methods were validated using images taken before and during compression of a breast-shaped phantom, for which 12 inclusions were tracked between imaging modalities. These methods were then applied to X-ray and MR images from six breast cancer patients. Error measures (such as centroid and surface distances) of segmented tumours in simulated and actual X-ray mammograms were used to assess the accuracy of the methods. Sensitivity analysis of the lesion co-localisation accuracy to rotation about the anterior-posterior axis was then performed. For 10 of the 12 X-ray mammograms, lesion localisation accuracies of 14 mm and less were achieved. This analysis on the rotation about the anterior-posterior axis indicated that, in cases where the lesion lies in the plane parallel to the mammographic compression plates, that cuts through the nipple, such rotations have relatively minor effects.This has important implications for clinical applicability of this multi-modality lesion registration technique, which will aid in the diagnosis and treatment of breast cancer.