Shaping the future through innovations: From medical imaging to precision medicine

Medical images constitute a source of information essential for disease diagnosis, treatment and follow-up. In addition, due to its patient-specific nature, imaging information represents a critical component required for advancing precision medicine into clinical practice. This manuscript describes recently developed technologies for better handling of image information: photorealistic visualization of medical images with Cinematic Rendering, artificial agents for in-depth image understanding, support for minimally invasive procedures, and patient-specific computational models with enhanced predictive power. Throughout the manuscript we will analyze the capabilities of such technologies and extrapolate on their potential impact to advance the quality of medical care, while reducing its cost.

Marginal Space Deep Learning: Efficient Architecture for Volumetric Image Parsing

Robust and fast solutions for anatomical object detection and segmentation support the entire clinical workflow from diagnosis, patient stratification, therapy planning, intervention and follow-up. Current state-of-the-art techniques for parsing volumetric medical image data are typically based on machine learning methods that exploit large annotated image databases. Two main challenges need to be addressed, these are the efficiency in scanning high-dimensional parametric spaces and the need for representative image features which require significant efforts of manual engineering. We propose a pipeline for object detection and segmentation in the context of volumetric image parsing, solving a two-step learning problem: anatomical pose estimation and boundary delineation. For this task we introduce Marginal Space Deep Learning (MSDL), a novel framework exploiting both the strengths of efficient object parametrization in hierarchical marginal spaces and the automated feature design of Deep Learning (DL) network architectures. In the 3D context, the application of deep learning systems is limited by the very high complexity of the parametrization. More specifically 9 parameters are necessary to describe a restricted affine transformation in 3D, resulting in a prohibitive amount of billions of scanning hypotheses. The mechanism of marginal space learning provides excellent run-time performance by learning classifiers in clustered, high-probability regions in spaces of gradually increasing dimensionality. To further increase computational efficiency and robustness, in our system we learn sparse adaptive data sampling patterns that automatically capture the structure of the input. Given the object localization, we propose a DL-based active shape model to estimate the non-rigid object boundary. Experimental results are presented on the aortic valve in ultrasound using an extensive dataset of 2891 volumes from 869 patients, showing significant improvements of up to 45.2% over the state-of-the-art. To our knowledge, this is the first successful demonstration of the DL potential to detection and segmentation in full 3D data with parametrized representations.

A self-taught artificial agent for multi-physics computational model personalization

Personalization is the process of fitting a model to patient data, a critical step towards application of multi-physics computational models in clinical practice. Designing robust personalization algorithms is often a tedious, time-consuming, model- and data-specific process. We propose to use artificial intelligence concepts to learn this task, inspired by how human experts manually perform it. The problem is reformulated in terms of reinforcement learning. In an off-line phase, Vito, our self-taught artificial agent, learns a representative decision process model through exploration of the computational model: it learns how the model behaves under change of parameters. The agent then automatically learns an optimal strategy for on-line personalization. The algorithm is model-independent; applying it to a new model requires only adjusting few hyper-parameters of the agent and defining the observations to match. The full knowledge of the model itself is not required. Vito was tested in a synthetic scenario, showing that it could learn how to optimize cost functions generically. Then Vito was applied to the inverse problem of cardiac electrophysiology and the personalization of a whole-body circulation model. The obtained results suggested that Vito could achieve equivalent, if not better goodness of fit than standard methods, while being more robust (up to 11% higher success rates) and with faster (up to seven times) convergence rate. Our artificial intelligence approach could thus make personalization algorithms generalizable and self-adaptable to any patient and any model.

A machine-learning approach for computation of fractional flow reserve from coronary computed tomography

Fractional flow reserve (FFR) is a functional index quantifying the severity of coronary artery lesions and is clinically obtained using an invasive, catheter-based measurement. Recently, physics-based models have shown great promise in being able to noninvasively estimate FFR from patient-specific anatomical information, e.g., obtained from computed tomography scans of the heart and the coronary arteries. However, these models have high computational demand, limiting their clinical adoption. In this paper, we present a machine-learning-based model for predicting FFR as an alternative to physics-based approaches. The model is trained on a large database of synthetically generated coronary anatomies, where the target values are computed using the physics-based model. The trained model predicts FFR at each point along the centerline of the coronary tree, and its performance was assessed by comparing the predictions against physics-based computations and against invasively measured FFR for 87 patients and 125 lesions in total. Correlation between machine-learning and physics-based predictions was excellent (0.9994, P < 0.001), and no systematic bias was found in Bland-Altman analysis: mean difference was -0.00081 ± 0.0039. Invasive FFR ≤ 0.80 was found in 38 lesions out of 125 and was predicted by the machine-learning algorithm with a sensitivity of 81.6%, a specificity of 83.9%, and an accuracy of 83.2%. The correlation was 0.729 (P < 0.001). Compared with the physics-based computation, average execution time was reduced by more than 80 times, leading to near real-time assessment of FFR. Average execution time went down from 196.3 ± 78.5 s for the CFD model to ∼2.4 ± 0.44 s for the machine-learning model on a workstation with 3.4-GHz Intel i7 8-core processor.

Noninvasive hemodynamic assessment, treatment outcome prediction and follow-up of aortic coarctation from MR imaging

PURPOSE

Coarctation of the aorta (CoA) is a congenital heart disease characterized by an abnormal narrowing of the proximal descending aorta. Severity of this pathology is quantified by the blood pressure drop (△P) across the stenotic coarctation lesion. In order to evaluate the physiological significance of the preoperative coarctation and to assess the postoperative results, the hemodynamic analysis is routinely performed by measuring the △P across the coarctation site via invasive cardiac catheterization. The focus of this work is to present an alternative, noninvasive measurement of blood pressure drop △P through the introduction of a fast, image-based workflow for personalized computational modeling of the CoA hemodynamics.

METHODS

The authors propose an end-to-end system comprised of shape and computational models, their personalization setup using MR imaging, and a fast, noninvasive method based on computational fluid dynamics (CFD) to estimate the pre- and postoperative hemodynamics for coarctation patients. A virtual treatment method is investigated to assess the predictive power of our approach.

RESULTS

Automatic thoracic aorta segmentation was applied on a population of 212 3D MR volumes, with mean symmetric point-to-mesh error of 3.00 ± 1.58 mm and average computation time of 8 s. Through quantitative evaluation of 6 CoA patients, good agreement between computed blood pressure drop and catheter measurements is shown: average differences are 2.38 ± 0.82 mm Hg (pre-), 1.10 ± 0.63 mm Hg (postoperative), and 4.99 ± 3.00 mm Hg (virtual stenting), respectively.

CONCLUSIONS

The complete workflow is realized in a fast, mostly-automated system that is integrable in the clinical setting. To the best of our knowledge, this is the first time that three different settings (preoperative--severity assessment, poststenting--follow-up, and virtual stenting--treatment outcome prediction) of CoA are investigated on multiple subjects. We believe that in future-given wider clinical validation-our noninvasive in-silico method could replace invasive pressure catheterization for CoA.

Comparison of Fractional Flow Reserve Based on Computational Fluid Dynamics Modeling Using Coronary Angiographic Vessel Morphology Versus Invasively Measured Fractional Flow Reserve

Invasive fractional flow reserve (FFRinvasive), although gold standard to identify hemodynamically relevant coronary stenoses, is time consuming and potentially associated with complications. We developed and evaluated a new approach to determine lesion-specific FFR on the basis of coronary anatomy as visualized by invasive coronary angiography (FFRangio): 100 coronary lesions (50% to 90% diameter stenosis) in 73 patients (48 men, 25 women; mean age 67 ± 9 years) were studied. On the basis of coronary angiograms acquired at rest from 2 views at angulations at least 30° apart, a PC-based computational fluid dynamics modeling software used personalized boundary conditions determined from 3-dimensional reconstructed angiography, heart rate, and blood pressure to derive FFRangio. The results were compared with FFRinvasive. Interobserver variability was determined in a subset of 25 narrowings. Twenty-nine of 100 coronary lesions were hemodynamically significant (FFRinvasive ≤ 0.80). FFRangio identified these with an accuracy of 90%, sensitivity of 79%, specificity of 94%, positive predictive value of 85%, and negative predictive value of 92%. The area under the receiver operating characteristic curve was 0.93. Correlation between FFRinvasive (mean: 0.84 ± 0.11) and FFRangio (mean: 0.85 ± 0.12) was r = 0.85. Interobserver variability of FFRangio was low, with a correlation of r = 0.88. In conclusion, estimation of coronary FFR with PC-based computational fluid dynamics modeling on the basis of lesion morphology as determined by invasive angiography is possible with high diagnostic accuracy compared to invasive measurements.

Model based non-invasive estimation of PV loop from echocardiography

We introduce a model-based approach for the non-invasive estimation of patient specific, left ventricular PV loops. A lumped parameter circulation model is used, composed of the pulmonary venous circulation, left atrium, left ventricle and the systemic circulation. A fully automated parameter estimation framework is introduced for model personalization, composed of two sequential steps: first, a series of parameters are computed directly, and, next, a fully automatic optimization-based calibration method is employed to iteratively estimate the values of the remaining parameters. The proposed methodology is first evaluated for three healthy volunteers: a perfect agreement is obtained between the computed quantities and the clinical measurements. Additionally, for an initial validation of the methodology, we computed the PV loop for a patient with mild aortic valve regurgitation and compared the results against the invasively determined quantities: there is a close agreement between the time-varying LV and aortic pressures, time-varying LV volumes, and PV loops.

Towards Personalized Cardiology: Multi-Scale Modeling of the Failing Heart

BACKGROUND

Despite modern pharmacotherapy and advanced implantable cardiac devices, overall prognosis and quality of life of HF patients remain poor. This is in part due to insufficient patient stratification and lack of individualized therapy planning, resulting in less effective treatments and a significant number of non-responders.

METHODS AND RESULTS

State-of-the-art clinical phenotyping was acquired, including magnetic resonance imaging (MRI) and biomarker assessment. An individualized, multi-scale model of heart function covering cardiac anatomy, electrophysiology, biomechanics and hemodynamics was estimated using a robust framework. The model was computed on n=46 HF patients, showing for the first time that advanced multi-scale models can be fitted consistently on large cohorts. Novel multi-scale parameters derived from the model of all cases were analyzed and compared against clinical parameters, cardiac imaging, lab tests and survival scores to evaluate the explicative power of the model and its potential for better patient stratification. Model validation was pursued by comparing clinical parameters that were not used in the fitting process against model parameters.

CONCLUSION

This paper illustrates how advanced multi-scale models can complement cardiovascular imaging and how they could be applied in patient care. Based on obtained results, it becomes conceivable that, after thorough validation, such heart failure models could be applied for patient management and therapy planning in the future, as we illustrate in one patient of our cohort who received CRT-D implantation.

A novel coupling algorithm for computing blood flow in viscoelastic arterial models

We propose a novel coupling algorithm, based on the operator-splitting scheme, which implements the viscoelastic wall law at the coupling nodes of the vessels. Two different viscoelastic models are used (V1 and V2), leading to five different computational setups: elastic wall law, model V1 applied at interior and coupling grid points, model V1 applied only at the interior grid points (V1-int), model V2 applied at interior and coupling grid points, model V2 applied only at the interior grid points (V2-int). These have been tested with two arterial configurations: (i) single artery, and (ii) complete arterial tree. Models V1-int and V2-int lead to incorrect conclusions and to errors which can be of the same order as, and are at least 1/5 of, the difference between the results with the elastic and the viscoelastic laws. Both test cases demonstrate the importance of modeling the viscous component of the pressure-area relationship at all grid points, including the coupling points between vessels or at the inlet/outlet of the model.

Efficient Lattice Boltzmann Solver for Patient-Specific Radiofrequency Ablation of Hepatic Tumors

Radiofrequency ablation (RFA) is an established treatment for liver cancer when resection is not possible. Yet, its optimal delivery is challenged by the presence of large blood vessels and the time-varying thermal conductivity of biological tissue. Incomplete treatment and an increased risk of recurrence are therefore common. A tool that would enable the accurate planning of RFA is hence necessary. This manuscript describes a new method to compute the extent of ablation required based on the Lattice Boltzmann Method (LBM) and patient-specific, pre-operative images. A detailed anatomical model of the liver is obtained from volumetric images. Then a computational model of heat diffusion, cellular necrosis, and blood flow through the vessels and liver is employed to compute the extent of ablated tissue given the probe location, ablation duration and biological parameters. The model was verified against an analytical solution, showing good fidelity. We also evaluated the predictive power of the proposed framework on ten patients who underwent RFA, for whom pre- and post-operative images were available. Comparisons between the computed ablation extent and ground truth, as observed in postoperative images, were promising (DICE index: 42%, sensitivity: 67%, positive predictive value: 38%). The importance of considering liver perfusion while simulating electrical-heating ablation was also highlighted. Implemented on graphics processing units (GPU), our method simulates 1 minute of ablation in 1.14 minutes, allowing near real-time computation.

Automatic detection and measurement of structures in fetal head ultrasound volumes using sequential estimation and Integrated Detection Network (IDN)

Routine ultrasound exam in the second and third trimesters of pregnancy involves manually measuring fetal head and brain structures in 2-D scans. The procedure requires a sonographer to find the standardized visualization planes with a probe and manually place measurement calipers on the structures of interest. The process is tedious, time consuming, and introduces user variability into the measurements. This paper proposes an automatic fetal head and brain (AFHB) system for automatically measuring anatomical structures from 3-D ultrasound volumes. The system searches the 3-D volume in a hierarchy of resolutions and by focusing on regions that are likely to be the measured anatomy. The output is a standardized visualization of the plane with correct orientation and centering as well as the biometric measurement of the anatomy. The system is based on a novel framework for detecting multiple structures in 3-D volumes. Since a joint model is difficult to obtain in most practical situations, the structures are detected in a sequence, one-by-one. The detection relies on Sequential Estimation techniques, frequently applied to visual tracking. The interdependence of structure poses and strong prior information embedded in our domain yields faster and more accurate results than detecting the objects individually. The posterior distribution of the structure pose is approximated at each step by sequential Monte Carlo. The samples are propagated within the sequence across multiple structures and hierarchical levels. The probabilistic model helps solve many challenges present in the ultrasound images of the fetus such as speckle noise, signal drop-out, shadows caused by bones, and appearance variations caused by the differences in the fetus gestational age. This is possible by discriminative learning on an extensive database of scans comprising more than two thousand volumes and more than thirteen thousand annotations. The average difference between ground truth and automatic measurements is below 2 mm with a running time of 6.9 s (GPU) or 14.7 s (CPU). The accuracy of the AFHB system is within inter-user variability and the running time is fast, which meets the requirements for clinical use.

Data-driven estimation of cardiac electrical diffusivity from 12-lead ECG signals

Diagnosis and treatment of dilated cardiomyopathy (DCM) is challenging due to a large variety of causes and disease stages. Computational models of cardiac electrophysiology (EP) can be used to improve the assessment and prognosis of DCM, plan therapies and predict their outcome, but require personalization. In this work, we present a data-driven approach to estimate the electrical diffusivity parameter of an EP model from standard 12-lead electrocardiograms (ECG). An efficient forward model based on a mono-domain, phenomenological Lattice-Boltzmann model of cardiac EP, and a boundary element-based mapping of potentials to the body surface is employed. The electrical diffusivity of myocardium, left ventricle and right ventricle endocardium is then estimated using polynomial regression which takes as input the QRS duration and electrical axis. After validating the forward model, we computed 9500 EP simulations on 19 different DCM patients in just under three seconds each to learn the regression model. Using this database, we quantify the intrinsic uncertainty of electrical diffusion for given ECG features and show in a leave-one-patient-out cross-validation that the regression method is able to predict myocardium diffusion within the uncertainty range. Finally, our approach is tested on the 19 cases using their clinical ECG. 84% of them could be personalized using our method, yielding mean prediction errors of 18.7ms for the QRS duration and 6.5° for the electrical axis, both values being within clinical acceptability. By providing an estimate of diffusion parameters from readily available clinical data, our data-driven approach could therefore constitute a first calibration step toward a more complete personalization of cardiac EP.

Multi-part modeling and segmentation of left atrium in C-arm CT for image-guided ablation of atrial fibrillation

As a minimally invasive surgery to treat atrial fibrillation (AF), catheter based ablation uses high radio-frequency energy to eliminate potential sources of abnormal electrical events, especially around the ostia of pulmonary veins (PV). Fusing a patient-specific left atrium (LA) model (including LA chamber, appendage, and PVs) with electro-anatomical maps or overlaying the model onto 2-D real-time fluoroscopic images provides valuable visual guidance during the intervention. In this work, we present a fully automatic LA segmentation system on nongated C-arm computed tomography (C-arm CT) data, where thin boundaries between the LA and surrounding tissues are often blurred due to the cardiac motion artifacts. To avoid segmentation leakage, the shape prior should be exploited to guide the segmentation. A single holistic shape model is often not accurate enough to represent the whole LA shape population under anatomical variations, e.g., the left common PVs vs. separate left PVs. Instead, a part based LA model is proposed, which includes the chamber, appendage, four major PVs, and right middle PVs. Each part is a much simpler anatomical structure compared to the holistic one and can be segmented using a model-based approach (except the right middle PVs). After segmenting the LA parts, the gaps and overlaps among the parts are resolved and segmentation of the ostia region is further refined. As a common anatomical variation, some patients may contain extra right middle PVs, which are segmented using a graph cuts algorithm under the constraints from the already extracted major right PVs. Our approach is computationally efficient, taking about 2.6 s to process a volume with 256 × 256 × 245 voxels. Experiments on 687 C-arm CT datasets demonstrate its robustness and state-of-the-art segmentation accuracy.

Image-based Co-Registration of Angiography and Intravascular Ultrasound Images

In image-guided cardiac interventions, X-ray imaging and intravascular ultrasound (IVUS) imaging are two often used modalities. Interventional X-ray images, including angiography and fluoroscopy, are used to assess the lumen of the coronary arteries and to monitor devices in real time. IVUS provides rich intravascular information, such as vessel wall composition, plaque, and stent expansions, but lacks spatial orientations. Since the two imaging modalities are complementary to each other, it is highly desirable to co-register the two modalities to provide a comprehensive picture of the coronaries for interventional cardiologists. In this paper, we present a solution for co-registering 2-D angiography and IVUS through image-based device tracking. The presented framework includes learning-based vessel detection and device detections, model-based tracking, and geodesic distance-based registration. The system first interactively detects the coronary branch under investigation in a reference angiography image. During the pullback of the IVUS transducers, the system acquires both ECG-triggered fluoroscopy and IVUS images, and automatically tracks the position of the medical devices in fluoroscopy. The localization of tracked IVUS transducers and guiding catheter tips is used to associate an IVUS imaging plane to a corresponding location on the vessel branch under investigation. The presented image-based solution can be conveniently integrated into existing cardiology workflow. The system is validated with a set of clinical cases, and achieves good accuracy and robustness.