Machine learning for predicting Plasmodium liver stage development in vitro using microscopy imaging

Malaria, a significant global health challenge, is caused by Plasmodium parasites. The Plasmodium liver stage plays a pivotal role in the establishment of the infection. This study focuses on the liver stage development of the model organism Plasmodium berghei, employing fluorescent microscopy imaging and convolutional neural networks (CNNs) for analysis. Convolutional neural networks have been recently proposed as a viable option for tasks such as malaria detection, prediction of host-pathogen interactions, or drug discovery. Our research aimed to predict the transition of Plasmodium-infected liver cells to the merozoite stage, a key development phase, 15 hours in advance. We collected and analyzed hourly imaging data over a span of at least 38 hours from 400 sequences, encompassing 502 parasites. Our method was compared to human annotations to validate its efficacy. Performance metrics, including the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity, were evaluated on an independent test dataset. The outcomes revealed an AUC of 0.873, a sensitivity of 84.6%, and a specificity of 83.3%, underscoring the potential of our CNN-based framework to predict liver stage development of P. berghei. These findings not only demonstrate the feasibility of our methodology but also could potentially contribute to the broader understanding of parasite biology.

A deep learning-based approach for efficient detection and classification of local Ca²⁺ release events in Full-Frame confocal imaging

The release of Ca2+ ions from intracellular stores plays a crucial role in many cellular processes, acting as a secondary messenger in various cell types, including cardiomyocytes, smooth muscle cells, hepatocytes, and many others. Detecting and classifying associated local Ca2+ release events is particularly important, as these events provide insight into the mechanisms, interplay, and interdependencies of local Ca2+release events underlying global intracellular Ca2+signaling. However, time-consuming and labor-intensive procedures often complicate analysis, especially with low signal-to-noise ratio imaging data. Here, we present an innovative deep learning-based approach for automatically detecting and classifying local Ca2+ release events. This approach is exemplified with rapid full-frame confocal imaging data recorded in isolated cardiomyocytes. To demonstrate the robustness and accuracy of our method, we first use conventional evaluation methods by comparing the intersection between manual annotations and the segmentation of Ca2+ release events provided by the deep learning method, as well as the annotated and recognized instances of individual events. In addition to these methods, we compare the performance of the proposed model with the annotation of six experts in the field. Our model can recognize more than 75 % of the annotated Ca2+ release events and correctly classify more than 75 %. A key result was that there were no significant differences between the annotations produced by human experts and the result of the proposed deep learning model. We conclude that the proposed approach is a robust and time-saving alternative to conventional full-frame confocal imaging analysis of local intracellular Ca2+ events.

RF-ULM: Ultrasound Localization Microscopy Learned from Radio-Frequency Wavefronts

In Ultrasound Localization Microscopy (ULM), achieving high-resolution images relies on the precise localization of contrast agent particles across a series of beamformed frames. However, our study uncovers an enormous potential: The process of delay-and-sum beamforming leads to an irreversible reduction of Radio-Frequency (RF) channel data, while its implications for localization remain largely unexplored. The rich contextual information embedded within RF wavefronts, including their hyperbolic shape and phase, offers great promise for guiding Deep Neural Networks (DNNs) in challenging localization scenarios. To fully exploit this data, we propose to directly localize scatterers in RF channel data. Our approach involves a custom super-resolution DNN using learned feature channel shuffling, non-maximum suppression, and a semi-global convolutional block for reliable and accurate wavefront localization. Additionally, we introduce a geometric point transformation that facilitates seamless mapping to the B-mode coordinate space. To understand the impact of beamforming on ULM, we validate the effectiveness of our method by conducting an extensive comparison with State-Of-The-Art (SOTA) techniques. We present the inaugural in vivo results from a wavefront-localizing DNN, highlighting its real-world practicality. Our findings show that RF-ULM bridges the domain shift between synthetic and real datasets, offering a considerable advantage in terms of precision and complexity. To enable the broader research community to benefit from our findings, our code and the associated SOTA methods are made available at https://github.com/hahnec/rf-ulm.

Cataract-1K Dataset for Deep-Learning-Assisted Analysis of Cataract Surgery Videos

In recent years, the landscape of computer-assisted interventions and post-operative surgical video analysis has been dramatically reshaped by deep-learning techniques, resulting in significant advancements in surgeons' skills, operation room management, and overall surgical outcomes. However, the progression of deep-learning-powered surgical technologies is profoundly reliant on large-scale datasets and annotations. In particular, surgical scene understanding and phase recognition stand as pivotal pillars within the realm of computer-assisted surgery and post-operative assessment of cataract surgery videos. In this context, we present the largest cataract surgery video dataset that addresses diverse requisites for constructing computerized surgical workflow analysis and detecting post-operative irregularities in cataract surgery. We validate the quality of annotations by benchmarking the performance of several state-of-the-art neural network architectures for phase recognition and surgical scene segmentation. Besides, we initiate the research on domain adaptation for instrument segmentation in cataract surgery by evaluating cross-domain instrument segmentation performance in cataract surgery videos. The dataset and annotations are publicly available in Synapse.

Deep Learning-Based Automated Detection of Retinal Breaks and Detachments on Fundus Photography

Purpose

The purpose of this study was to develop a deep learning algorithm, to detect retinal breaks and retinal detachments on ultra-widefield fundus (UWF) optos images using artificial intelligence (AI).

Methods

Optomap UWF images of the database were annotated to four groups by two retina specialists: (1) retinal breaks without detachment, (2) retinal breaks with retinal detachment, (3) retinal detachment without visible retinal breaks, and (4) a combination of groups 1 to 3. The fundus image data set was split into a training set and an independent test set following an 80% to 20% ratio. Image preprocessing methods were applied. An EfficientNet classification model was trained with the training set and evaluated with the test set.

Results

A total of 2489 UWF images were included into the dataset, resulting in a training set size of 2008 UWF images and a test set size of 481 images. The classification models achieved an area under the receiver operating characteristic curve (AUC) on the testing set of 0.975 regarding lesion detection, an AUC of 0.972 for retinal detachment and an AUC of 0.913 for retinal breaks.

Conclusions

A deep learning system to detect retinal breaks and retinal detachment using UWF images is feasible and has a good specificity. This is relevant for clinical routine as there can be a high rate of missed breaks in clinics. Future clinical studies will be necessary to evaluate the cost-effectiveness of applying such an algorithm as an automated auxiliary tool in a large practices or tertiary referral centers.

Translational Relevance

This study demonstrates the relevance of applying AI in diagnosing peripheral retinal breaks in clinical routine in UWF fundus images.

Prediction of chronic central serous chorioretinopathy through combined manual annotation and AI-assisted volume measurement of flat irregular pigment epithelium

INTRODUCTION

The aim of this study is to investigate the role of an artificial intelligence (AI)-developed OCT program to predict the clinical course of central serous chorioretinopathy (CSC ) based on baseline pigment epithelium detachment (PED) features.

METHODS

Single-center, observational study with a retrospective design. Treatment-naïve patients with acute CSC and chronic CSC were recruited and OCTs were analyzed by an AI-developed platform (Discovery OCT Fluid and Biomarker Detector, RetinAI AG, Switzerland), providing automatic detection and volumetric quantification of PEDs. Flat irregular PED presence was annotated manually and afterwards measured by the AI program automatically.

RESULTS

115 eyes of 101 patients with CSC were included, of which 70 were diagnosed with chronic CSC and 45 with acute CSC. It was found that patients with baseline presence of foveal flat PEDs and multiple flat foveal and extrafoveal PEDs had a higher chance of developing chronic form. AI-based volumetric analysis revealed no significant differences between the groups.

CONCLUSIONS

While more evidence is needed to confirm the effectiveness of AI-based PED quantitative analysis, this study highlights the significance of identifying flat irregular PEDs at the earliest stage possible in patients with CSC, to optimize patient management and long-term visual outcomes.

BASELINE SPECTRAL DOMAIN OPTICAL COHERENCE TOMOGRAPHIC RETINAL LAYER FEATURES IDENTIFIED BY ARTIFICIAL INTELLIGENCE PREDICT THE COURSE OF CENTRAL SEROUS CHORIORETINOPATHY

PURPOSE

To identify optical coherence tomography (OCT) features to predict the course of central serous chorioretinopathy (CSC) with an artificial intelligence-based program.

METHODS

Multicenter, observational study with a retrospective design. Treatment-naïve patients with acute CSC and chronic CSC were enrolled. Baseline OCTs were examined by an artificial intelligence-developed platform (Discovery OCT Fluid and Biomarker Detector, RetinAI AG, Switzerland). Through this platform, automated retinal layer thicknesses and volumes, including intaretinal and subretinal fluid, and pigment epithelium detachment were measured. Baseline OCT features were compared between acute CSC and chronic CSC patients.

RESULTS

One hundred and sixty eyes of 144 patients with CSC were enrolled, of which 100 had chronic CSC and 60 acute CSC. Retinal layer analysis of baseline OCT scans showed that the inner nuclear layer, the outer nuclear layer, and the photoreceptor-retinal pigmented epithelium complex were significantly thicker at baseline in eyes with acute CSC in comparison with those with chronic CSC ( P < 0.001). Similarly, choriocapillaris and choroidal stroma and retinal thickness (RT) were thicker in acute CSC than chronic CSC eyes ( P = 0.001). Volume analysis revealed average greater subretinal fluid volumes in the acute CSC group in comparison with chronic CSC ( P = 0.041).

CONCLUSION

Optical coherence tomography features may be helpful to predict the clinical course of CSC. The baseline presence of an increased thickness in the outer retinal layers, choriocapillaris and choroidal stroma, and subretinal fluid volume seems to be associated with acute course of the disease.

DeepPyramid+: medical image segmentation using Pyramid View Fusion and Deformable Pyramid Reception

PURPOSE

Semantic segmentation plays a pivotal role in many applications related to medical image and video analysis. However, designing a neural network architecture for medical image and surgical video segmentation is challenging due to the diverse features of relevant classes, including heterogeneity, deformability, transparency, blunt boundaries, and various distortions. We propose a network architecture, DeepPyramid+, which addresses diverse challenges encountered in medical image and surgical video segmentation.

METHODS

The proposed DeepPyramid+ incorporates two major modules, namely "Pyramid View Fusion" (PVF) and "Deformable Pyramid Reception" (DPR), to address the outlined challenges. PVF replicates a deduction process within the neural network, aligning with the human visual system, thereby enhancing the representation of relative information at each pixel position. Complementarily, DPR introduces shape- and scale-adaptive feature extraction techniques using dilated deformable convolutions, enhancing accuracy and robustness in handling heterogeneous classes and deformable shapes.

RESULTS

Extensive experiments conducted on diverse datasets, including endometriosis videos, MRI images, OCT scans, and cataract and laparoscopy videos, demonstrate the effectiveness of DeepPyramid+ in handling various challenges such as shape and scale variation, reflection, and blur degradation. DeepPyramid+ demonstrates significant improvements in segmentation performance, achieving up to a 3.65% increase in Dice coefficient for intra-domain segmentation and up to a 17% increase in Dice coefficient for cross-domain segmentation.

CONCLUSIONS

DeepPyramid+ consistently outperforms state-of-the-art networks across diverse modalities considering different backbone networks, showcasing its versatility. Accordingly, DeepPyramid+ emerges as a robust and effective solution, successfully overcoming the intricate challenges associated with relevant content segmentation in medical images and surgical videos. Its consistent performance and adaptability indicate its potential to enhance precision in computerized medical image and surgical video analysis applications.

Predicting OCT biological marker localization from weak annotations

Recent developments in deep learning have shown success in accurately predicting the location of biological markers in Optical Coherence Tomography (OCT) volumes of patients with Age-Related Macular Degeneration (AMD) and Diabetic Retinopathy (DR). We propose a method that automatically locates biological markers to the Early Treatment Diabetic Retinopathy Study (ETDRS) rings, only requiring B-scan-level presence annotations. We trained a neural network using 22,723 OCT B-Scans of 460 eyes (433 patients) with AMD and DR, annotated with slice-level labels for Intraretinal Fluid (IRF) and Subretinal Fluid (SRF). The neural network outputs were mapped into the corresponding ETDRS rings. We incorporated the class annotations and domain knowledge into a loss function to constrain the output with biologically plausible solutions. The method was tested on a set of OCT volumes with 322 eyes (189 patients) with Diabetic Macular Edema, with slice-level SRF and IRF presence annotations for the ETDRS rings. Our method accurately predicted the presence of IRF and SRF in each ETDRS ring, outperforming previous baselines even in the most challenging scenarios. Our model was also successfully applied to en-face marker segmentation and showed consistency within C-scans, despite not incorporating volume information in the training process. We achieved a correlation coefficient of 0.946 for the prediction of the IRF area.

Automated liver segmental volume ratio quantification on non-contrast T1-Vibe Dixon liver MRI using deep learning

PURPOSE

To evaluate the effectiveness of automated liver segmental volume quantification and calculation of the liver segmental volume ratio (LSVR) on a non-contrast T1-vibe Dixon liver MRI sequence using a deep learning segmentation pipeline.

METHOD

A dataset of 200 liver MRI with a non-contrast 3 mm T1-vibe Dixon sequence was manually labeledslice-by-sliceby an expert for Couinaud liver segments, while portal and hepatic veins were labeled separately. A convolutional neural networkwas trainedusing 170 liver MRI for training and 30 for evaluation. Liver segmental volumes without liver vessels were retrieved and LSVR was calculated as the liver segmental volumes I-III divided by the liver segmental volumes IV-VIII. LSVR was compared with the expert manual LSVR calculation and the LSVR calculated on CT scans in 30 patients with CT and MRI within 6 months.

RESULTS

Theconvolutional neural networkclassified the Couinaud segments I-VIII with an average Dice score of 0.770 ± 0.03, ranging between 0.726 ± 0.13 (segment IVb) and 0.810 ± 0.09 (segment V). The calculated mean LSVR with liver MRI unseen by the model was 0.32 ± 0.14, as compared with manually quantified LSVR of 0.33 ± 0.15, resulting in a mean absolute error (MAE) of 0.02. A comparable LSVR of 0.35 ± 0.14 with a MAE of 0.04 resulted with the LSRV retrieved from the CT scans. The automated LSVR showed significant correlation with the manual MRI LSVR (Spearman r = 0.97, p < 0.001) and CT LSVR (Spearman r = 0.95, p < 0.001).

CONCLUSIONS

A convolutional neural network allowed for accurate automated liver segmental volume quantification and calculation of LSVR based on a non-contrast T1-vibe Dixon sequence.

Müller matrix polarimetry for pancreatic tissue characterization

Polarimetry is an optical characterization technique capable of analyzing the polarization state of light reflected by materials and biological samples. In this study, we investigate the potential of Müller matrix polarimetry (MMP) to analyze fresh pancreatic tissue samples. Due to its highly heterogeneous appearance, pancreatic tissue type differentiation is a complex task. Furthermore, its challenging location in the body makes creating direct imaging difficult. However, accurate and reliable methods for diagnosing pancreatic diseases are critical for improving patient outcomes. To this end, we measured the Müller matrices of ex-vivo unfixed human pancreatic tissue and leverage the feature-learning capabilities of a machine-learning model to derive an optimized data representation that minimizes normal-abnormal classification error. We show experimentally that our approach accurately differentiates between normal and abnormal pancreatic tissue. This is, to our knowledge, the first study to use ex-vivo unfixed human pancreatic tissue combined with feature-learning from raw Müller matrix readings for this purpose.

Learning how to robustly estimate camera pose in endoscopic videos

PURPOSE

Surgical scene understanding plays a critical role in the technology stack of tomorrow's intervention-assisting systems in endoscopic surgeries. For this, tracking the endoscope pose is a key component, but remains challenging due to illumination conditions, deforming tissues and the breathing motion of organs.

METHOD

We propose a solution for stereo endoscopes that estimates depth and optical flow to minimize two geometric losses for camera pose estimation. Most importantly, we introduce two learned adaptive per-pixel weight mappings that balance contributions according to the input image content. To do so, we train a Deep Declarative Network to take advantage of the expressiveness of deep learning and the robustness of a novel geometric-based optimization approach. We validate our approach on the publicly available SCARED dataset and introduce a new in vivo dataset, StereoMIS, which includes a wider spectrum of typically observed surgical settings.

RESULTS

Our method outperforms state-of-the-art methods on average and more importantly, in difficult scenarios where tissue deformations and breathing motion are visible. We observed that our proposed weight mappings attenuate the contribution of pixels on ambiguous regions of the images, such as deforming tissues.

CONCLUSION

We demonstrate the effectiveness of our solution to robustly estimate the camera pose in challenging endoscopic surgical scenes. Our contributions can be used to improve related tasks like simultaneous localization and mapping (SLAM) or 3D reconstruction, therefore advancing surgical scene understanding in minimally invasive surgery.

Artificial intelligence for reducing the radiation burden of medical imaging for the diagnosis of coronavirus disease

Medical imaging has been intensively employed in screening, diagnosis and monitoring during the COVID-19 pandemic. With the improvement of RT-PCR and rapid inspection technologies, the diagnostic references have shifted. Current recommendations tend to limit the application of medical imaging in the acute setting. Nevertheless, efficient and complementary values of medical imaging have been recognized at the beginning of the pandemic when facing unknown infectious diseases and a lack of sufficient diagnostic tools. Optimizing medical imaging for pandemics may still have encouraging implications for future public health, especially for long-lasting post-COVID-19 syndrome theranostics. A critical concern for the application of medical imaging is the increased radiation burden, particularly when medical imaging is used for screening and rapid containment purposes. Emerging artificial intelligence (AI) technology provides the opportunity to reduce the radiation burden while maintaining diagnostic quality. This review summarizes the current AI research on dose reduction for medical imaging, and the retrospective identification of their potential in COVID-19 may still have positive implications for future public health.

Unsupervised out-of-distribution detection for safer robotically guided retinal microsurgery

PURPOSE

A fundamental problem in designing safe machine learning systems is identifying when samples presented to a deployed model differ from those observed at training time. Detecting so-called out-of-distribution (OoD) samples is crucial in safety-critical applications such as robotically guided retinal microsurgery, where distances between the instrument and the retina are derived from sequences of 1D images that are acquired by an instrument-integrated optical coherence tomography (iiOCT) probe.

METHODS

This work investigates the feasibility of using an OoD detector to identify when images from the iiOCT probe are inappropriate for subsequent machine learning-based distance estimation. We show how a simple OoD detector based on the Mahalanobis distance can successfully reject corrupted samples coming from real-world ex vivo porcine eyes.

RESULTS

Our results demonstrate that the proposed approach can successfully detect OoD samples and help maintain the performance of the downstream task within reasonable levels. MahaAD outperformed a supervised approach trained on the same kind of corruptions and achieved the best performance in detecting OoD cases from a collection of iiOCT samples with real-world corruptions.

CONCLUSION

The results indicate that detecting corrupted iiOCT data through OoD detection is feasible and does not need prior knowledge of possible corruptions. Consequently, MahaAD could aid in ensuring patient safety during robotically guided microsurgery by preventing deployed prediction models from estimating distances that put the patient at risk.

A reinforcement learning approach for VQA validation: An application to diabetic macular edema grading

Recent advances in machine learning models have greatly increased the performance of automated methods in medical image analysis. However, the internal functioning of such models is largely hidden, which hinders their integration in clinical practice. Explainability and trust are viewed as important aspects of modern methods, for the latter's widespread use in clinical communities. As such, validation of machine learning models represents an important aspect and yet, most methods are only validated in a limited way. In this work, we focus on providing a richer and more appropriate validation approach for highly powerful Visual Question Answering (VQA) algorithms. To better understand the performance of these methods, which answer arbitrary questions related to images, this work focuses on an automatic visual Turing test (VTT). That is, we propose an automatic adaptive questioning method, that aims to expose the reasoning behavior of a VQA algorithm. Specifically, we introduce a reinforcement learning (RL) agent that observes the history of previously asked questions, and uses it to select the next question to pose. We demonstrate our approach in the context of evaluating algorithms that automatically answer questions related to diabetic macular edema (DME) grading. The experiments show that such an agent has similar behavior to a clinician, whereby asking questions that are relevant to key clinical concepts.

A machine-learning model for reducing misdiagnosis in heparin-induced thrombocytopenia: A prospective, multicenter, observational study

BACKGROUND

Diagnosing heparin-induced thrombocytopenia (HIT) at the bedside remains challenging, exposing a significant number of patients at risk of delayed diagnosis or overtreatment. We hypothesized that machine-learning algorithms could be utilized to develop a more accurate and user-friendly diagnostic tool that integrates diverse clinical and laboratory information and accounts for complex interactions.

METHODS

We conducted a prospective cohort study including 1393 patients with suspected HIT between 2018 and 2021 from 10 study centers. Detailed clinical information and laboratory data were collected, and various immunoassays were conducted. The washed platelet heparin-induced platelet activation assay (HIPA) served as the reference standard.

FINDINGS

HIPA diagnosed HIT in 119 patients (prevalence 8.5%). The feature selection process in the training dataset (75% of patients) yielded the following predictor variables: (1) immunoassay test result, (2) platelet nadir, (3) unfractionated heparin use, (4) CRP, (5) timing of thrombocytopenia, and (6) other causes of thrombocytopenia. The best performing models were a support vector machine in case of the chemiluminescent immunoassay (CLIA) and the ELISA, as well as a gradient boosting machine in particle-gel immunoassay (PaGIA). In the validation dataset (25% of patients), the AUROC of all models was 0.99 (95% CI: 0.97, 1.00). Compared to the currently recommended diagnostic algorithm (4Ts score, immunoassay), the numbers of false-negative patients were reduced from 12 to 6 (-50.0%; ELISA), 9 to 3 (-66.7%, PaGIA) and 14 to 5 (-64.3%; CLIA). The numbers of false-positive individuals were reduced from 87 to 61 (-29.8%; ELISA), 200 to 63 (-68.5%; PaGIA) and increased from 50 to 63 (+29.0%) for the CLIA.

INTERPRETATION

Our user-friendly machine-learning algorithm for the diagnosis of HIT (https://toradi-hit.org) was substantially more accurate than the currently recommended diagnostic algorithm. It has the potential to reduce delayed diagnosis and overtreatment in clinical practice. Future studies shall validate this model in wider settings.

FUNDING

Swiss National Science Foundation (SNSF), and International Society on Thrombosis and Haemostasis (ISTH).

Evaluation of an Artificial Intelligence-Based Detector of Sub- and Intraretinal Fluid on a Large Set of Optical Coherence Tomography Volumes in Age-Related Macular Degeneration and Diabetic Macular Edema

INTRODUCTION

In this retrospective cohort study, we wanted to evaluate the performance and analyze the insights of an artificial intelligence (AI) algorithm in detecting retinal fluid in spectral-domain OCT volume scans from a large cohort of patients with neovascular age-related macular degeneration (AMD) and diabetic macular edema (DME).

METHODS

A total of 3,981 OCT volumes from 374 patients with AMD and 11,501 OCT volumes from 811 patients with DME were acquired with Heidelberg-Spectralis OCT device (Heidelberg Engineering Inc., Heidelberg, Germany) between 2013 and 2021. Each OCT volume was annotated for the presence or absence of intraretinal fluid (IRF) and subretinal fluid (SRF) by masked reading center graders (ground truth). The performance of an already published AI algorithm to detect IRF and SRF separately, and a combined fluid detector (IRF and/or SRF) of the same OCT volumes was evaluated. An analysis of the sources of disagreement between annotation and prediction and their relationship to central retinal thickness was performed. We computed the mean areas under the curves (AUC) and under the precision-recall curves (AP), accuracy, sensitivity, specificity, and precision.

RESULTS

The AUC for IRF was 0.92 and 0.98, for SRF 0.98 and 0.99, in the AMD and DME cohort, respectively. The AP for IRF was 0.89 and 1.00, for SRF 0.97 and 0.93, in the AMD and DME cohort, respectively. The accuracy, specificity, and sensitivity for IRF were 0.87, 0.88, 0.84, and 0.93, 0.95, 0.93, and for SRF 0.93, 0.93, 0.93, and 0.95, 0.95, 0.95 in the AMD and DME cohort, respectively. For detecting any fluid, the AUC was 0.95 and 0.98, and the accuracy, specificity, and sensitivity were 0.89, 0.93, and 0.90 and 0.95, 0.88, and 0.93, in the AMD and DME cohort, respectively. False positives were present when retinal shadow artifacts and strong retinal deformation were present. False negatives were due to small hyporeflective areas in combination with poor image quality. The combined detector correctly predicted more OCT volumes than the single detectors for IRF and SRF, 89.0% versus 81.6% in the AMD and 93.1% versus 88.6% in the DME cohort.

DISCUSSION/CONCLUSION

The AI-based fluid detector achieves high performance for retinal fluid detection in a very large dataset dedicated to AMD and DME. Combining single detectors provides better fluid detection accuracy than considering the single detectors separately. The observed independence of the single detectors ensures that the detectors learned features particular to IRF and SRF.

Using domain knowledge for robust and generalizable deep learning-based CT-free PET attenuation and scatter correction

Despite the potential of deep learning (DL)-based methods in substituting CT-based PET attenuation and scatter correction for CT-free PET imaging, a critical bottleneck is their limited capability in handling large heterogeneity of tracers and scanners of PET imaging. This study employs a simple way to integrate domain knowledge in DL for CT-free PET imaging. In contrast to conventional direct DL methods, we simplify the complex problem by a domain decomposition so that the learning of anatomy-dependent attenuation correction can be achieved robustly in a low-frequency domain while the original anatomy-independent high-frequency texture can be preserved during the processing. Even with the training from one tracer on one scanner, the effectiveness and robustness of our proposed approach are confirmed in tests of various external imaging tracers on different scanners. The robust, generalizable, and transparent DL development may enhance the potential of clinical translation.

Deep learning-based methods have been proposed to substitute CT-based PET attenuation and scatter correction to achieve CT-free PET imaging. Here, the authors present a simple way to integrate domain knowledge in deep learning for CT-free PET imaging.

Image-level trajectory inference of tau pathology using variational autoencoder for Flortaucipir PET

Purpose

Alzheimer’s disease (AD) studies revealed that abnormal deposition of tau spreads in a specific spatial pattern, namely Braak stage. However, Braak staging is based on post mortem brains, each of which represents the cross section of the tau trajectory in disease progression, and numerous studies were reported that do not conform to that model. This study thus aimed to identify the tau trajectory and quantify the tau progression in a data-driven approach with the continuous latent space learned by variational autoencoder (VAE).

Methods

A total of 1080 [¹⁸F]Flortaucipir brain positron emission tomography (PET) images were collected from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. VAE was built to compress the hidden features from tau images in latent space. Hierarchical agglomerative clustering and minimum spanning tree (MST) were applied to organize the features and calibrate them to the tau progression, thus deriving pseudo-time. The image-level tau trajectory was inferred by continuously sampling across the calibrated latent features. We assessed the pseudo-time with regard to tau standardized uptake value ratio (SUVr) in AD-vulnerable regions, amyloid deposit, glucose metabolism, cognitive scores, and clinical diagnosis.

Results

We identified four clusters that plausibly capture certain stages of AD and organized the clusters in the latent space. The inferred tau trajectory agreed with the Braak staging. According to the derived pseudo-time, tau first deposits in the parahippocampal and amygdala, and then spreads to the fusiform, inferior temporal lobe, and posterior cingulate. Prior to the regional tau deposition, amyloid accumulates first.

Conclusion

The spatiotemporal trajectory of tau progression inferred in this study was consistent with Braak staging. The profile of other biomarkers in disease progression agreed well with previous findings. We addressed that this approach additionally has the potential to quantify tau progression as a continuous variable by taking a whole-brain tau image into account.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00259-021-05662-z.

Surgical data science - from concepts toward clinical translation

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Multiparametric Cardiovascular Magnetic Resonance Approach in Diagnosing, Monitoring, and Prognostication of Myocarditis

Myocarditis represents the entity of an inflamed myocardium and is a diagnostic challenge caused by its heterogeneous presentation. Contemporary noninvasive evaluation of patients with clinically suspected myocarditis using cardiac magnetic resonance (CMR) includes dimensions and function of the heart chambers, conventional T2-weighted imaging, late gadolinium enhancement, novel T1 and T2 mapping, and extracellular volume fraction calculation. CMR feature-tracking, texture analysis, and artificial intelligence emerge as potential modern techniques to further improve diagnosis and prognostication in this clinical setting. This review will describe the evidence surrounding different CMR methods and image postprocessing methods and highlight their values for clinical decision making, monitoring, and risk stratification across stages of this condition.

A cross-scanner and cross-tracer deep learning method for the recovery of standard-dose imaging quality from low-dose PET

Purpose

A critical bottleneck for the credibility of artificial intelligence (AI) is replicating the results in the diversity of clinical practice. We aimed to develop an AI that can be independently applied to recover high-quality imaging from low-dose scans on different scanners and tracers.

Methods

Brain [¹⁸F]FDG PET imaging of 237 patients scanned with one scanner was used for the development of AI technology. The developed algorithm was then tested on [¹⁸F]FDG PET images of 45 patients scanned with three different scanners, [¹⁸F]FET PET images of 18 patients scanned with two different scanners, as well as [¹⁸F]Florbetapir images of 10 patients. A conditional generative adversarial network (GAN) was customized for cross-scanner and cross-tracer optimization. Three nuclear medicine physicians independently assessed the utility of the results in a clinical setting.

Results

The improvement achieved by AI recovery significantly correlated with the baseline image quality indicated by structural similarity index measurement (SSIM) (r = −0.71, p < 0.05) and normalized dose acquisition (r = −0.60, p < 0.05). Our cross-scanner and cross-tracer AI methodology showed utility based on both physical and clinical image assessment (p < 0.05).

Conclusion

The deep learning development for extensible application on unknown scanners and tracers may improve the trustworthiness and clinical acceptability of AI-based dose reduction.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00259-021-05644-1.

Virtual Reality-Based and Conventional Visual Field Examination Comparison in Healthy and Glaucoma Patients

Purpose

Clinically evaluate the noninferiority of a custom virtual reality (VR) perimetry system when compared to a clinically and routinely used perimeter on both healthy subjects and glaucoma patients.

Methods

We use a custom-designed VR perimetry system tailored for visual field testing. The system uses Oculus Quest VR headset (Facebook Technologies, LLC, Bern, Switzerland), that includes a clicker for participant response feedback. A prospective, single center, study was conducted at the Department of Ophthalmology of the Bern University Hospital (Bern, Switzerland) for 12 months. Of the 114 participants recruited 70 subjects (36 healthy and 34 glaucoma patients with early to moderate visual field loss) were included in the study. Participants underwent perimetry tests on an Octopus 900 (Haag-Streit, Köniz, Switzerland) as well as on the custom VR perimeter. In both cases, standard dynamic strategy (DS) was used in conjunction with the G testing pattern. Collected visual fields (VFs) from both devices were then analyzed and compared.

Results

High mean defect (MD) correlations between the two systems (Spearman, ρ ≥ 0.75) were obtained. The VR system was found to slightly underestimate VF defects in glaucoma subjects (1.4 dB). No significant bias was found with respect to eccentricity or subject age. On average, a similar number of stimuli presentations per VF was necessary when measuring glaucoma patients and healthy subjects.

Conclusions

This study demonstrates that a clinically used perimeter and the proposed VR perimetry system have comparable performances with respect to a number of perimetry parameters in healthy and glaucoma patients with early to moderate visual field loss.

Translational Relevance

This suggests that VR perimeters have the potential to assess VFs with high enough confidence, whereby alleviating challenges in current perimetry practices by providing a portable and more accessible visual field test.

Bayesian brain in tinnitus: Computational modeling of three perceptual phenomena using a modified Hierarchical Gaussian Filter

Recently, Bayesian brain-based models emerged as a possible composite of existing theories, providing an universal explanation of tinnitus phenomena. Yet, the involvement of multiple synergistic mechanisms complicates the identification of behavioral and physiological evidence. To overcome this, an empirically tested computational model could support the evaluation of theoretical hypotheses by intrinsically encompassing different mechanisms. The aim of this work was to develop a generative computational tinnitus perception model based on the Bayesian brain concept. The behavioral responses of 46 tinnitus subjects who underwent ten consecutive residual inhibition assessments were used for model fitting. Our model was able to replicate the behavioral responses during residual inhibition in our cohort (median linear correlation coefficient of 0.79). Using the same model, we simulated two additional tinnitus phenomena: residual excitation and occurrence of tinnitus in non-tinnitus subjects after sensory deprivation. In the simulations, the trajectories of the model were consistent with previously obtained behavioral and physiological observations. Our work introduces generative computational modeling to the research field of tinnitus. It has the potential to quantitatively link experimental observations to theoretical hypotheses and to support the search for neural signatures of tinnitus by finding correlates between the latent variables of the model and measured physiological data.

A positive/unlabeled approach for the segmentation of medical sequences using point-wise supervision

The ability to quickly annotate medical imaging data plays a critical role in training deep learning frameworks for segmentation. Doing so for image volumes or video sequences is even more pressing as annotating these is particularly burdensome. To alleviate this problem, this work proposes a new method to efficiently segment medical imaging volumes or videos using point-wise annotations only. This allows annotations to be collected extremely quickly and remains applicable to numerous segmentation tasks. Our approach trains a deep learning model using an appropriate Positive/Unlabeled objective function using sparse point-wise annotations. While most methods of this kind assume that the proportion of positive samples in the data is known a-priori, we introduce a novel self-supervised method to estimate this prior efficiently by combining a Bayesian estimation framework and new stopping criteria. Our method iteratively estimates appropriate class priors and yields high segmentation quality for a variety of object types and imaging modalities. In addition, by leveraging a spatio-temporal tracking framework, we regularize our predictions by leveraging the complete data volume. We show experimentally that our approach outperforms state-of-the-art methods tailored to the same problem.

Mask then classify: multi-instance segmentation for surgical instruments

PURPOSE

The detection and segmentation of surgical instruments has been a vital step for many applications in minimally invasive surgical robotics. Previously, the problem was tackled from a semantic segmentation perspective, yet these methods fail to provide good segmentation maps of instrument types and do not contain any information on the instance affiliation of each pixel. We propose to overcome this limitation by using a novel instance segmentation method which first masks instruments and then classifies them into their respective type.

METHODS

We introduce a novel method for instance segmentation where a pixel-wise mask of each instance is found prior to classification. An encoder-decoder network is used to extract instrument instances, which are then separately classified using the features of the previous stages. Furthermore, we present a method to incorporate instrument priors from surgical robots.

RESULTS

Experiments are performed on the robotic instrument segmentation dataset of the 2017 endoscopic vision challenge. We perform a fourfold cross-validation and show an improvement of over 18% to the previous state-of-the-art. Furthermore, we perform an ablation study which highlights the importance of certain design choices and observe an increase of 10% over semantic segmentation methods.

CONCLUSIONS

We have presented a novel instance segmentation method for surgical instruments which outperforms previous semantic segmentation-based methods. Our method further provides a more informative output of instance level information, while retaining a precise segmentation mask. Finally, we have shown that robotic instrument priors can be used to further increase the performance.

CODEX, a neural network approach to explore signaling dynamics landscapes

Current studies of cell signaling dynamics that use live cell fluorescent biosensors routinely yield thousands of single-cell, heterogeneous, multi-dimensional trajectories. Typically, the extraction of relevant information from time series data relies on predefined, human-interpretable features. Without a priori knowledge of the system, the predefined features may fail to cover the entire spectrum of dynamics. Here we present CODEX, a data-driven approach based on convolutional neural networks (CNNs) that identifies patterns in time series. It does not require a priori information about the biological system and the insights into the data are built through explanations of the CNNs' predictions. CODEX provides several views of the data: visualization of all the single-cell trajectories in a low-dimensional space, identification of prototypic trajectories, and extraction of distinctive motifs. We demonstrate how CODEX can provide new insights into ERK and Akt signaling in response to various growth factors, and we recapitulate findings in p53 and TGFβ-SMAD2 signaling.

Volumetric Quantitative Ablation Margins for Assessment of Ablation Completeness in Thermal Ablation of Liver Tumors

BACKGROUND

In thermal ablation of liver tumors, complete coverage of the tumor volume by the ablation volume with a sufficient ablation margin is the most important factor for treatment success. Evaluation of ablation completeness is commonly performed by visual inspection in 2D and is prone to inter-reader variability. This work aimed to introduce a standardized approach for evaluation of ablation completeness after CT-guided thermal ablation of liver tumors, using volumetric quantitative ablation margins (QAM).

METHODS

A QAM computation metric based on volumetric segmentations of tumor and ablation areas and signed Euclidean surface distance maps was developed, including a novel algorithm to address QAM computation in subcapsular tumors. The code for QAM computation was verified in artificial examples of tumor and ablation spheres simulating varying scenarios of ablation margins. The applicability of the QAM metric was investigated in representative cases extracted from a prospective database of colorectal liver metastases (CRLM) treated with stereotactic microwave ablation (SMWA).

RESULTS

Applicability of the proposed QAM metric was confirmed in artificial and clinical example cases. Numerical and visual options of data presentation displaying substrata of QAM distributions were proposed. For subcapsular tumors, the underestimation of tumor coverage by the ablation volume when applying an unadjusted QAM method was confirmed, supporting the benefits of using the proposed algorithm for QAM computation in these cases. The computational code for developed QAM was made publicly available, encouraging the use of a standard and objective metric in reporting ablation completeness and margins.

CONCLUSION

The proposed volumetric approach for QAM computation including a novel algorithm to address subcapsular liver tumors enables precision and reproducibility in the assessment of ablation margins. The quantitative feedback on ablation completeness opens possibilities for intra-operative decision making and for refined analyses on predictability and consistency of local tumor control after thermal ablation of liver tumors.

Screening test accuracy to improve detection of precancerous lesions of the cervix in women living with HIV: a study protocol

INTRODUCTION

The simplest and cheapest method for cervical cancer screening is visual inspection after application of acetic acid (VIA). However, this method has limitations for correctly identifying precancerous cervical lesions (sensitivity) and women free from these lesions (specificity). We will assess alternative screening methods that could improve sensitivity and specificity in women living with humanimmunodeficiency virus (WLHIV) in Southern Africa.

METHODS AND ANALYSIS

We will conduct a paired, prospective, screening test accuracy study among consecutive, eligible women aged 18-65 years receiving treatment for HIV/AIDS at Kanyama Hospital, Lusaka, Zambia. We will assess a portable magnification device (Gynocular, Gynius Plus AB, Sweden) based on the Swede score assessment of the cervix, test for high-risk subtypes of human papillomavirus (HR-HPV, GeneXpert, Cepheid, USA) and VIA. All study participants will receive all three tests and the reference standard at baseline and at six-month follow-up. The reference standard is histological assessment of two to four biopsies of the transformation zone. The primary histological endpoint is cervical intraepithelial neoplasia grade two and above (CIN2+). Women who are VIA-positive or have histologically confirmed CIN2+ lesions will be treated as per national guidelines. We plan to enrol 450 women. Primary outcome measures for test accuracy include sensitivity and specificity of each stand-alone test. In the secondary analyses, we will evaluate the combination of tests. Pre-planned additional studies include use of cervigrams to test an automated visual assessment tool using image pattern recognition, cost-analysis and associations with trichomoniasis.

ETHICS AND DISSEMINATION

Ethical approval was obtained from the University of Zambia Biomedical Research Ethics Committee, Zambian National Health Regulatory Authority, Zambia Medicines Regulatory Authority, Swissethics and the International Agency for Research on Cancer Ethics Committee. Results of the study will be submitted for publication in a peer-reviewed journal.

TRIAL REGISTRATION NUMBER

NCT03931083; Pre-results.

Comparative Study Between the SORS and Dynamic Strategy Visual Field Testing Methods on Glaucomatous and Healthy Subjects

Purpose

To clinically validate the noninferiority of the sequentially optimized reconstruction strategy (SORS) when compared to the dynamic strategy (DS).

Methods

SORS is a novel perimetry testing strategy that evaluates a subset of test locations of a visual field (VF) test pattern and estimates the untested locations by linear approximation. When testing fewer locations, SORS has been shown in computer simulations to bring improvements in speed over conventional perimetry tests, while maintaining acquisition at high-quality acquisition. To validate SORS, a prospective clinical study was conducted at the Department of Ophthalmology of Bern University Hospital, over 12 months. Eighty-three subjects (32 healthy and 51 glaucoma patients with early to moderate visual field loss) of 114 participants were included in the study. The subjects underwent perimetry tests on an Octopus 900 (Haag-Streit, Köniz, Switzerland) using the G pattern with both DS and SORS. The acquired sensitivity thresholds (ST) by both tests were analyzed and compared.

Results

DS-acquired VFs were used as a reference. High correlations between individual STs (r ≥ 0.74), as well as between mean defect values (r ≥ 0.88) given by DS and SORS were obtained. The mean absolute error of SORS was under 3 dB with a 70% reduction in acquisition time. SORS overestimated healthy VFs while slightly underestimating glaucomatous VFs. Qualitatively, SORS acquisition yielded VF with detectable defect patterns, albeit some isolated and small defects were occasionally missed.

Conclusions

This clinical study showed that for healthy and glaucomatous patients, SORS-acquired VFs sufficiently correlated with the DS-acquired VFs with up to 70% reduction in acquisition time.

Translational Relevance

This clinical study suggests that the novel perimetry strategy SORS could be used in routine clinical practice with comparable utility to the current standard DS, whereby providing a shorter and more comfortable perimetry experience.

A Question-Centric Model for Visual Question Answering in Medical Imaging

Deep learning methods have proven extremely effective at performing a variety of medical image analysis tasks. With their potential use in clinical routine, their lack of transparency has however been one of their few weak points, raising concerns regarding their behavior and failure modes. While most research to infer model behavior has focused on indirect strategies that estimate prediction uncertainties and visualize model support in the input image space, the ability to explicitly query a prediction model regarding its image content offers a more direct way to determine the behavior of trained models. To this end, we present a novel Visual Question Answering approach that allows an image to be queried by means of a written question. Experiments on a variety of medical and natural image datasets show that by fusing image and question features in a novel way, the proposed approach achieves an equal or higher accuracy compared to current methods.

Automatically Enhanced OCT Scans of the Retina: A proof of concept study

In this work we evaluated a postprocessing, customized automatic retinal OCT B-scan enhancement software for noise reduction, contrast enhancement and improved depth quality applicable to Heidelberg Engineering Spectralis OCT devices. A trained deep neural network was used to process images from an OCT dataset with ground truth biomarker gradings. Performance was assessed by the evaluation of two expert graders who evaluated image quality for B-scan with a clear preference for enhanced over original images. Objective measures such as SNR and noise estimation showed a significant improvement in quality. Presence grading of seven biomarkers IRF, SRF, ERM, Drusen, RPD, GA and iRORA resulted in similar intergrader agreement. Intergrader agreement was also compared with improvement in IRF and RPD, and disagreement in high variance biomarkers such as GA and iRORA.

Comparison of Choroidal Thickness Measurements Using Spectral Domain Optical Coherence Tomography in Six Different Settings and With Customized Automated Segmentation Software

Purpose

We investigate which spectral domain-optical coherence tomography (SD-OCT) setting is superior when measuring subfoveal choroidal thickness (CT) and compared results to an automated segmentation software.

Methods

Thirty patients underwent enhanced depth imaging (EDI)-OCT. B-scans were extracted in six different settings (W+N = white background/normal contrast 9; W+H = white background/maximum contrast 16; B+N = black background/normal contrast 12; B+H = black background/maximum contrast 16; C+N = Color-encoded image on black background at predefined contrast of 9, and C+H = Color-encoded image on black background at high/maximal contrast of 16), resulting in 180 images. Subfoveal CT was manually measured by nine graders and by automated segmentation software. Intraclass correlation (ICC) was assessed.

Results

ICC was higher in normal than in high contrast images, and better for achromatic black than for white background images. Achromatic images were better than color images. Highest ICC was achieved in B+N (ICC = 0.64), followed by B+H (ICC = 0.54), W+N, and W+H (ICC = 0.5 each). Weakest ICC was obtained with Spectral-color (ICC = 0.47). Mean manual CT versus mean computer estimated CT showed a correlation of r = 0.6 (P = 0.001).

Conclusion

Black background with white image at normal contrast (B+N) seems the best setting to manually assess subfoveal CT. Automated assessment of CT seems to be a reliable tool for CT assessment.

Translational Relevance

To define optimized OCT analysis settings to improve the evaluation of in vivo imaging.

Patient-attentive sequential strategy for perimetry-based visual field acquisition

Perimetry is a non-invasive clinical psychometric examination used for diagnosing ophthalmic and neurological conditions. At its core, perimetry relies on a subject pressing a button whenever they see a visual stimulus within their field of view. This sequential process then yields a 2D visual field image that is critical for clinical use. Perimetry is painfully slow however, with examinations lasting 7-8 minutes per eye. Maintaining high levels of concentration during that time is exhausting for the patient and negatively affects the acquired visual field. We introduce PASS, a novel perimetry testing strategy, based on reinforcement learning, that requires fewer locations in order to effectively estimate 2D visual fields. PASS uses a selection policy that determines what locations should be tested in order to reconstruct the complete visual field as accurately as possible, and then separately reconstructs the visual field from sparse observations. Furthermore, PASS is patient-specific and non-greedy. It adaptively selects what locations to query based on the patient's answers to previous queries, and the locations are jointly selected to maximize the quality of the final reconstruction. In our experiments, we show that PASS outperforms state-of-the-art methods, leading to more accurate reconstructions while reducing between 30% and 70% the duration of the patient examination.

Association Between Residual Inhibition and Neural Activity in Patients with Tinnitus: Protocol for a Controlled Within- and Between-Subject Comparison Study

Background

Electroencephalography (EEG) studies indicate possible associations between tinnitus and changes in the neural activity. However, inconsistent results require further investigation to better understand such heterogeneity and inform the interpretation of previous findings.

Objective

This study aims to investigate the feasibility of EEG measurements as an objective indicator for the identification of tinnitus-associated neural activities.

Methods

To reduce heterogeneity, participants served as their own control using residual inhibition (RI) to modulate the tinnitus perception in a within-subject EEG study design with a tinnitus group. In addition, comparison with a nontinnitus control group allowed for a between-subjects comparison. We will apply RI stimulation to generate tinnitus and nontinnitus conditions in the same subject. Furthermore, high-frequency audiometry (up to 13 kHz) and tinnitometry will be performed.

Results

This work was funded by the Infrastructure Grant of the University of Bern, Bern, Switzerland and Bernafon AG, Bern, Switzerland. Enrollment for the study described in this protocol commenced in February 2018. Data analysis is currently under way and the first results are expected to be submitted for publication in 2019.

Conclusions

This study design helps in comparing the neural activity between conditions in the same individual, thereby addressing a notable limitation of previous EEG tinnitus studies. In addition, the high-frequency assessment will help to analyze and classify tinnitus symptoms beyond the conventional clinical standard.

International Registered Report Identifier (IRRID)

RR1-10.2196/12270

A deep learning approach to automatic detection of early glaucoma from visual fields

PURPOSE

To investigate the suitability of multi-scale spatial information in 30o visual fields (VF), computed from a Convolutional Neural Network (CNN) classifier, for early-glaucoma vs. control discrimination.

METHOD

Two data sets of VFs acquired with the OCTOPUS 101 G1 program and the Humphrey Field Analyzer 24-2 pattern were subdivided into control and early-glaucomatous groups, and converted into a new image using a novel voronoi representation to train a custom-designed CNN so to discriminate between control and early-glaucomatous eyes. Saliency maps that highlight what regions of the VF are contributing maximally to the classification decision were computed to provide classification justification. Model fitting was cross-validated and average precision (AP) score performances were computed for our method, Mean Defect (MD), square-root of Loss Variance (sLV), their combination (MD+sLV), and a Neural Network (NN) that does not use convolutional features.

RESULTS

CNN achieved the best AP score (0.874±0.095) across all test folds for one data set compared to others (MD = 0.869±0.064, sLV = 0.775±0.137, MD+sLV = 0.839±0.085, NN = 0.843±0.089) and the third best AP score (0.986 ±0.019) on the other one with slight difference from the other methods (MD = 0.986±0.023, sLV = 0.992±0.016, MD+sLV = 0.987±0.017, NN = 0.985±0.017). In general, CNN consistently led to high AP across different data sets. Qualitatively, computed saliency maps appeared to provide clinically relevant information on the CNN decision for individual VFs.

CONCLUSION

The proposed CNN offers high classification performance for the discrimination of control and early-glaucoma VFs when compared with standard clinical decision measures. The CNN classification, aided by saliency visualization, may support clinicians in the automatic discrimination of early-glaucomatous and normal VFs.

Personalized Anatomic Eye Model From T1-Weighted Volume Interpolated Gradient Echo Magnetic Resonance Imaging of Patients With Uveal Melanoma

PURPOSE

We present a 3-dimensional patient-specific eye model from magnetic resonance imaging (MRI) for proton therapy treatment planning of uveal melanoma (UM). During MRI acquisition of UM patients, the point fixation can be difficult and, together with physiological blinking, can introduce motion artifacts in the images, thus challenging the model creation. Furthermore, the unclear boundary of the small objects (eg, lens, optic nerve) near the muscle or of the tumors with hemorrhage and tantalum clips can limit model accuracy.

METHODS AND MATERIALS

A dataset of 37 subjects, including 30 healthy eyes of volunteers and 7 eyes of UM patients, was investigated. In our previous work, active shape model was successfully applied to retinoblastoma eye segmentation in T1-weighted 3T MRI. Here, we evaluate this method in a more challenging setting, based on 1.5T MRI acquisition and different datasets of awake adult eyes with UM. The lens and cornea together with the sclera, vitreous humor, and optic nerve were automatically segmented and validated against manual delineations of a senior ocular radiation oncologist, in terms of the Dice similarity coefficient and Hausdorff distance.

RESULTS

Leave-one-out cross validation (mixing both volunteers and UM patients) yielded median Dice similarity coefficient values (respective of Hausdorff distance) of 94.5% (1.64 mm) for the sclera, 92.2% (1.73 mm) for the vitreous humor, 88.3% (1.09 mm) for the lens, and 81.9% (1.86 mm) for the optic nerve. The average computation time for an eye was 10 seconds.

CONCLUSIONS

To our knowledge, our work is the first attempt to automatically segment adult eyes, including patients with UM. Our results show that automated active shape model segmentation can succeed in the presence of motion, tumors, and tantalum clips. These results are promising for inclusion in clinical practice.

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

Instrument detection, pose estimation, and tracking in surgical videos are an important vision component for computer-assisted interventions. While significant advances have been made in recent years, articulation detection is still a major challenge. In this paper, we propose a deep neural network for articulated multi-instrument 2-D pose estimation, which is trained on detailed annotations of endoscopic and microscopic data sets. Our model is formed by a fully convolutional detection-regression network. Joints and associations between joint pairs in our instrument model are located by the detection subnetwork and are subsequently refined through a regression subnetwork. Based on the output from the model, the poses of the instruments are inferred using maximum bipartite graph matching. Our estimation framework is powered by deep learning techniques without any direct kinematic information from a robot. Our framework is tested on single-instrument RMIT data, and also on multi-instrument EndoVis and in vivo data with promising results. In addition, the data set annotations are publicly released along with our code and model.

Motion-invariant SRT treatment detection from direct M-scan OCT imaging

PURPOSE

Selective retina therapy (SRT) is a laser treatment targeting specific posterior retinal layers. It is focused on inducing damage to the retinal pigment epithelium (RPE), while sparing other retinal tissue compared to traditional photocoagulation. However, the targeted RPE layer is invisible with most imaging modalities and induced SRT lesions cannot be monitored. In this work, imaging scans acquired from an experimental setup that couples the SRT laser beam with an optical coherence tomography (OCT) beam are analyzed in order to evaluate the treatment as they occur.

METHODS

We isolated a small part of the time-resolved scan corresponding to the end of the treatment, for which we have microscopic evidence of the SRT outcome. We then use a convolutional neural network to correspond each scan to the treatment result. We explore which aspects of the scan convey more valuable information for a robust therapy evaluation. By only using this adequately small part, we can achieve an online estimation, while being resilient to eye movement.

RESULTS

The available dataset consists of time- resolved OCT scans of 98 ex vivo porcine eyes, treated with different energy levels. The proposed method yields high performance in the task of predicting whether the applied energy was adequate for SRT treatment, by focusing on the immediate OCT signal acquired during treatment time.

CONCLUSIONS

We propose a strategy toward online noninvasive SRT treatment assessment, able to provide a satisfying evaluation of a treatment status, that therefore could be used for the planning of the treatment continuation.

Reconstructing Evolving Tree Structures in Time Lapse Sequences by Enforcing Time-Consistency

We propose a novel approach to reconstructing curvilinear tree structures evolving over time, such as road networks in 2D aerial images or neural structures in 3D microscopy stacks acquired in vivo. To enforce temporal consistency, we simultaneously process all images in a sequence, as opposed to reconstructing structures of interest in each image independently. We formulate the problem as a Quadratic Mixed Integer Program and demonstrate the additional robustness that comes from using all available visual clues at once, instead of working frame by frame. Furthermore, when the linear structures undergo local changes over time, our approach automatically detects them.

Sequentially optimized reconstruction strategy: A meta-strategy for perimetry testing

Perimetry testing is an automated method to measure visual function and is heavily used for diagnosing ophthalmic and neurological conditions. Its working principle is to sequentially query a subject about perceived light using different brightness levels at different visual field locations. At a given location, this query-patient-feedback process is expected to converge at a perceived sensitivity, such that a shown stimulus intensity is observed and reported 50% of the time. Given this inherently time-intensive and noisy process, fast testing strategies are necessary in order to measure existing regions more effectively and reliably. In this work, we present a novel meta-strategy which relies on the correlative nature of visual field locations in order to strongly reduce the necessary number of locations that need to be examined. To do this, we sequentially determine locations that most effectively reduce visual field estimation errors in an initial training phase. We then exploit these locations at examination time and show that our approach can easily be combined with existing perceived sensitivity estimation schemes to speed up the examinations. Compared to state-of-the-art strategies, our approach shows marked performance gains with a better accuracy-speed trade-off regime for both mixed and sub-populations.

Multi-channel MRI segmentation of eye structures and tumors using patient-specific features

Retinoblastoma and uveal melanoma are fast spreading eye tumors usually diagnosed by using 2D Fundus Image Photography (Fundus) and 2D Ultrasound (US). Diagnosis and treatment planning of such diseases often require additional complementary imaging to confirm the tumor extend via 3D Magnetic Resonance Imaging (MRI). In this context, having automatic segmentations to estimate the size and the distribution of the pathological tissue would be advantageous towards tumor characterization. Until now, the alternative has been the manual delineation of eye structures, a rather time consuming and error-prone task, to be conducted in multiple MRI sequences simultaneously. This situation, and the lack of tools for accurate eye MRI analysis, reduces the interest in MRI beyond the qualitative evaluation of the optic nerve invasion and the confirmation of recurrent malignancies below calcified tumors. In this manuscript, we propose a new framework for the automatic segmentation of eye structures and ocular tumors in multi-sequence MRI. Our key contribution is the introduction of a pathological eye model from which Eye Patient-Specific Features (EPSF) can be computed. These features combine intensity and shape information of pathological tissue while embedded in healthy structures of the eye. We assess our work on a dataset of pathological patient eyes by computing the Dice Similarity Coefficient (DSC) of the sclera, the cornea, the vitreous humor, the lens and the tumor. In addition, we quantitatively show the superior performance of our pathological eye model as compared to the segmentation obtained by using a healthy model (over 4% DSC) and demonstrate the relevance of our EPSF, which improve the final segmentation regardless of the classifier employed.

Efficient OCT Volume Reconstruction From Slitlamp Microscopes

Since its introduction 25 years ago, Optical Coherence Tomography (OCT) has contributed tremendously to diagnostic and monitoring capabilities of pathologies in the field of ophthalmology. Despite rapid progress in hardware and software technology however, the price of OCT devices has remained high, limiting their use in private practice, and in screening examinations. In this paper, we present a slitlamp-integrated OCT device, built with off-the-shelf components, which can generate high-quality volumetric images of the posterior eye segment. To do so, we present a novel strategy for 3D image reconstruction in this challenging domain that allows us for state-of-the-art OCT volumes to be generated at fast speeds. The result is an OCT device that can match current systems in clinical practice, at a significantly lower cost.