Deep learning in macroscopic diffuse optical imaging

Deep learning-enabled fluorescence imaging for surgical guidance: in silico training for oral cancer depth quantification

Significance

Oral cancer surgery requires accurate margin delineation to balance complete resection with post-operative functionality. Current in vivo fluorescence imaging systems provide two-dimensional margin assessment yet fail to quantify tumor depth prior to resection. Harnessing structured light in combination with deep learning (DL) may provide near real-time three-dimensional margin detection.

Aim

A DL-enabled fluorescence spatial frequency domain imaging (SFDI) system trained with in silico tumor models was developed to quantify the depth of oral tumors.

Approach

A convolutional neural network was designed to produce tumor depth and concentration maps from SFDI images. Three in silico representations of oral cancer lesions were developed to train the DL architecture: cylinders, spherical harmonics, and composite spherical harmonics (CSHs). Each model was validated with in silico SFDI images of patient-derived tongue tumors, and the CSH model was further validated with optical phantoms.

Results

The performance of the CSH model was superior when presented with patient-derived tumors ( P -value < 0.05 ). The CSH model could predict depth and concentration within 0.4 mm and 0.4    μ g / mL , respectively, for in silico tumors with depths less than 10 mm.

Conclusions

A DL-enabled SFDI system trained with in silico CSH demonstrates promise in defining the deep margins of oral tumors.

Fluorescence diffuse optical monitoring of bioreactors: a hybrid deep learning and model-based approach for tomography

Biosynthesis in bioreactors plays a vital role in many applications, but tools for accurate in situ monitoring of the cells are still lacking. By engineering the cells such that their conditions are reported through fluorescence, it is possible to fill in the gap using fluorescence diffuse optical tomography (fDOT). However, the spatial accuracy of the reconstruction can still be limited, due to e.g. undersampling and inaccurate estimation of the optical properties. Utilizing controlled phantom studies, we use a two-step hybrid approach, where a preliminary fDOT result is first obtained using the classic model-based optimization, and then enhanced using a neural network. We show in this paper using both simulated and phantom experiments that the proposed method can lead to a 8-fold improvement (Intersection over Union) of fluorescence inclusion reconstruction in noisy conditions, at the same speed of conventional neural network-based methods. This is an important step towards our ultimate goal of fDOT monitoring of bioreactors.

Roadmap on computational methods in optical imaging and holography [invited]

Computational methods have been established as cornerstones in optical imaging and holography in recent years. Every year, the dependence of optical imaging and holography on computational methods is increasing significantly to the extent that optical methods and components are being completely and efficiently replaced with computational methods at low cost. This roadmap reviews the current scenario in four major areas namely incoherent digital holography, quantitative phase imaging, imaging through scattering layers, and super-resolution imaging. In addition to registering the perspectives of the modern-day architects of the above research areas, the roadmap also reports some of the latest studies on the topic. Computational codes and pseudocodes are presented for computational methods in a plug-and-play fashion for readers to not only read and understand but also practice the latest algorithms with their data. We believe that this roadmap will be a valuable tool for analyzing the current trends in computational methods to predict and prepare the future of computational methods in optical imaging and holography.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00340-024-08280-3.

Deep orthogonal multi-wavelength fusion for tomogram-free diagnosis in diffuse optical imaging

Novel portable diffuse optical tomography (DOT) devices for breast cancer lesions hold great promise for non-invasive, non-ionizing breast cancer screening. Critical to this capability is not just the identification of lesions but rather the complex problem of discriminating between malignant and benign lesions. To accurately reconstruct the highly heterogeneous tissue of a cancer lesion in healthy breast tissue using DOT, multiple wavelengths can be leveraged to maximize signal penetration while minimizing sensitivity to noise. However, these wavelength responses can overlap, capture common information, and correlate, potentially confounding reconstruction and downstream end tasks. We show that an orthogonal fusion loss regularizes multi-wavelength DOT leading to improved reconstruction and accuracy of end-to-end discrimination of malignant versus benign lesions. We further show that our raw-to-task model significantly reduces computational complexity without sacrificing accuracy, making it ideal for real-time throughput, desired in medical settings where handheld devices have severely restricted power budgets. Furthermore, our results indicate that image reconstruction is not necessary for unbiased classification of lesions with a balanced accuracy of 77% and 66% on the synthetic dataset and clinical dataset, respectively, using the raw-to-task model. Code is available at https://github.com/sfu-mial/FuseNet.

Subsurface fluorescence time-of-flight imaging using a large-format single-photon avalanche diode sensor for tumor depth assessment

Significance

Fluorescence guidance is used clinically by surgeons to visualize anatomical and/or physiological phenomena in the surgical field that are difficult or impossible to detect by the naked eye. Such phenomena include tissue perfusion or molecular phenotypic information about the disease being resected. Conventional fluorescence-guided surgery relies on long, microsecond scale laser pulses to excite fluorescent probes. However, this technique only provides two-dimensional information; crucial depth information, such as the location of malignancy below the tissue surface, is not provided.

Aim

We developed a depth sensing imaging technique using light detection and ranging (LiDAR) time-of-flight (TOF) technology to sense the depth of target tissue while overcoming the influence of tissue optical properties and fluorescent probe concentration.

Approach

The technology is based on a large-format (512 × 512   pixel ), binary, gated, single-photon avalanche diode (SPAD) sensor with an 18 ps time-gate step, synchronized with a picosecond pulsed laser. The fast response of the sensor was developed and tested for its ability to quantify fluorescent inclusions at depth and optical properties in tissue-like phantoms through analytical model fitting of the fast temporal remission data.

Results

After calibration and algorithmic extraction of the data, the SPAD LiDAR technique allowed for sub-mm resolution depth sensing of fluorescent inclusions embedded in tissue-like phantoms, up to a maximum of 5 mm in depth. The approach provides robust depth sensing even in the presence of variable tissue optical properties and separates the effects of fluorescence depth from absorption and scattering variations.

Conclusions

LiDAR TOF fluorescence imaging using an SPAD camera provides both fluorescence intensity images and the temporal profile of fluorescence, which can be used to determine the depth at which the signal is emitted over a wide field of view. The proposed tool enables fluorescence imaging at a higher depth in tissue and with higher spatial precision than standard, steady-state fluorescence imaging tools, such as intensity-based near-infrared fluorescence imaging, optical coherence tomography, Raman spectroscopy, or confocal microscopy. Integration of this technique into a standard surgical tool could enable rapid, more accurate estimation of resection boundaries, thereby improving the surgeon's efficacy and efficiency, and ultimately improving patient outcomes.

Noninvasive Visualization of Amyloid-Beta Deposits in Alzheimer's Amyloidosis Mice via Fluorescence Molecular Tomography Using Contrast Agent

Alzheimer's disease is pathologically featured by the accumulation of amyloid-beta (Aβ) plaque and neurofibrillary tangles. Compared to small animal positron emission tomography, optical imaging features nonionizing radiation, low cost, and logistic convenience. Optical detection of Aβ deposits is typically implemented by 2D macroscopic imaging and various microscopic techniques assisted with Aβ-targeted contrast agents. Here, we introduce fluorescence molecular tomography (FMT), a macroscopic 3D fluorescence imaging technique, convenient for in vivo longitudinal monitoring of the animal brain without the involvement of cranial window opening operation. This chapter aims to provide the protocols for FMT in vivo imaging of Aβ deposits in the brain of rodent model of Alzheimer's disease. The materials, stepwise method, notes, limitations of FMT, and emerging opportunities for FMT techniques are presented.

In vivo quantification of programmed death-ligand-1 expression heterogeneity in tumors using fluorescence lifetime imaging

Cancer patient selection for immunotherapy is often based on programmed death-ligand-1 (PD-L1) expression as a biomarker. PD-L1 expression is currently quantified using immunohistochemistry, which can only provide snapshots of PD-L1 expression status in microscopic regions of ex vivo specimens. In vivo imaging using targeted agents can capture dynamic variations of PD-L1 expression in entire tumors within and across multiple subjects. Towards this goal, several PD-L1 targeted molecular imaging probes have been evaluated in murine models and humans. However, clinical translation of these probes has been limited due to a significant non-specific accumulation of the imaging probes and the inability of conventional imaging modalities to provide quantitative readouts that can be compared across multiple subjects. Here we report that in vivo time-domain (TD) fluorescence imaging can provide quantitative estimates of baseline tumor PD-L1 heterogeneity across untreated mice and variations in PD-L1 expression across mice undergoing clinically relevant anti-PD1 treatment. This approach relies on a significantly longer fluorescence lifetime (FLT) of PD-L1 specific anti-PD-L1 antibody tagged to IRDye 800CW (αPDL1-800) compared to nonspecific αPDL1-800. Leveraging this unique FLT contrast, we show that PD-L1 expression can be quantified across mice both in superficial breast tumors using planar FLT imaging, and in deep-seated liver tumors (>5 mm depth) using the asymptotic TD algorithm for fluorescence tomography. Our results suggest that FLT contrast can accelerate the preclinical investigation and clinical translation of novel molecular imaging probes by providing robust quantitative readouts of receptor expression that can be readily compared across subjects.

Fusion deep learning approach combining diffuse optical tomography and ultrasound for improving breast cancer classification

Diffuse optical tomography (DOT) is a promising technique that provides functional information related to tumor angiogenesis. However, reconstructing the DOT function map of a breast lesion is an ill-posed and underdetermined inverse process. A co-registered ultrasound (US) system that provides structural information about the breast lesion can improve the localization and accuracy of DOT reconstruction. Additionally, the well-known US characteristics of benign and malignant breast lesions can further improve cancer diagnosis based on DOT alone. Inspired by a fusion model deep learning approach, we combined US features extracted by a modified VGG-11 network with images reconstructed from a DOT deep learning auto-encoder-based model to form a new neural network for breast cancer diagnosis. The combined neural network model was trained with simulation data and fine-tuned with clinical data: it achieved an AUC of 0.931 (95% CI: 0.919-0.943), superior to those achieved using US images alone (0.860) or DOT images alone (0.842).

Unrolled-DOT: an interpretable deep network for diffuse optical tomography

Significance

Imaging through scattering media is critical in many biomedical imaging applications, such as breast tumor detection and functional neuroimaging. Time-of-flight diffuse optical tomography (ToF-DOT) is one of the most promising methods for high-resolution imaging through scattering media. ToF-DOT and many traditional DOT methods require an image reconstruction algorithm. Unfortunately, this algorithm often requires long computational runtimes and may produce lower quality reconstructions in the presence of model mismatch or improper hyperparameter tuning.

Aim

We used a data-driven unrolled network as our ToF-DOT inverse solver. The unrolled network is faster than traditional inverse solvers and achieves higher reconstruction quality by accounting for model mismatch.

Approach

Our model "Unrolled-DOT" uses the learned iterative shrinkage thresholding algorithm. In addition, we incorporate a refinement U-Net and Visual Geometry Group (VGG) perceptual loss to further increase the reconstruction quality. We trained and tested our model on simulated and real-world data and benchmarked against physics-based and learning-based inverse solvers.

Results

In experiments on real-world data, Unrolled-DOT outperformed learning-based algorithms and achieved over 10× reduction in runtime and mean-squared error, compared to traditional physics-based solvers.

Conclusion

We demonstrated a learning-based ToF-DOT inverse solver that achieves state-of-the-art performance in speed and reconstruction quality, which can aid in future applications for noninvasive biomedical imaging.

Periodic-net: an end-to-end data driven framework for diffuse optical imaging of breast cancer from noisy boundary data

Significance

The machine learning (ML) approach plays a critical role in assessing biomedical imaging processes especially optical imaging (OI) including segmentation, classification, and reconstruction, intending to achieve higher accuracy efficiently.

Aim

This research aims to develop an end-to-end deep learning framework for diffuse optical imaging (DOI) with multiple datasets to detect breast cancer and reconstruct its optical properties in the early stages.

Approach

The proposed Periodic-net is a nondestructive deep learning (DL) algorithm for the reconstruction and evaluation of inhomogeneities in an inverse model with high accuracy, while boundary measurements are calculated by solving a forward problem with sources/detectors arranged uniformly around a circular domain in various combinations, including 16 × 15 , 20 × 19 , and 36 × 35 boundary measurement setups.

Results

The results of image reconstruction on numerical and phantom datasets demonstrate that the proposed network provides higher-quality images with a greater amount of small details, superior immunity to noise, and sharper edges with a reduction in image artifacts than other state-of-the-art competitors.

Conclusions

The network is highly effective at the simultaneous reconstruction of optical properties, i.e., absorption and reduced scattering coefficients, by optimizing the imaging time without degrading inclusions localization and image quality.

Deep learning in fNIRS: a review

Significance: Optical neuroimaging has become a well-established clinical and research tool to monitor cortical activations in the human brain. It is notable that outcomes of functional near-infrared spectroscopy (fNIRS) studies depend heavily on the data processing pipeline and classification model employed. Recently, deep learning (DL) methodologies have demonstrated fast and accurate performances in data processing and classification tasks across many biomedical fields. Aim: We aim to review the emerging DL applications in fNIRS studies. Approach: We first introduce some of the commonly used DL techniques. Then, the review summarizes current DL work in some of the most active areas of this field, including brain-computer interface, neuro-impairment diagnosis, and neuroscience discovery. Results: Of the 63 papers considered in this review, 32 report a comparative study of DL techniques to traditional machine learning techniques where 26 have been shown outperforming the latter in terms of the classification accuracy. In addition, eight studies also utilize DL to reduce the amount of preprocessing typically done with fNIRS data or increase the amount of data via data augmentation. Conclusions: The application of DL techniques to fNIRS studies has shown to mitigate many of the hurdles present in fNIRS studies such as lengthy data preprocessing or small sample sizes while achieving comparable or improved classification accuracy.

Design and characterization of a time-domain optical tomography platform for mesoscopic lifetime imaging

We report on the system design and instrumental characteristics of a novel time-domain mesoscopic fluorescence molecular tomography (TD-MFMT) system for multiplexed molecular imaging in turbid media. The system is equipped with a supercontinuum pulsed laser for broad spectral excitation, based on a high-density descanned raster scanning intensity-based acquisition for 2D and 3D imaging and augmented with a high-dynamical range linear time-resolved single-photon avalanche diode (SPAD) array for lifetime quantification. We report on the system's spatio-temporal and spectral characteristics and its sensitivity and specificity in controlled experimental settings. Also, a phantom study is undertaken to test the performance of the system to image deeply-seated fluorescence inclusions in tissue-like media. In addition, ex vivo tumor xenograft imaging is performed to validate the system's applicability to the biological sample. The characterization results manifest the capability to sense small fluorescence concentrations (on the order of nanomolar) while quantifying fluorescence lifetimes and lifetime-based parameters at high resolution. The phantom results demonstrate the system's potential to perform 3D multiplexed imaging thanks to spectral and lifetime contrast in the mesoscopic range (at millimeters depth). The ex vivo imaging exhibits the prospect of TD-MFMT to resolve intra-tumoral heterogeneity in a depth-dependent manner.

Lawrence K, Tachtsidis I, Tong Y, Torricelli A, Urner T, Wabnitz H, Wolf M, Wolf U, Xu S, Yang C, Yodh AG, Yücel MA, Zhou W. Optical imaging and spectroscopy for the study of the human brain: status report

This report is the second part of a comprehensive two-part series aimed at reviewing an extensive and diverse toolkit of novel methods to explore brain health and function. While the first report focused on neurophotonic tools mostly applicable to animal studies, here, we highlight optical spectroscopy and imaging methods relevant to noninvasive human brain studies. We outline current state-of-the-art technologies and software advances, explore the most recent impact of these technologies on neuroscience and clinical applications, identify the areas where innovation is needed, and provide an outlook for the future directions.

Monte Carlo-based data generation for efficient deep learning reconstruction of macroscopic diffuse optical tomography and topography applications

SIGNIFICANCE

Deep learning (DL) models are being increasingly developed to map sensor data to the image domain directly. However, DL methodologies are data-driven and require large and diverse data sets to provide robust and accurate image formation performances. For research modalities such as 2D/3D diffuse optical imaging, the lack of large publicly available data sets and the wide variety of instrumentation designs, data types, and applications leads to unique challenges in obtaining well-controlled data sets for training and validation. Meanwhile, great efforts over the last four decades have focused on developing accurate and computationally efficient light propagation models that are flexible enough to simulate a wide variety of experimental conditions.

AIM

Recent developments in Monte Carlo (MC)-based modeling offer the unique advantage of simulating accurately light propagation spatially, temporally, and over an extensive range of optical parameters, including minimally to highly scattering tissue within a computationally efficient platform. Herein, we demonstrate how such MC platforms, namely "Monte Carlo eXtreme" and "Mesh-based Monte Carlo," can be leveraged to generate large and representative data sets for training the DL model efficiently.

APPROACH

We propose data generator pipeline strategies using these platforms and demonstrate their potential in fluorescence optical topography, fluorescence optical tomography, and single-pixel diffuse optical tomography. These applications represent a large variety in instrumentation design, sample properties, and contrast function.

RESULTS

DL models trained using the MC-based in silico datasets, validated further with experimental data not used during training, show accurate and promising results.

CONCLUSION

Overall, these MC-based data generation pipelines are expected to support the development of DL models for rapid, robust, and user-friendly image formation in a wide variety of applications.

3D k-space reflectance fluorescence tomography via deep learning

We report on the potential to perform image reconstruction in 3D k-space reflectance fluorescence tomography (FT) using deep learning (DL). Herein, we adopt a modified AUTOMAP architecture and develop a training methodology that leverages an open-source Monte-Carlo-based simulator to generate a large dataset. Using an enhanced EMNIST (EEMNIST) dataset as an embedded contrast function allows us to train the network efficiently. The optical strategy utilizes k-space illumination in a reflectance configuration to probe tissue in the mesoscopic regime with high sensitivity and resolution. The proposed DL model training and validation is performed with both in silico data and a phantom experiment. Overall, our results indicate that the approach can correctly reconstruct both single and multiple fluorescent embedding(s) in a 3D volume. Furthermore, the presented technique is shown to outperform the traditional approaches [least-squares (LSQ) and total-variation minimization (TVAL)], especially at higher depths. We, therefore, expect the proposed computational technique to have future implications in preclinical studies.