A Test Statistic Estimation-Based Approach for Establishing Self-Interpretable CNN-Based Binary Classifiers

Interpretability is highly desired for deep neural network-based classifiers, especially when addressing high-stake decisions in medical imaging. Commonly used post-hoc interpretability methods have the limitation that they can produce plausible but different interpretations of a given model, leading to ambiguity about which one to choose. To address this problem, a novel decision-theory-inspired approach is investigated to establish a self-interpretable model, given a pre-trained deep binary black-box medical image classifier. This approach involves utilizing a self-interpretable encoder-decoder model in conjunction with a single-layer fully connected network with unity weights. The model is trained to estimate the test statistic of the given trained black-box deep binary classifier to maintain a similar accuracy. The decoder output image, referred to as an equivalency map, is an image that represents a transformed version of the to-be-classified image that, when processed by the fixed fully connected layer, produces the same test statistic value as the original classifier. The equivalency map provides a visualization of the transformed image features that directly contribute to the test statistic value and, moreover, permits quantification of their relative contributions. Unlike the traditional post-hoc interpretability methods, the proposed method is self-interpretable, quantitative. Detailed quantitative and qualitative analyses have been performed with three different medical image binary classification tasks.

Context-aware deep learning enables high-efficacy localization of high concentration microbubbles for super-resolution ultrasound localization microscopy

Ultrasound localization microscopy (ULM) enables deep tissue microvascular imaging by localizing and tracking intravenously injected microbubbles circulating in the bloodstream. However, conventional localization techniques require spatially isolated microbubbles, resulting in prolonged imaging time to obtain detailed microvascular maps. Here, we introduce LOcalization with Context Awareness (LOCA)-ULM, a deep learning-based microbubble simulation and localization pipeline designed to enhance localization performance in high microbubble concentrations. In silico, LOCA-ULM enhanced microbubble detection accuracy to 97.8% and reduced the missing rate to 23.8%, outperforming conventional and deep learning-based localization methods up to 17.4% in accuracy and 37.6% in missing rate reduction. In in vivo rat brain imaging, LOCA-ULM revealed dense cerebrovascular networks and spatially adjacent microvessels undetected by conventional ULM. We further demonstrate the superior localization performance of LOCA-ULM in functional ULM (fULM) where LOCA-ULM significantly increased the functional imaging sensitivity of fULM to hemodynamic responses invoked by whisker stimulations in the rat brain.

Deep-learning-based image super-resolution of an end-expandable optical fiber probe for application in esophageal cancer diagnostics

Significance

Endoscopic screening for esophageal cancer (EC) may enable early cancer diagnosis and treatment. While optical microendoscopic technology has shown promise in improving specificity, the limited field of view (< 1    mm ) significantly reduces the ability to survey large areas efficiently in EC screening.

Aim

To improve the efficiency of endoscopic screening, we propose a novel concept of end-expandable endoscopic optical fiber probe for larger field of visualization and for the first time evaluate a deep-learning-based image super-resolution (DL-SR) method to overcome the issue of limited sampling capability.

Approach

To demonstrate feasibility of the end-expandable optical fiber probe, DL-SR was applied on simulated low-resolution microendoscopic images to generate super-resolved (SR) ones. Varying the degradation model of image data acquisition, we identified the optimal parameters for optical fiber probe prototyping. The proposed screening method was validated with a human pathology reading study.

Results

For various degradation parameters considered, the DL-SR method demonstrated different levels of improvement of traditional measures of image quality. The endoscopists' interpretations of the SR images were comparable to those performed on the high-resolution ones.

Conclusions

This work suggests avenues for development of DL-SR-enabled sparse image reconstruction to improve high-yield EC screening and similar clinical applications.

EVATOM: an optical, label-free, machine learning assisted embryo health assessment tool

The combination of a good quality embryo and proper maternal health factors promise higher chances of a successful in vitro fertilization (IVF) procedure leading to clinical pregnancy and live birth. Of these two factors, selection of a good embryo is a controllable aspect. The current gold standard in clinical practice is visual assessment of an embryo based on its morphological appearance by trained embryologists. More recently, machine learning has been incorporated into embryo selection "packages". Here, we report EVATOM: a machine-learning assisted embryo health assessment tool utilizing an optical quantitative phase imaging technique called artificial confocal microscopy (ACM). We present a label-free nucleus detection method with, to the best of our knowledge, novel quantitative embryo health biomarkers. Two viability assessment models are presented for grading embryos into two classes: healthy/intermediate (H/I) or sick (S) class. The models achieve a weighted F1 score of 1.0 and 0.99 respectively on the in-distribution test set of 72 fixed embryos and a weighted F1 score of 0.9 and 0.95 respectively on the out-of-distribution test dataset of 19 time-instances from 8 live embryos.

Grand Challenges at the Interface of Engineering and Medicine

Over the past two decades Biomedical Engineering has emerged as a major discipline that bridges societal needs of human health care with the development of novel technologies. Every medical institution is now equipped at varying degrees of sophistication with the ability to monitor human health in both non-invasive and invasive modes. The multiple scales at which human physiology can be interrogated provide a profound perspective on health and disease. We are at the nexus of creating "avatars" (herein defined as an extension of "digital twins") of human patho/physiology to serve as paradigms for interrogation and potential intervention. Motivated by the emergence of these new capabilities, the IEEE Engineering in Medicine and Biology Society, the Departments of Biomedical Engineering at Johns Hopkins University and Bioengineering at University of California at San Diego sponsored an interdisciplinary workshop to define the grand challenges that face biomedical engineering and the mechanisms to address these challenges. The Workshop identified five grand challenges with cross-cutting themes and provided a roadmap for new technologies, identified new training needs, and defined the types of interdisciplinary teams needed for addressing these challenges. The themes presented in this paper include: 1) accumedicine through creation of avatars of cells, tissues, organs and whole human; 2) development of smart and responsive devices for human function augmentation; 3) exocortical technologies to understand brain function and treat neuropathologies; 4) the development of approaches to harness the human immune system for health and wellness; and 5) new strategies to engineer genomes and cells.

High-Dimensional MR Reconstruction Integrating Subspace and Adaptive Generative Models

OBJECTIVE

To develop a new method that integrates subspace and generative image models for high-dimensional MR image reconstruction.

METHODS

We proposed a formulation that synergizes a low-dimensional subspace model of high-dimensional images, an adaptive generative image prior serving as spatial constraints on the sequence of "contrast-weighted" images or spatial coefficients of the subspace model, and a conventional sparsity regularization. A special pretraining plus subject-specific network adaptation strategy was proposed to construct an accurate generative-network-based representation for images with varying contrasts. An iterative algorithm was introduced to jointly update the subspace coefficients and the multi-resolution latent space of the generative image model that leveraged an recently proposed intermediate layer optimization technique for network inversion.

RESULTS

We evaluated the utility of the proposed method for two high-dimensional imaging applications: accelerated MR parameter mapping and high-resolution MR spectroscopic imaging. Improved performance over state-of-the-art subspace-based methods was demonstrated in both cases.

CONCLUSION

The proposed method provided a new way to address high-dimensional MR image reconstruction problems by incorporating an adaptive generative model as a data-driven spatial prior for constraining subspace reconstruction.

SIGNIFICANCE

Our work demonstrated the potential of integrating data-driven and adaptive generative priors with canonical low-dimensional modeling for high-dimensional imaging problems.

Attention-Based CNN-BiLSTM for Sleep State Classification of Spatiotemporal Wide-Field Calcium Imaging Data

BACKGROUND

Wide-field calcium imaging (WFCI) with genetically encoded calcium indicators allows for spatiotemporal recordings of neuronal activity in mice. When applied to the study of sleep, WFCI data are manually scored into the sleep states of wakefulness, non-REM (NREM) and REM by use of adjunct EEG and EMG recordings. However, this process is time-consuming, invasive and often suffers from low inter- and intra-rater reliability. Therefore, an automated sleep state classification method that operates on spatiotemporal WFCI data is desired.

NEW METHOD

A hybrid network architecture consisting of a convolutional neural network (CNN) to extract spatial features of image frames and a bidirectional long short-term memory network (BiLSTM) with attention mechanism to identify temporal dependencies among different time points was proposed to classify WFCI data into states of wakefulness, NREM and REM sleep.

RESULTS

Sleep states were classified with an accuracy of 84% and Cohen's kappa of 0.64. Gradient-weighted class activation maps revealed that the frontal region of the cortex carries more importance when classifying WFCI data into NREM sleep while posterior area contributes most to the identification of wakefulness. The attention scores indicated that the proposed network focuses on short- and long-range temporal dependency in a state-specific manner.

COMPARISON WITH EXISTING METHOD

On a 3-hour WFCI recording, the CNN-BiLSTM achieved a kappa of 0.67, comparable to a kappa of 0.65 corresponding to the human EEG/EMG-based scoring.

CONCLUSIONS

The CNN-BiLSTM effectively classifies sleep states from spatiotemporal WFCI data and will enable broader application of WFCI in sleep.

Spatiotemporal image reconstruction to enable high-frame-rate dynamic photoacoustic tomography with rotating-gantry volumetric imagers

Significance

Dynamic photoacoustic computed tomography (PACT) is a valuable imaging technique for monitoring physiological processes. However, current dynamic PACT imaging techniques are often limited to two-dimensional spatial imaging. Although volumetric PACT imagers are commercially available, these systems typically employ a rotating measurement gantry in which the tomographic data are sequentially acquired as opposed to being acquired simultaneously at all views. Because the dynamic object varies during the data-acquisition process, the sequential data-acquisition process poses substantial challenges to image reconstruction associated with data incompleteness. The proposed image reconstruction method is highly significant in that it will address these challenges and enable volumetric dynamic PACT imaging with existing preclinical imagers.

Aim

The aim of this study is to develop a spatiotemporal image reconstruction (STIR) method for dynamic PACT that can be applied to commercially available volumetric PACT imagers that employ a sequential scanning strategy. The proposed reconstruction method aims to overcome the challenges caused by the limited number of tomographic measurements acquired per frame.

Approach

A low-rank matrix estimation-based STIR (LRME-STIR) method is proposed to enable dynamic volumetric PACT. The LRME-STIR method leverages the spatiotemporal redundancies in the dynamic object to accurately reconstruct a four-dimensional (4D) spatiotemporal image.

Results

The conducted numerical studies substantiate the LRME-STIR method's efficacy in reconstructing 4D dynamic images from tomographic measurements acquired with a rotating measurement gantry. The experimental study demonstrates the method's ability to faithfully recover the flow of a contrast agent with a frame rate of 10 frames per second, even when only a single tomographic measurement per frame is available.

Conclusions

The proposed LRME-STIR method offers a promising solution to the challenges faced by enabling 4D dynamic imaging using commercially available volumetric PACT imagers. By enabling accurate STIRs, this method has the potential to significantly advance preclinical research and facilitate the monitoring of critical physiological biomarkers.

Ideal Observer Computation by Use of Markov-Chain Monte Carlo With Generative Adversarial Networks

Medical imaging systems are often evaluated and optimized via objective, or task-specific, measures of image quality (IQ) that quantify the performance of an observer on a specific clinically-relevant task. The performance of the Bayesian Ideal Observer (IO) sets an upper limit among all observers, numerical or human, and has been advocated for use as a figure-of-merit (FOM) for evaluating and optimizing medical imaging systems. However, the IO test statistic corresponds to the likelihood ratio that is intractable to compute in the majority of cases. A sampling-based method that employs Markov-chain Monte Carlo (MCMC) techniques was previously proposed to estimate the IO performance. However, current applications of MCMC methods for IO approximation have been limited to a small number of situations where the considered distribution of to-be-imaged objects can be described by a relatively simple stochastic object model (SOM). As such, there remains an important need to extend the domain of applicability of MCMC methods to address a large variety of scenarios where IO-based assessments are needed but the associated SOMs have not been available. In this study, a novel MCMC method that employs a generative adversarial network (GAN)-based SOM, referred to as MCMC-GAN, is described and evaluated. The MCMC-GAN method was quantitatively validated by use of test-cases for which reference solutions were available. The results demonstrate that the MCMC-GAN method can extend the domain of applicability of MCMC methods for conducting IO analyses of medical imaging systems.

Identifying Naturalistic Movies from Human Brain Activity with High-Density Diffuse Optical Tomography

Modern neuroimaging modalities, particularly functional MRI (fMRI), can decode detailed human experiences. Thousands of viewed images can be identified or classified, and sentences can be reconstructed. Decoding paradigms often leverage encoding models that reduce the stimulus space into a smaller yet generalizable feature set. However, the neuroimaging devices used for detailed decoding are non-portable, like fMRI, or invasive, like electrocorticography, excluding application in naturalistic use. Wearable, non-invasive, but lower-resolution devices such as electroencephalography and functional near-infrared spectroscopy (fNIRS) have been limited to decoding between stimuli used during training. Herein we develop and evaluate model-based decoding with high-density diffuse optical tomography (HD-DOT), a higher-resolution expansion of fNIRS with demonstrated promise as a surrogate for fMRI. Using a motion energy model of visual content, we decoded the identities of novel movie clips outside the training set with accuracy far above chance for single-trial decoding. Decoding was robust to modulations of testing time window, different training and test imaging sessions, hemodynamic contrast, and optode array density. Our results suggest that HD-DOT can translate detailed decoding into naturalistic use.

A Forward Model Incorporating Elevation-Focused Transducer Properties for 3-D Full-Waveform Inversion in Ultrasound Computed Tomography

Ultrasound computed tomography (USCT) is an emerging medical imaging modality that holds great promise for improving human health. Full-waveform inversion (FWI)-based image reconstruction methods account for the relevant wave physics to produce high spatial resolution images of the acoustic properties of the breast tissues. A practical USCT design employs a circular ring-array comprised of elevation-focused ultrasonic transducers, and volumetric imaging is achieved by translating the ring-array orthogonally to the imaging plane. In commonly deployed slice-by-slice (SBS) reconstruction approaches, the 3-D volume is reconstructed by stacking together 2-D images reconstructed for each position of the ring-array. A limitation of the SBS reconstruction approach is that it does not account for 3-D wave propagation physics and the focusing properties of the transducers, which can result in significant image artifacts and inaccuracies. To perform 3-D image reconstruction when elevation-focused transducers are employed, a numerical description of the focusing properties of the transducers should be included in the forward model. To address this, a 3-D computational model of an elevation-focused transducer is developed to enable 3-D FWI-based reconstruction methods to be deployed in ring-array-based USCT. The focusing is achieved by applying a spatially varying temporal delay to the ultrasound pulse (emitter mode) and recorded signal (receiver mode). The proposed numerical transducer model is quantitatively validated and employed in computer simulation studies that demonstrate its use in image reconstruction for ring-array USCT.

Contrast-Free Super-Resolution Power Doppler (CS-PD) Based on Deep Neural Networks

Super-resolution ultrasound microvessel imaging based on ultrasound localization microscopy (ULM) is an emerging imaging modality that is capable of resolving micrometer-scaled vessels deep into tissue. In practice, ULM is limited by the need for contrast injection, long data acquisition, and computationally expensive postprocessing times. In this study, we present a contrast-free super-resolution power Doppler (CS-PD) technique that uses deep networks to achieve super-resolution with short data acquisition. The training dataset is comprised of spatiotemporal ultrafast ultrasound signals acquired from in vivo mouse brains, while the testing dataset includes in vivo mouse brain, chicken embryo chorioallantoic membrane (CAM), and healthy human subjects. The in vivo mouse imaging studies demonstrate that CS-PD could achieve an approximate twofold improvement in spatial resolution when compared with conventional power Doppler. In addition, the microvascular images generated by CS-PD showed good agreement with the corresponding ULM images as indicated by a structural similarity index of 0.7837 and a peak signal-to-noise ratio (PSNR) of 25.52. Moreover, CS-PD was able to preserve the temporal profile of the blood flow (e.g., pulsatility) that is similar to conventional power Doppler. Finally, the generalizability of CS-PD was demonstrated on testing data of different tissues using different imaging settings. The fast inference time of the proposed deep neural network also allows CS-PD to be implemented for real-time imaging. These features of CS-PD offer a practical, fast, and robust microvascular imaging solution for many preclinical and clinical applications of Doppler ultrasound.

A forward model incorporating elevation-focused transducer properties for 3D full-waveform inversion in ultrasound computed tomography

Ultrasound computed tomography (USCT) is an emerging medical imaging modality that holds great promise for improving human health. Full-waveform inversion (FWI)-based image reconstruction methods account for the relevant wave physics to produce high spatial resolution images of the acoustic properties of the breast tissues. A practical USCT design employs a circular ring-array comprised of elevation-focused ultrasonic transducers, and volumentric imaging is achieved by translating the ring-array orthogonally to the imaging plane. In commonly deployed slice-by-slice (SBS) reconstruction approaches, the three-dimensional (3D) volume is reconstructed by stacking together two-dimensional (2D) images reconstructed for each position of the ring-array. A limitation of the SBS reconstruction approach is that it does not account for 3D wave propagation physics and the focusing properties of the transducers, which can result in significant image artifacts and inaccuracies. To perform 3D image reconstruction when elevation-focused transducers are employed, a numerical description of the focusing properties of the transducers should be included in the forward model. To address this, a 3D computational model of an elevation-focused transducer is developed to enable 3D FWI-based reconstruction methods to be deployed in ring-array-based USCT. The focusing is achieved by applying a spatially varying temporal delay to the ultrasound pulse (emitter mode) and recorded signal (receiver mode). The proposed numerical transducer model is quantitatively validated and employed in computer-simulation studies that demonstrate its use in image reconstruction for ring-array USCT.

Approximating the Hotelling observer with autoencoder-learned efficient channels for binary signal detection tasks

Purpose

The objective assessment of image quality (IQ) has been advocated for the analysis and optimization of medical imaging systems. One method of computing such IQ metrics is through a numerical observer. The Hotelling observer (HO) is the optimal linear observer, but conventional methods for obtaining the HO can become intractable due to large image sizes or insufficient data. Channelized methods are sometimes employed in such circumstances to approximate the HO. The performance of such channelized methods varies, with different methods obtaining superior performance to others depending on the imaging conditions and detection task. A channelized HO method using an AE is presented and implemented across several tasks to characterize its performance.

Approach

The process for training an AE is demonstrated to be equivalent to developing a set of channels for approximating the HO. The efficiency of the learned AE-channels is increased by modifying the conventional AE loss function to incorporate task-relevant information. Multiple binary detection tasks involving lumpy and breast phantom backgrounds across varying dataset sizes are considered to evaluate the performance of the proposed method and compare to current state-of-the-art channelized methods. Additionally, the ability of the channelized methods to generalize to images outside of the training dataset is investigated.

Results

AE-learned channels are demonstrated to have comparable performance with other state-of-the-art channel methods in the detection studies and superior performance in the generalization studies. Incorporating a cleaner estimate of the signal for the detection task is also demonstrated to significantly improve the performance of the proposed method, particularly in datasets with fewer images.

Conclusions

AEs are demonstrated to be capable of learning efficient channels for the HO. The resulting significant increase in detection performance for small dataset sizes when incorporating a signal prior holds promising implications for future assessments of imaging technologies.

Learned Full Waveform Inversion Incorporating Task Information for Ultrasound Computed Tomography

Ultrasound computed tomography (USCT) is an emerging imaging modality that holds great promise for breast imaging. Full-waveform inversion (FWI)-based image reconstruction methods incorporate accurate wave physics to produce high spatial resolution quantitative images of speed of sound or other acoustic properties of the breast tissues from USCT measurement data. However, the high computational cost of FWI reconstruction represents a significant burden for its widespread application in a clinical setting. The research reported here investigates the use of a convolutional neural network (CNN) to learn a mapping from USCT waveform data to speed of sound estimates. The CNN was trained using a supervised approach with a task-informed loss function aiming at preserving features of the image that are relevant to the detection of lesions. A large set of anatomically and physiologically realistic numerical breast phantoms (NBPs) and corresponding simulated USCT measurements was employed during training. Once trained, the CNN can perform real-time FWI image reconstruction from USCT waveform data. The performance of the proposed method was assessed and compared against FWI using a hold-out sample of 41 NBPs and corresponding USCT data. Accuracy was measured using relative mean square error (RMSE), structural self-similarity index measure (SSIM), and lesion detection performance (DICE score). This numerical experiment demonstrates that a supervised learning model can achieve accuracy comparable to FWI in terms of RMSE and SSIM, and better performance in terms of task performance, while significantly reducing computational time.

Machine learning assisted health viability assay for mouse embryos with artificial confocal microscopy (ACM)

The combination of a good quality embryo and proper maternal health factors promise higher chances of a successful in vitro fertilization (IVF) procedure leading to clinical pregnancy and live birth. Of these two factors, selection of a good embryo is a controllable aspect. The current gold standard in clinical practice is visual assessment of an embryo based on its morphological appearance by trained embryologists. More recently, machine learning has been incorporated into embryo selection "packages". Here, we report a machine-learning assisted embryo health assessment tool utilizing a quantitative phase imaging technique called artificial confocal microscopy (ACM). We present a label-free nucleus detection method with novel quantitative embryo health biomarkers. Two viability assessment models are presented for grading embryos into two classes: healthy/intermediate (H/I) or sick (S) class. The models achieve a weighted F1 score of 1.0 and 0.99 respectively on the in-distribution test set of 72 fixed embryos and a weighted F1 score of 0.9 and 0.95 respectively on the out-of-distribution test dataset of 19 time-instances from 8 live embryos.

Stochastic three-dimensional numerical phantoms to enable computational studies in quantitative optoacoustic computed tomography of breast cancer

Significance

When developing a new quantitative optoacoustic computed tomography (OAT) system for diagnostic imaging of breast cancer, objective assessments of various system designs through human trials are infeasible due to cost and ethical concerns. In prototype stages, however, different system designs can be cost-efficiently assessed via virtual imaging trials (VITs) employing ensembles of digital breast phantoms, i.e., numerical breast phantoms (NBPs), that convey clinically relevant variability in anatomy and optoacoustic tissue properties.

Aim

The aim is to develop a framework for generating ensembles of realistic three-dimensional (3D) anatomical, functional, optical, and acoustic NBPs and numerical lesion phantoms (NLPs) for use in VITs of OAT applications in the diagnostic imaging of breast cancer.

Approach

The generation of the anatomical NBPs was accomplished by extending existing NBPs developed by the U.S. Food and Drug Administration. As these were designed for use in mammography applications, substantial modifications were made to improve blood vasculature modeling for use in OAT. The NLPs were modeled to include viable tumor cells only or a combination of viable tumor cells, necrotic core, and peripheral angiogenesis region. Realistic optoacoustic tissue properties were stochastically assigned in the NBPs and NLPs.

Results

To advance optoacoustic and optical imaging research, 84 datasets have been released; these consist of anatomical, functional, optical, and acoustic NBPs and the corresponding simulated multi-wavelength optical fluence, initial pressure, and OAT measurements. The generated NBPs were compared with clinical data with respect to the volume of breast blood vessels and spatially averaged effective optical attenuation. The usefulness of the proposed framework was demonstrated through a case study to investigate the impact of acoustic heterogeneity on OAT images of the breast.

Conclusions

The proposed framework will enhance the authenticity of virtual OAT studies and can be widely employed for the investigation and development of advanced image reconstruction and machine learning-based methods, as well as the objective evaluation and optimization of the OAT system designs.

Assessing the Ability of Generative Adversarial Networks to Learn Canonical Medical Image Statistics

In recent years, generative adversarial networks (GANs) have gained tremendous popularity for potential applications in medical imaging, such as medical image synthesis, restoration, reconstruction, translation, as well as objective image quality assessment. Despite the impressive progress in generating high-resolution, perceptually realistic images, it is not clear if modern GANs reliably learn the statistics that are meaningful to a downstream medical imaging application. In this work, the ability of a state-of-the-art GAN to learn the statistics of canonical stochastic image models (SIMs) that are relevant to objective assessment of image quality is investigated. It is shown that although the employed GAN successfully learned several basic first- and second-order statistics of the specific medical SIMs under consideration and generated images with high perceptual quality, it failed to correctly learn several per-image statistics pertinent to the these SIMs, highlighting the urgent need to assess medical image GANs in terms of objective measures of image quality.

Investigating the robustness of a deep learning-based method for quantitative phase retrieval from propagation-based x-ray phase contrast measurements under laboratory conditions

Objective.Quantitative phase retrieval (QPR) in propagation-based x-ray phase contrast imaging of heterogeneous and structurally complicated objects is challenging under laboratory conditions due to partial spatial coherence and polychromaticity. A deep learning-based method (DLBM) provides a nonlinear approach to this problem while not being constrained by restrictive assumptions about object properties and beam coherence. The objective of this work is to assess a DLBM for its applicability under practical scenarios by evaluating its robustness and generalizability under typical experimental variations.Approach.Towards this end, an end-to-end DLBM was employed for QPR under laboratory conditions and its robustness was investigated across various system and object conditions. The robustness of the method was tested via varying propagation distances and its generalizability with respect to object structure and experimental data was also tested.Main results.Although the end-to-end DLBM was stable under the studied variations, its successful deployment was found to be affected by choices pertaining to data pre-processing, network training considerations and system modeling.Significance.To our knowledge, we demonstrated for the first time, the potential applicability of an end-to-end learning-based QPR method, trained on simulated data, to experimental propagation-based x-ray phase contrast measurements acquired under laboratory conditions with a commercial x-ray source and a conventional detector. We considered conditions of polychromaticity, partial spatial coherence, and high noise levels, typical to laboratory conditions. This work further explored the robustness of this method to practical variations in propagation distances and object structure with the goal of assessing its potential for experimental use. Such an exploration of any DLBM (irrespective of its network architecture) before practical deployment provides an understanding of its potential behavior under experimental settings.

White blood cell detection, classification and analysis using phase imaging with computational specificity (PICS)

Treatment of blood smears with Wright's stain is one of the most helpful tools in detecting white blood cell abnormalities. However, to diagnose leukocyte disorders, a clinical pathologist must perform a tedious, manual process of locating and identifying individual cells. Furthermore, the staining procedure requires considerable preparation time and clinical infrastructure, which is incompatible with point-of-care diagnosis. Thus, rapid and automated evaluations of unlabeled blood smears are highly desirable. In this study, we used color spatial light interference microcopy (cSLIM), a highly sensitive quantitative phase imaging (QPI) technique, coupled with deep learning tools, to localize, classify and segment white blood cells (WBCs) in blood smears. The concept of combining QPI label-free data with AI for the purpose of extracting cellular specificity has recently been introduced in the context of fluorescence imaging as phase imaging with computational specificity (PICS). We employed AI models to first translate SLIM images into brightfield micrographs, then ran parallel tasks of locating and labelling cells using EfficientNet, which is an object detection model. Next, WBC binary masks were created using U-net, a convolutional neural network that performs precise segmentation. After training on digitally stained brightfield images of blood smears with WBCs, we achieved a mean average precision of 75% for localizing and classifying neutrophils, eosinophils, lymphocytes, and monocytes, and an average pixel-wise majority-voting F1 score of 80% for determining the cell class from semantic segmentation maps. Therefore, PICS renders and analyzes synthetically stained blood smears rapidly, at a reduced cost of sample preparation, providing quantitative clinical information.

Dynamic Heterochromatin States in Anisotropic Nuclei of Cells on Aligned Nanofibers

The cancer cell nucleus deforms as it invades the interstitial spaces in tissues and the tumor microenvironment. While alteration of the chromatin structure in a deformed nucleus is expected and documented, the chromatin structure in the nuclei of cells on aligned matrices has not been elucidated. In this work we elucidate the spatiotemporal organization of heterochromatin in the elongated nuclei of cells on aligned nanofibers with stimulated emission depletion nanoscopy and fluorescence correlation spectroscopy. We show that the anisotropy of nuclei is sufficient to drive H3K9me3-heterochromatin alterations, with enhanced H3K9me3 nanocluster compaction and aggregation states that otherwise are indistinguishable from diffraction-limited microscopy. We interrogated the higher-order heterochromatin structures within major chromatin compartments in anisotropic nuclei and discovered a wider spatial dispersion of nanodomain clusters in the nucleoplasm and condensed larger nanoclusters near the periphery and pericentromeric heterochromatin. Upon examining the spatiotemporal dynamics of heterochromatin in anisotropic nuclei, we observed reduced mobility of the constitutive heterochromatin mark H3K9me3 and the associated heterochromatin protein 1 (HP1α) at the nucleoplasm and periphery regions, correlating with increased viscosity and changes in gene expression. Since heterochromatin remodeling is crucial to genome integrity, our results reveal an unconventional H3K9me3 heterochromatin distribution, providing cues to an altered chromatin state due to perturbations of the nuclei in aligned fiber configurations.

A Hybrid Approach for Approximating the Ideal Observer for Joint Signal Detection and Estimation Tasks by Use of Supervised Learning and Markov-Chain Monte Carlo Methods

The ideal observer (IO) sets an upper performance limit among all observers and has been advocated for assessing and optimizing imaging systems. For general joint detection and estimation (detection-estimation) tasks, estimation ROC (EROC) analysis has been established for evaluating the performance of observers. However, in general, it is difficult to accurately approximate the IO that maximizes the area under the EROC curve. In this study, a hybrid method that employs machine learning is proposed to accomplish this. Specifically, a hybrid approach is developed that combines a multi-task convolutional neural network and a Markov-Chain Monte Carlo (MCMC) method in order to approximate the IO for detection-estimation tasks. Unlike traditional MCMC methods, the hybrid method is not limited to use of specific utility functions. In addition, a purely supervised learning-based sub-ideal observer is proposed. Computer-simulation studies are conducted to validate the proposed method, which include signal-known-statistically/background-known-exactly and signal-known-statistically/background-known-statistically tasks. The EROC curves produced by the proposed method are compared to those produced by the MCMC approach or analytical computation when feasible. The proposed method provides a new approach for approximating the IO and may advance the application of EROC analysis for optimizing imaging systems.

Cell Cycle Stage Classification Using Phase Imaging with Computational Specificity

Traditional methods for cell cycle stage classification rely heavily on fluorescence microscopy to monitor nuclear dynamics. These methods inevitably face the typical phototoxicity and photobleaching limitations of fluorescence imaging. Here, we present a cell cycle detection workflow using the principle of phase imaging with computational specificity (PICS). The proposed method uses neural networks to extract cell cycle-dependent features from quantitative phase imaging (QPI) measurements directly. Our results indicate that this approach attains very good accuracy in classifying live cells into G1, S, and G2/M stages, respectively. We also demonstrate that the proposed method can be applied to study single-cell dynamics within the cell cycle as well as cell population distribution across different stages of the cell cycle. We envision that the proposed method can become a nondestructive tool to analyze cell cycle progression in fields ranging from cell biology to biopharma applications.

A Multivariate Functional Connectivity Approach to Mapping Brain Networks and Imputing Neural Activity in Mice

Temporal correlation analysis of spontaneous brain activity (e.g., Pearson "functional connectivity," FC) has provided insights into the functional organization of the human brain. However, bivariate analysis techniques such as this are often susceptible to confounding physiological processes (e.g., sleep, Mayer-waves, breathing, motion), which makes it difficult to accurately map connectivity in health and disease as these physiological processes affect FC. In contrast, a multivariate approach to imputing individual neural networks from spontaneous neuroimaging data could be influential to our conceptual understanding of FC and provide performance advantages. Therefore, we analyzed neural calcium imaging data from Thy1-GCaMP6f mice while either awake, asleep, anesthetized, during low and high bouts of motion, or before and after photothrombotic stroke. A linear support vector regression approach was used to determine the optimal weights for integrating the signals from the remaining pixels to accurately predict neural activity in a region of interest (ROI). The resultant weight maps for each ROI were interpreted as multivariate functional connectivity (MFC), resembled anatomical connectivity, and demonstrated a sparser set of strong focused positive connections than traditional FC. While global variations in data have large effects on standard correlation FC analysis, the MFC mapping methods were mostly impervious. Lastly, MFC analysis provided a more powerful connectivity deficit detection following stroke compared to traditional FC.

Normalization of optical fluence distribution for three-dimensional functional optoacoustic tomography of the breast

SIGNIFICANCE

In three-dimensional (3D) functional optoacoustic tomography (OAT), wavelength-dependent optical attenuation and nonuniform incident optical fluence limit imaging depth and field of view and can hinder accurate estimation of functional quantities, such as the vascular blood oxygenation. These limitations hinder OAT of large objects, such as a human female breast.

AIM

We aim to develop a measurement-data-driven method for normalization of the optical fluence distribution and to investigate blood vasculature detectability and accuracy for estimating vascular blood oxygenation.

APPROACH

The proposed method is based on reasonable assumptions regarding breast anatomy and optical properties. The nonuniform incident optical fluence is estimated based on the illumination geometry in the OAT system, and the depth-dependent optical attenuation is approximated using Beer-Lambert law.

RESULTS

Numerical studies demonstrated that the proposed method significantly enhanced blood vessel detectability and improved estimation accuracy of the vascular blood oxygenation from multiwavelength OAT measurements, compared with direct application of spectral linear unmixing without optical fluence compensation. Experimental results showed that the proposed method revealed previously invisible structures in regions deeper than 15 mm and/or near the chest wall.

CONCLUSIONS

The proposed method provides a straightforward and computationally inexpensive approximation of wavelength-dependent effective optical attenuation and, thus, enables mitigation of the spectral coloring effect in functional 3D OAT imaging.

Computing a projection operator onto the null space of a linear imaging operator: tutorial

Many imaging systems can be approximately described by a linear operator that maps an object property to a collection of discrete measurements. However, even in the absence of measurement noise, such operators are generally "blind" to certain components of the object, and hence information is lost in the imaging process. Mathematically, this is explained by the fact that the imaging operator can possess a null space. All objects in the null space, by definition, are mapped to a collection of identically zero measurements and are hence invisible to the imaging system. As such, characterizing the null space of an imaging operator is of fundamental importance when comparing and/or designing imaging systems. A characterization of the null space can also facilitate the design of regularization strategies for image reconstruction methods. Characterizing the null space via an associated projection operator is, in general, a computationally demanding task. In this tutorial, computational procedures for establishing projection operators that map an object to the null space of a discrete-to-discrete imaging operator are surveyed. A new machine-learning-based approach that employs a linear autoencoder is also presented. The procedures are demonstrated by use of biomedical imaging examples, and their computational complexities and memory requirements are compared.

Automated sleep state classification of wide-field calcium imaging data via multiplex visibility graphs and deep learning

BACKGROUND

Wide-field calcium imaging (WFCI) allows for monitoring of cortex-wide neural dynamics in mice. When applied to the study of sleep, WFCI data are manually scored into the sleep states of wakefulness, non-REM (NREM) and REM by use of adjunct EEG and EMG recordings. However, this process is time-consuming and often suffers from low inter- and intra-rater reliability and invasiveness. Therefore, an automated sleep state classification method that operates on WFCI data alone is needed.

NEW METHOD

A hybrid, two-step method is proposed. In the first step, spatial-temporal WFCI data is mapped to multiplex visibility graphs (MVGs). Subsequently, a two-dimensional convolutional neural network (2D CNN) is employed on the MVGs to be classified as wakefulness, NREM and REM.

RESULTS

Sleep states were classified with an accuracy of 84% and Cohen's κ of 0.67. The method was also effectively applied on a binary classification of wakefulness/sleep (accuracy=0.82, κ = 0.62) and a four-class wakefulness/sleep/anesthesia/movement classification (accuracy=0.74, κ = 0.66). Gradient-weighted class activation maps revealed that the CNN focused on short- and long-term temporal connections of MVGs in a sleep state-specific manner. Sleep state classification performance when using individual brain regions was highest for the posterior area of the cortex and when cortex-wide activity was considered.

COMPARISON WITH EXISTING METHOD

On a 3-hour WFCI recording, the MVG-CNN achieved a κ of 0.65, comparable to a κ of 0.60 corresponding to the human EEG/EMG-based scoring.

CONCLUSIONS

The hybrid MVG-CNN method accurately classifies sleep states from WFCI data and will enable future sleep-focused studies with WFCI.

Learning stochastic object models from medical imaging measurements by use of advanced ambient generative adversarial networks

Purpose: To objectively assess new medical imaging technologies via computer-simulations, it is important to account for the variability in the ensemble of objects to be imaged. This source of variability can be described by stochastic object models (SOMs). It is generally desirable to establish SOMs from experimental imaging measurements acquired by use of a well-characterized imaging system, but this task has remained challenging. Approach: A generative adversarial network (GAN)-based method that employs AmbientGANs with modern progressive or multiresolution training approaches is proposed. AmbientGANs established using the proposed training procedure are systematically validated in a controlled way using computer-simulated magnetic resonance imaging (MRI) data corresponding to a stylized imaging system. Emulated single-coil experimental MRI data are also employed to demonstrate the methods under less stylized conditions. Results: The proposed AmbientGAN method can generate clean images when the imaging measurements are contaminated by measurement noise. When the imaging measurement data are incomplete, the proposed AmbientGAN can reliably learn the distribution of the measurement components of the objects. Conclusions: Both visual examinations and quantitative analyses, including task-specific validations using the Hotelling observer, demonstrated that the proposed AmbientGAN method holds promise to establish realistic SOMs from imaging measurements.

3-D Stochastic Numerical Breast Phantoms for Enabling Virtual Imaging Trials of Ultrasound Computed Tomography

Ultrasound computed tomography (USCT) is an emerging imaging modality for breast imaging that can produce quantitative images that depict the acoustic properties of tissues. Computer-simulation studies, also known as virtual imaging trials, provide researchers with an economical and convenient route to systematically explore imaging system designs and image reconstruction methods. When simulating an imaging technology intended for clinical use, it is essential to employ realistic numerical phantoms that can facilitate the objective, or task-based, assessment of image quality (IQ). Moreover, when computing objective IQ measures, an ensemble of such phantoms should be employed, which displays the variability in anatomy and object properties that are representative of the to-be-imaged patient cohort. Such stochastic phantoms for clinically relevant applications of USCT are currently lacking. In this work, a methodology for producing realistic 3-D numerical breast phantoms for enabling clinically relevant computer-simulation studies of USCT breast imaging is presented. By extending and adapting an existing stochastic 3-D breast phantom for use with USCT, methods for creating ensembles of numerical acoustic breast phantoms are established. These breast phantoms will possess clinically relevant variations in breast size, composition, acoustic properties, tumor locations, and tissue textures. To demonstrate the use of the phantoms in virtual USCT studies, two brief case studies are presented, which addresses the development and assessment of image reconstruction procedures. Examples of breast phantoms produced by use of the proposed methods and a collection of 52 sets of simulated USCT measurement data have been made open source for use in image reconstruction development.

On Hallucinations in Tomographic Image Reconstruction

Tomographic image reconstruction is generally an ill-posed linear inverse problem. Such ill-posed inverse problems are typically regularized using prior knowledge of the sought-after object property. Recently, deep neural networks have been actively investigated for regularizing image reconstruction problems by learning a prior for the object properties from training images. However, an analysis of the prior information learned by these deep networks and their ability to generalize to data that may lie outside the training distribution is still being explored. An inaccurate prior might lead to false structures being hallucinated in the reconstructed image and that is a cause for serious concern in medical imaging. In this work, we propose to illustrate the effect of the prior imposed by a reconstruction method by decomposing the image estimate into generalized measurement and null components. The concept of a hallucination map is introduced for the general purpose of understanding the effect of the prior in regularized reconstruction methods. Numerical studies are conducted corresponding to a stylized tomographic imaging modality. The behavior of different reconstruction methods under the proposed formalism is discussed with the help of the numerical studies.

Impact of deep learning-based image super-resolution on binary signal detection

Purpose: Deep learning-based image super-resolution (DL-SR) has shown great promise in medical imaging applications. To date, most of the proposed methods for DL-SR have only been assessed using traditional measures of image quality (IQ) that are commonly employed in the field of computer vision. However, the impact of these methods on objective measures of IQ that are relevant to medical imaging tasks remains largely unexplored. We investigate the impact of DL-SR methods on binary signal detection performance. Approach: Two popular DL-SR methods, the super-resolution convolutional neural network and the super-resolution generative adversarial network, were trained using simulated medical image data. Binary signal-known-exactly with background-known-statistically and signal-known-statistically with background-known-statistically detection tasks were formulated. Numerical observers (NOs), which included a neural network-approximated ideal observer and common linear NOs, were employed to assess the impact of DL-SR on task performance. The impact of the complexity of the DL-SR network architectures on task performance was quantified. In addition, the utility of DL-SR for improving the task performance of suboptimal observers was investigated. Results: Our numerical experiments confirmed that, as expected, DL-SR improved traditional measures of IQ. However, for many of the study designs considered, the DL-SR methods provided little or no improvement in task performance and even degraded it. It was observed that DL-SR improved the task performance of suboptimal observers under certain conditions. Conclusions: Our study highlights the urgent need for the objective assessment of DL-SR methods and suggests avenues for improving their efficacy in medical imaging applications.

Model-based optical and acoustical compensation for photoacoustic tomography of heterogeneous mediums

Photoacoustic tomography (PAT) is a non-invasive, high-resolution imaging modality, capable of providing functional and molecular information of various pathologies, such as cancer. One limitation of PAT is the depth and wavelength dependent optical fluence, which results in reduced PA signal amplitude from deeper tissue regions. These factors can therefore introduce errors into quantitative measurements such as oxygen saturation (sO₂) or the localization and concentration of various chromophores. The variation in the speed-of-sound between different tissues can also lead to distortions in object location and shape. Compensating for these effects allows PAT to be used more quantitatively. We have developed a proof-of-concept algorithm capable of compensating for the heterogeneity in speed-of-sound and depth dependent optical fluence. Speed-of-sound correction was done by using a straight ray-based algorithm for calculating the family of iso-time-of-flight contours between the transducers and every pixel in the imaging grid, while fluence compensation was done by utilizing the graphics processing unit (GPU) accelerated software MCXCL for Monte Carlo modeling of optical fluence variation. This algorithm was tested on a polyvinyl chloride plastisol (PVCP) phantom, which contained cyst mimics and blood inclusions to test the algorithm under relatively heterogeneous conditions. Our results indicate that our PAT algorithm can compensate for the speed-of-sound variation and depth dependent fluence effects within a heterogeneous phantom. The results of this study will pave the way for further development and evaluation of the proposed method in more complex in-vitro and ex-vivo phantoms, as well as compensating for the wavelength-dependent optical fluence in spectroscopic PAT.

Assessing the Impact of Deep Neural Network-Based Image Denoising on Binary Signal Detection Tasks

A variety of deep neural network (DNN)-based image denoising methods have been proposed for use with medical images. Traditional measures of image quality (IQ) have been employed to optimize and evaluate these methods. However, the objective evaluation of IQ for the DNN-based denoising methods remains largely lacking. In this work, we evaluate the performance of DNN-based denoising methods by use of task-based IQ measures. Specifically, binary signal detection tasks under signal-known-exactly (SKE) with background-known-statistically (BKS) conditions are considered. The performance of the ideal observer (IO) and common linear numerical observers are quantified and detection efficiencies are computed to assess the impact of the denoising operation on task performance. The numerical results indicate that, in the cases considered, the application of a denoising network can result in a loss of task-relevant information in the image. The impact of the depth of the denoising networks on task performance is also assessed. The presented results highlight the need for the objective evaluation of IQ for DNN-based denoising technologies and may suggest future avenues for improving their effectiveness in medical imaging applications.

A signal detection model for quantifying overregularization in nonlinear image reconstruction

Many useful image quality metrics for evaluating linear image reconstruction techniques do not apply to or are difficult to interpret for nonlinear image reconstruction. The vast majority of metrics employed for evaluating nonlinear image reconstruction are based on some form of global image fidelity, such as image root mean square error (RMSE). Use of such metrics can lead to overregularization in the sense that they can favor removal of subtle details in the image. To address this shortcoming, we develop an image quality metric based on signal detection that serves as a surrogate to the qualitative loss of fine image details. The metric is demonstrated in the context of a breast CT simulation, where different equal-dose configurations are considered. The configurations differ in the number of projections acquired. Image reconstruction is performed with a nonlinear algorithm based on total variation constrained least-squares (TV-LSQ). The resulting images are studied as a function of three parameters: number of views acquired, total variation constraint value, and number of iterations. The images are evaluated visually, with image RMSE, and with the proposed signal-detection-based metric. The latter uses a small signal, and computes detectability in the sinogram and in the reconstructed image. Loss of signal detectability through the image reconstruction process is taken as a quantitative measure of loss of fine details in the image. Loss of signal detectability is seen to correlate well with the blocky or patchy appearance due to overregularization with TV-LSQ, and this trend runs counter to the image RMSE metric, which tends to favor the over-regularized images. The proposed signal detection-based metric provides an image quality assessment that is complimentary to that of image RMSE. Using the two metrics in concert may yield a useful prescription for determining CT algorithm and configuration parameters when nonlinear image reconstruction is used.

Special Series Guest Editorial: Artificial Intelligence and Machine Learning in Biomedical Optics

Guest editors Behrouz Shabestri, Mark Anastasio, Baowei Fei, and Frédéric Leblond provide an overview of the JBO Special Series on Artificial Intelligence Machine Learning in Biomedical Optics.

Deeply-supervised density regression for automatic cell counting in microscopy images

Accurately counting the number of cells in microscopy images is required in many medical diagnosis and biological studies. This task is tedious, time-consuming, and prone to subjective errors. However, designing automatic counting methods remains challenging due to low image contrast, complex background, large variance in cell shapes and counts, and significant cell occlusions in two-dimensional microscopy images. In this study, we proposed a new density regression-based method for automatically counting cells in microscopy images. The proposed method processes two innovations compared to other state-of-the-art density regression-based methods. First, the density regression model (DRM) is designed as a concatenated fully convolutional regression network (C-FCRN) to employ multi-scale image features for the estimation of cell density maps from given images. Second, auxiliary convolutional neural networks (AuxCNNs) are employed to assist in the training of intermediate layers of the designed C-FCRN to improve the DRM performance on unseen datasets. Experimental studies evaluated on four datasets demonstrate the superior performance of the proposed method.

3D microscopy and deep learning reveal the heterogeneity of crown-like structure microenvironments in intact adipose tissue

Crown-like structures (CLSs) are adipose microenvironments of macrophages engulfing adipocytes. Their histological density in visceral adipose tissue (VAT) predicts metabolic disorder progression in obesity and is believed to initiate obesity comorbidities. Here, we use three-dimensional (3D) light sheet microscopy and deep learning to quantify 3D features of VAT CLSs in lean and obese states. Obese CLS densities are significantly higher, composing 3.9% of tissue volume compared with 0.46% in lean tissue. Across the states, individual CLS structural characteristics span similar ranges; however, subpopulations are distinguishable. Obese VAT contains large CLSs absent from lean tissues, located near the tissue center, while lean CLSs have higher volumetric cell densities and prolate shapes. These features are consistent with inefficient adipocyte elimination in obesity that contributes to chronic inflammation, representing histological biomarkers to assess adipose pathogenesis. This tissue processing, imaging, and analysis pipeline can be applied to quantitatively classify 3D microenvironments across diverse tissues.

Compressible Latent-Space Invertible Networks for Generative Model-Constrained Image Reconstruction

There remains an important need for the development of image reconstruction methods that can produce diagnostically useful images from undersampled measurements. In magnetic resonance imaging (MRI), for example, such methods can facilitate reductions in data-acquisition times. Deep learning-based methods hold potential for learning object priors or constraints that can serve to mitigate the effects of data-incompleteness on image reconstruction. One line of emerging research involves formulating an optimization-based reconstruction method in the latent space of a generative deep neural network. However, when generative adversarial networks (GANs) are employed, such methods can result in image reconstruction errors if the sought-after solution does not reside within the range of the GAN. To circumvent this problem, in this work, a framework for reconstructing images from incomplete measurements is proposed that is formulated in the latent space of invertible neural network-based generative models. A novel regularization strategy is introduced that takes advantage of the multiscale architecture of certain invertible neural networks, which can result in improved reconstruction performance over classical methods in terms of traditional metrics. The proposed method is investigated for reconstructing images from undersampled MRI data. The method is shown to achieve comparable performance to a state-of-the-art generative model-based reconstruction method while benefiting from a deterministic reconstruction procedure and easier control over regularization parameters.

Approximating the Ideal Observer for Joint Signal Detection and Localization Tasks by use of Supervised Learning Methods

Medical imaging systems are commonly assessed and optimized by use of objective measures of image quality (IQ). The Ideal Observer (IO) performance has been advocated to provide a figure-of-merit for use in assessing and optimizing imaging systems because the IO sets an upper performance limit among all observers. When joint signal detection and localization tasks are considered, the IO that employs a modified generalized likelihood ratio test maximizes observer performance as characterized by the localization receiver operating characteristic (LROC) curve. Computations of likelihood ratios are analytically intractable in the majority of cases. Therefore, sampling-based methods that employ Markov-Chain Monte Carlo (MCMC) techniques have been developed to approximate the likelihood ratios. However, the applications of MCMC methods have been limited to relatively simple object models. Supervised learning-based methods that employ convolutional neural networks have been recently developed to approximate the IO for binary signal detection tasks. In this paper, the ability of supervised learning-based methods to approximate the IO for joint signal detection and localization tasks is explored. Both background-known-exactly and background-known-statistically signal detection and localization tasks are considered. The considered object models include a lumpy object model and a clustered lumpy model, and the considered measurement noise models include Laplacian noise, Gaussian noise, and mixed Poisson-Gaussian noise. The LROC curves produced by the supervised learning-based method are compared to those produced by the MCMC approach or analytical computation when feasible. The potential utility of the proposed method for computing objective measures of IQ for optimizing imaging system performance is explored.

High-resolution transcriptional and morphogenetic profiling of cells from micropatterned human ESC gastruloid cultures

During mammalian gastrulation, germ layers arise and are shaped into the body plan while extraembryonic layers sustain the embryo. Human embryonic stem cells, cultured with BMP4 on extracellular matrix micro-discs, reproducibly differentiate into gastruloids, expressing markers of germ layers and extraembryonic cells in radial arrangement. Using single-cell RNA sequencing and cross-species comparisons with mouse, cynomolgus monkey gastrulae, and post-implantation human embryos, we reveal that gastruloids contain cells transcriptionally similar to epiblast, ectoderm, mesoderm, endoderm, primordial germ cells, trophectoderm, and amnion. Upon gastruloid dissociation, single cells reseeded onto micro-discs were motile and aggregated with the same but segregated from distinct cell types. Ectodermal cells segregated from endodermal and extraembryonic but mixed with mesodermal cells. Our work demonstrates that the gastruloid system models primate-specific features of embryogenesis, and that gastruloid cells exhibit evolutionarily conserved sorting behaviors. This work generates a resource for transcriptomes of human extraembryonic and embryonic germ layers differentiated in a stereotyped arrangement.

High-resolution transcriptional and morphogenetic profiling of cells from micropatterned human ESC gastruloid cultures

During mammalian gastrulation, germ layers arise and are shaped into the body plan while extraembryonic layers sustain the embryo. Human embryonic stem cells, cultured with BMP4 on extracellular matrix micro-discs, reproducibly differentiate into gastruloids, expressing markers of germ layers and extraembryonic cells in radial arrangement. Using single-cell RNA sequencing and cross-species comparisons with mouse, cynomolgus monkey gastrulae, and post-implantation human embryos, we reveal that gastruloids contain cells transcriptionally similar to epiblast, ectoderm, mesoderm, endoderm, primordial germ cells, trophectoderm, and amnion. Upon gastruloid dissociation, single cells reseeded onto micro-discs were motile and aggregated with the same but segregated from distinct cell types. Ectodermal cells segregated from endodermal and extraembryonic but mixed with mesodermal cells. Our work demonstrates that the gastruloid system models primate-specific features of embryogenesis, and that gastruloid cells exhibit evolutionarily conserved sorting behaviors. This work generates a resource for transcriptomes of human extraembryonic and embryonic germ layers differentiated in a stereotyped arrangement.

Gold nanorods enable noninvasive longitudinal monitoring of hydrogels in vivo with photoacoustic tomography

Longitudinal in vivo monitoring is essential for the design and evaluation of biomaterials. An ideal method would provide three-dimensional quantitative information, high spatial resolution, deep tissue penetration, and contrast between tissue and material structures. Photoacoustic (PA) or optoacoustic imaging is a hybrid technique that allows three-dimensional imaging with high spatial resolution. In addition, photoacoustic imaging allows for imaging of vascularization based on the intrinsic contrast of hemoglobin. In this study, we investigated photoacoustic computed tomography (PACT) as a tool for longitudinal monitoring of an implanted hydrogel in a small animal model. Hydrogels were loaded with gold nanorods to enhance contrast and imaged weekly for 8 weeks. PACT allowed non-invasive three-dimensional, quantitative imaging of the hydrogels over the entire 8 weeks. Quantitative volume analysis was used to evaluate the in vivo degradation kinetics of the implants which deviated slightly from in vitro predictions. Multispectral imaging allowed for the simultaneous analysis of hydrogel degradation and local vascularization. These results provide support for the substantial potential of PACT as a tool for insight into biomaterial performance in vivo.

A partial-dithering strategy for edge-illumination x-ray phase-contrast tomography enabled by a joint reconstruction method

Edge-illumination x-ray phase-contrast tomography (EIXPCT) is a promising imaging technology where partially opaque masks are utilized with laboratory-based x-ray sources to estimate the distribution of the complex-valued refractive index. EIXPCT resolution is mainly determined by the period of a sample mask, but can be significantly improved by a dithering technique. Here, dithering means that multiple images per tomographic view angle are acquired as the object is moved over sub-pixel distances. Drawbacks of dithering include increased data-acquisition times and radiation doses. Motivated by the flexibility in data-acquisition designs enabled by a recently developed joint reconstruction method, a novel partial-dithering strategy for EIXPCT data-acquisition is proposed. In this strategy, dithering is implemented at only a subset of the tomographic view angles. The strategy can result in spatial resolution comparable to that of the conventional full-dithering strategy, where dithering is performed at every view angle, but the acquisition time is substantially decreased. Here, the effect of dithering parameters on image resolution is explored.

Iterative image reconstruction in transcranial photoacoustic tomography based on the elastic wave equation

Photoacoustic computed tomography (PACT) is an emerging computed imaging modality that exploits optical contrast and ultrasonic detection principles to form images of the photoacoustically induced initial pressure distribution within tissue. The PACT reconstruction problem corresponds to a time-domain inverse source problem, where the initial pressure distribution is recovered from the measurements recorded on an aperture outside the support of the source. A major challenge in transcranial PACT of the brain is to compensate for aberrations and attenuation in the measured data due to the propagation of the photoacoustic wavefields through the skull. To properly account for these effects, a wave equation-based inversion method can be employed that can model the heterogeneous elastic properties of the medium. In this study, an optimization-based image reconstruction method for 3D transcranial PACT is developed based on the elastic wave equation. To accomplish this, a forward-adjoint operator pair based on a finite-difference time-domain discretization of the 3D elastic wave equation is utilized to compute penalized least squares estimates of the initial pressure distribution. Computer-simulation and experimental studies are conducted to investigate the robustness of the reconstruction method to model mismatch and its ability to effectively resolve cortical and superficial brain structures.

Photoacoustic Imaging in Tissue Engineering and Regenerative Medicine

Several imaging modalities are available for investigation of the morphological, functional, and molecular features of engineered tissues in small animal models. While research in tissue engineering and regenerative medicine (TERM) would benefit from a comprehensive longitudinal analysis of new strategies, researchers have not always applied the most advanced methods. Photoacoustic imaging (PAI) is a rapidly emerging modality that has received significant attention due to its ability to exploit the strong endogenous contrast of optical methods with the high spatial resolution of ultrasound methods. Exogenous contrast agents can also be used in PAI for targeted imaging. Applications of PAI relevant to TERM include stem cell tracking, longitudinal monitoring of scaffolds in vivo, and evaluation of vascularization. In addition, the emerging capabilities of PAI applied to the detection and monitoring of cancer and other inflammatory diseases could be exploited by tissue engineers. This article provides an overview of the operating principles of PAI and its broad potential for application in TERM. Impact statement Photoacoustic imaging, a new hybrid imaging technique, has demonstrated high potential in the clinical diagnostic applications. The optical and acoustic aspect of the photoacoustic imaging system works in harmony to provide better resolution at greater tissue depth. Label-free imaging of vasculature with this imaging can be used to track and monitor disease, as well as the therapeutic progression of treatment. Photoacoustic imaging has been utilized in tissue engineering to some extent; however, the full benefit of this technique is yet to be explored. The increasing availability of commercial photoacoustic systems will make application as an imaging tool for tissue engineering application more feasible. This review first provides a brief description of photoacoustic imaging and summarizes its current and potential application in tissue engineering.

Comparison of data-acquisition designs for single-shot edge-illumination X-ray phase-contrast tomography

Edge-illumination X-ray phase-contrast tomography (EIXPCT) is an emerging technique that enables practical phase-contrast imaging with laboratory-based X-ray sources. A joint reconstruction method was proposed for reconstructing EIXPCT images, enabling novel flexible data-acquisition designs. However, only limited efforts have been devoted to optimizing data-acquisition designs for use with the joint reconstruction method. In this study, several promising designs are introduced, such as the constant aperture position (CAP) strategy and the alternating aperture position (AAP) strategy covering different angular ranges. In computer-simulation studies, these designs are analyzed and compared. Experimental data are employed to test the designs in real-world applications. All candidate designs are also compared for their implementation complexity. The tradeoff between data-acquisition time and image quality is discussed.

Approximating the Ideal Observer and Hotelling Observer for Binary Signal Detection Tasks by Use of Supervised Learning Methods

It is widely accepted that the optimization of medical imaging system performance should be guided by task-based measures of image quality (IQ). Task-based measures of IQ quantify the ability of an observer to perform a specific task, such as detection or estimation of a signal (e.g., a tumor). For binary signal detection tasks, the Bayesian Ideal Observer (IO) sets an upper limit of observer performance and has been advocated for use in optimizing medical imaging systems and data-acquisition designs. Except in special cases, the determination of the IO test statistic is analytically intractable. Markov-chain Monte Carlo (MCMC) techniques can be employed to approximate the IO detection performance, but their reported applications have been limited to relatively simple object models. In cases where the IO test statistic is difficult to compute, the Hotelling Observer (HO) can be employed. To compute the HO test statistic, potentially large covariance matrices must be accurately estimated and subsequently inverted, which can present computational challenges. This paper investigates the supervised learning-based methodologies for approximating the IO and HO test statistics. Convolutional neural networks (CNNs) and single-layer neural networks (SLNNs) are employed to approximate the IO and HO test statistics, respectively. The numerical simulations were conducted for both signal-known-exactly (SKE) and signal-known-statistically (SKS) signal detection tasks. The considered background models include the lumpy object model and the clustered lumpy object model. The measurement noise models considered are Gaussian, Laplacian, and mixed Poisson-Gaussian. The performances of the supervised learning methods are assessed via receiver operating characteristic (ROC) analysis, and the results are compared to those produced by the use of traditional numerical methods or analytical calculations when feasible. The potential advantages of the proposed supervised learning approaches for approximating the IO and HO test statistics are discussed.

Quantification of image texture in X-ray phase-contrast-enhanced projection images of in vivo mouse lungs observed at varied inflation pressures

To date, there are very limited noninvasive, regional assays of in vivo lung microstructure near the alveolar level. It has been suggested that x-ray phase-contrast enhanced imaging reveals information about the air volume of the lung; however, the image texture information in these images remains underutilized. Projection images of in vivo mouse lungs were acquired via a tabletop, propagation-based, X-ray phase-contrast imaging system. Anesthetized mice were mechanically ventilated in an upright position. Consistent with previously published studies, a distinct image texture was observed uniquely within lung regions. Lung regions were automatically identified using supervised machine learning applied to summary measures of the image texture data. It was found that an unsupervised clustering within predefined lung regions colocates with expected differences in anatomy along the cranial-caudal axis in upright mice. It was also found that specifically selected inflation pressures-here, a purposeful surrogate of distinct states of mechanical expansion-can be predicted from the lung image texture alone, that the prediction model itself varies from apex to base and that prediction is accurate regardless of overlap with nonpulmonary structures such as the ribs, mediastinum, and heart. Cross-validation analysis indicated low inter-animal variation in the image texture classifications. Together, these results suggest that the image texture observed in a single X-ray phase-contrast-enhanced projection image could be used across a range of pressure states to study regional variations in regional lung function.

A survey of computational frameworks for solving the acoustic inverse problem in three-dimensional photoacoustic computed tomography

Photoacoustic computed tomography (PACT), also known as optoacoustic tomography, is an emerging imaging technique that holds great promise for biomedical imaging. PACT is a hybrid imaging method that can exploit the strong endogenous contrast of optical methods along with the high spatial resolution of ultrasound methods. In its canonical form that is addressed in this article, PACT seeks to estimate the photoacoustically-induced initial pressure distribution within the object. Image reconstruction methods are employed to solve the acoustic inverse problem associated with the image formation process. When an idealized imaging scenario is considered, analytic solutions to the PACT inverse problem are available; however, in practice, numerous challenges exist that are more readily addressed within an optimization-based, or iterative, image reconstruction framework. In this article, the PACT image reconstruction problem is reviewed within the context of modern optimization-based image reconstruction methodologies. Imaging models that relate the measured photoacoustic wavefields to the sought-after object function are described in their continuous and discrete forms. The basic principles of optimization-based image reconstruction from discrete PACT measurement data are presented, which includes a review of methods for modeling the PACT measurement system response and other important physical factors. Non-conventional formulations of the PACT image reconstruction problem, in which acoustic parameters of the medium are concurrently estimated along with the PACT image, are also introduced and reviewed.

Task-based image quality assessment in radiation therapy: initial characterization and demonstration with computer-simulation study

In the majority of current radiation therapy (RT) applications, image quality is still assessed subjectively or by utilizing physical measures. A novel theory that applies objective task-based image quality assessment in radiation therapy (IQA-in-RT) was recently proposed, in which the area under the therapeutic operating characteristic curve (AUTOC) was employed as the figure-of-merit (FOM) for evaluating RT effectiveness. Although theoretically more appealing than conventional subjective or physical measures, a comprehensive implementation and evaluation of this novel task-based IQA-in-RT theory is required for its further application in improving clinical RT. In this work, a practical and modular IQA-in-RT framework is presented for implementing this theory for the assessment of imaging components on the basis of RT treatment outcomes. Computer-simulation studies are conducted to demonstrate the feasibility and utility of the proposed IQA-in-RT framework in optimizing x-ray computed tomography (CT) pre-treatment imaging, including the optimization of CT imaging dose and image reconstruction parameters. The potential advantages of optimizing imaging components in the RT workflow by use of the AUTOC as the FOM are also compared against those of other physical measures. The results demonstrate that optimization using the AUTOC leads to selecting different parameters from those indicated by physical measures, potentially improving RT performance. The sources of systemic randomness and bias that affect the determination of the AUTOC are also analyzed. The presented work provides a practical solution for the further investigation and analysis of the task-based IQA-in-RT theory and advances its applications in improving RT clinical practice and cancer patient care.