PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning

AI-driven 3D bioprinting for regenerative medicine: From bench to bedside

In recent decades, 3D bioprinting has garnered significant research attention due to its ability to manipulate biomaterials and cells to create complex structures precisely. However, due to technological and cost constraints, the clinical translation of 3D bioprinted products (BPPs) from bench to bedside has been hindered by challenges in terms of personalization of design and scaling up of production. Recently, the emerging applications of artificial intelligence (AI) technologies have significantly improved the performance of 3D bioprinting. However, the existing literature remains deficient in a methodological exploration of AI technologies' potential to overcome these challenges in advancing 3D bioprinting toward clinical application. This paper aims to present a systematic methodology for AI-driven 3D bioprinting, structured within the theoretical framework of Quality by Design (QbD). This paper commences by introducing the QbD theory into 3D bioprinting, followed by summarizing the technology roadmap of AI integration in 3D bioprinting, including multi-scale and multi-modal sensing, data-driven design, and in-line process control. This paper further describes specific AI applications in 3D bioprinting's key elements, including bioink formulation, model structure, printing process, and function regulation. Finally, the paper discusses current prospects and challenges associated with AI technologies to further advance the clinical translation of 3D bioprinting.

Current Advancements in Digital Neuropathology and Machine Learning for the Study of Neurodegenerative Diseases

Computational neurodegenerative neuropathology represents a transformative approach in the analysis and understanding of neurodegenerative diseases through the utilization of whole slide images (WSI) and advanced machine learning/artificial intelligence (ML/AI) techniques. This review explores the emerging field of computational neurodegenerative neuropathology, emphasizing its potential to enhance neuropathological assessment, diagnosis, and research. Recent advancements in ML/AI technologies have significantly impacted image-based medical fields, including anatomic pathology, by automating disease staging, identifying novel morphological biomarkers, and uncovering new clinical insights via multi-modal AI approaches. Despite its promise, the field faces several challenges, including limited expert annotations, slide scanning inaccessibility, inter-institutional variability, and the complexities of sharing large WSI datasets. This review discusses the importance of improving deep learning model accuracy and efficiency for better interpretation of neuropathological data. It highlights the potential of unsupervised learning to identify patterns in unannotated data. Furthermore, the development of explainable AI models is crucial for experimental neuropathology. Through addressing these challenges and leveraging cutting-edge AI techniques, computational neurodegenerative neuropathology has the potential to revolutionize the field and significantly advance our understanding of disease.

Acetic acid enabled nuclear contrast enhancement in epi-mode quantitative phase imaging

Significance

The acetowhitening effect of acetic acid (AA) enhances light scattering of cell nuclei, an effect that has been widely leveraged to facilitate tissue inspection for (pre)cancerous lesions. Here, we show that a concomitant effect of acetowhitening-changes in refractive index composition-yields nuclear contrast enhancement in quantitative phase imaging (QPI) of thick tissue samples.

Aim

We aim to explore how changes in refractive index composition during acetowhitening can be captured through a novel epi-mode 3D QPI technique called quantitative oblique back-illumination microscopy (qOBM). We also aim to demonstrate the potential of using a machine learning-based approach to convert qOBM images of fresh tissues into virtually AA-stained images.

Approach

We implemented qOBM, an imaging technique that allows for epi-mode 3D QPI to observe phase changes induced by AA in thick tissue samples. We focus on detecting nuclear contrast changes caused by AA in mouse brain samples. As a proof of concept, we also applied a Cycle-GAN algorithm to convert the acquired qOBM images into virtually AA-stained images, simulating the effect of AA staining.

Results

Our findings demonstrate that AA-induced acetowhitening leads to significant nuclear contrast enhancement in qOBM images of thick tissue samples. In addition, the Cycle-GAN algorithm successfully converted qOBM images into virtually AA-stained images, further facilitating the nuclear enhancement process without any physical stains.

Conclusions

We show that the acetowhitening effect of acetic acid induces changes in refractive index composition that significantly enhance nuclear contrast in QPI. The application of qOBM with AA, along with the use of a Cycle-GAN algorithm to virtually stain tissues, highlights the potential of this approach for advancing label-free and slide-free, ex vivo, and in vivo histology.

A novel approach in cancer diagnosis: integrating holography microscopic medical imaging and deep learning techniques-challenges and future trends

Medical imaging is pivotal in early disease diagnosis, providing essential insights that enable timely and accurate detection of health anomalies. Traditional imaging techniques, such as Magnetic Resonance Imaging (MRI), Computer Tomography (CT), ultrasound, and Positron Emission Tomography (PET), offer vital insights into three-dimensional structures but frequently fall short of delivering a comprehensive and detailed anatomical analysis, capturing only amplitude details. Three-dimensional holography microscopic medical imaging provides a promising solution by capturing the amplitude (brightness) and phase (structural information) details of biological structures. In this study, we investigate the novel collaborative potential of Deep Learning (DL) and holography microscopic phase imaging for cancer diagnosis. The study comprehensively examines existing literature, analyzes advancements, identifies research gaps, and proposes future research directions in cancer diagnosis through the integrated Quantitative Phase Imaging (QPI) and DL methodology. This novel approach addresses a critical limitation of traditional imaging by capturing detailed structural information, paving the way for more accurate diagnostics. The proposed approach comprises tissue sample collection, holographic image scanning, preprocessing in case of imbalanced datasets, and training on annotated datasets using DL architectures like U-Net and Vision Transformer(ViT's). Furthermore, sophisticated concepts in DL, like the incorporation of Explainable AI (XAI) techniques, are suggested for comprehensive disease diagnosis and identification. The study thoroughly investigates the advantages of integrating holography imaging and DL for precise cancer diagnosis. Additionally, meticulous insights are presented by identifying the challenges associated with this integration methodology.

Cell-APP: A generalizable method for microscopic cell annotation, segmentation, and classification

High throughput fluorescence microscopy is an essential tool in systems biological studies of eukaryotic cells. Its power can be fully realized when all cells in a field of view and the entire time series can be accurately localized and quantified. These tasks can be mapped to the common paradigm in computer vision: instance segmentation. Recently, supervised deep learning-based methods have become state-of-the-art for cellular instance segmentation. However, these methods require large amounts of high-quality training data. This requirement challenges our ability to train increasingly performant object detectors due to the limited availability of annotated training data, which is typically assembled via laborious hand annotation. Here, we present a generalizable method for generating large instance segmentation training datasets for tissue-culture cells in transmitted light microscopy images. We use datasets created by this method to train vision transformer (ViT) based Mask-RCNNs (Region-based Convolutional Neural Networks) that produce instance segmentations wherein cells are classified as "m-phase" (dividing) or "interphase" (non-dividing). While training these models, we also address the dataset class imbalance between m-phase and interphase cell annotations, which arises for biological reasons, using probabilistically weighted loss functions and partisan training data collection methods. We demonstrate the validity of these approaches by producing highly accurate object detectors that can serve as general tools for the segmentation and classification of morphologically diverse cells. Since the methodology depends only on generic cellular features, we hypothesize that it can be further generalized to most adherent tissue culture cell lines.

Enhanced multiscale human brain imaging by semi-supervised digital staining and serial sectioning optical coherence tomography

A major challenge in neuroscience is visualizing the structure of the human brain at different scales. Traditional histology reveals micro- and meso-scale brain features but suffers from staining variability, tissue damage, and distortion, which impedes accurate 3D reconstructions. The emerging label-free serial sectioning optical coherence tomography (S-OCT) technique offers uniform 3D imaging capability across samples but has poor histological interpretability despite its sensitivity to cortical features. Here, we present a novel 3D imaging framework that combines S-OCT with a deep-learning digital staining (DS) model. This enhanced imaging modality integrates high-throughput 3D imaging, low sample variability and high interpretability, making it suitable for 3D histology studies. We develop a novel semi-supervised learning technique to facilitate DS model training on weakly paired images for translating S-OCT to Gallyas silver staining. We demonstrate DS on various human cerebral cortex samples, achieving consistent staining quality and enhancing contrast across cortical layer boundaries. Additionally, we show that DS preserves geometry in 3D on cubic-centimeter tissue blocks, allowing for visualization of meso-scale vessel networks in the white matter. We believe that our technique has the potential for high-throughput, multiscale imaging of brain tissues and may facilitate studies of brain structures.

Impact of stain variation and color normalization for prognostic predictions in pathology

In recent years, deep neural networks (DNNs) have demonstrated remarkable performance in pathology applications, potentially even outperforming expert pathologists due to their ability to learn subtle features from large datasets. One complication in preparing digital pathology datasets for DNN tasks is the variation in tinctorial qualities. A common way to address this is to perform stain normalization on the images. In this study, we show that a well-trained DNN model trained on one batch of histological slides failed to generalize to another batch prepared at a different time from the same tissue blocks, even when stain normalization methods were applied. This study used sample data from a previously reported DNN that was able to identify patients with early-stage non-small cell lung cancer (NSCLC) whose tumors did and did not metastasize, with high accuracy, based on training and then testing of digital images from H&E stained primary tumor tissue sections processed at the same time. In this study, we obtained a new series of histologic slides from the adjacent recuts of the same tissue blocks processed in the same lab but at a different time. We found that the DNN trained on either batch of slides/images was unable to generalize and failed to predict progression in the other batch of slides/images (AUCcross-batch = 0.52 - 0.53 compared to AUCsame-batch = 0.74 - 0.81). The failure to generalize did not improve even when the tinctorial difference corrections were made through either traditional color-tuning or stain normalization with the help of a Cycle Generative Adversarial Network (CycleGAN) process. This highlights the need to develop an entirely new way to process and collect consistent microscopy images from histologic slides that can be used to both train and allow for the general application of predictive DNN algorithms.

Virtual Gram staining of label-free bacteria using dark-field microscopy and deep learning

Gram staining has been a frequently used staining protocol in microbiology. It is vulnerable to staining artifacts due to, e.g., operator errors and chemical variations. Here, we introduce virtual Gram staining of label-free bacteria using a trained neural network that digitally transforms dark-field images of unstained bacteria into their Gram-stained equivalents matching bright-field image contrast. After a one-time training, the virtual Gram staining model processes an axial stack of dark-field microscopy images of label-free bacteria (never seen before) to rapidly generate Gram staining, bypassing several chemical steps involved in the conventional staining process. We demonstrated the success of virtual Gram staining on label-free bacteria samples containing Escherichia coli and Listeria innocua by quantifying the staining accuracy of the model and comparing the chromatic and morphological features of the virtually stained bacteria against their chemically stained counterparts. This virtual bacterial staining framework bypasses the traditional Gram staining protocol and its challenges, including stain standardization, operator errors, and sensitivity to chemical variations.

Deep learning-enabled filter-free fluorescence microscope

Optical filtering is an indispensable part of fluorescence microscopy for selectively highlighting molecules labeled with a specific fluorophore and suppressing background noise. However, the utilization of optical filtering sets increases the complexity, size, and cost of microscopic systems, making them less suitable for multifluorescence channel, high-speed imaging. Here, we present filter-free fluorescence microscopic imaging enabled with deep learning-based digital spectral filtering. This approach allows for automatic fluorescence channel selection after image acquisition and accurate prediction of fluorescence by computing color changes due to spectral shifts with the presence of excitation scattering. Fluorescence prediction for cells and tissues labeled with various fluorophores was demonstrated under different magnification powers. The technique offers accurate identification of labeling with robust sensitivity and specificity, achieving consistent results with the reference standard. Beyond fluorescence microscopy, the deep learning-enabled spectral filtering strategy has the potential to drive the development of other biomedical applications, including cytometry and endoscopy.

Low-cost and label-free blue light cystoscopy through digital staining of white light cystoscopy videos

BACKGROUND

Bladder cancer is the 10th most common malignancy and carries the highest treatment cost among all cancers. The elevated cost stems from its high recurrence rate, which necessitates frequent surveillance. White light cystoscopy (WLC), the standard of care surveillance tool to examine the bladder for lesions, has limited sensitivity for early-stage bladder cancer. Blue light cystoscopy (BLC) utilizes a fluorescent dye to induce contrast in cancerous regions, improving the sensitivity of detection by 43%. Nevertheless, the added equipment cost and lengthy dwell time of the dye limits the availability of BLC.

METHODS

Here, we report the first demonstration of digital staining as a promising strategy to convert WLC images collected with standard-of-care clinical equipment into accurate BLC-like images, providing enhanced sensitivity for WLC without the associated labor or equipment cost.

RESULTS

By introducing key pre-processing steps to circumvent color and brightness variations in clinical datasets needed for successful model performance, the results achieve a staining accuracy of 80.58% and show excellent qualitative and quantitative agreement of the digitally stained WLC (dsWLC) images with ground truth BLC images, including color consistency.

CONCLUSIONS

In short, dsWLC can affordably provide the fluorescent contrast needed to improve the detection sensitivity of bladder cancer, thereby increasing the accessibility of BLC contrast for bladder cancer surveillance. The broader implications of this work suggest digital staining is a cost-effective alternative to contrast-based endoscopy for other clinical scenarios outside of urology that can democratize access to better healthcare.

Single color digital H&E staining with In-and-Out Net

Digital staining streamlines traditional staining procedures by digitally generating stained images from unstained or differently stained images. While conventional staining methods involve time-consuming chemical processes, digital staining offers an efficient and low-infrastructure alternative. Researchers can expedite tissue analysis without physical sectioning by leveraging microscopy-based techniques, such as confocal microscopy. However, interpreting grayscale or pseudo-color microscopic images remains challenging for pathologists and surgeons accustomed to traditional histologically stained images. To fill this gap, various studies explore digitally simulating staining to mimic targeted histological stains. This paper introduces a novel network, In-and-Out Net, designed explicitly for digital staining tasks. Based on Generative Adversarial Networks (GAN), our model efficiently transforms Reflectance Confocal Microscopy (RCM) images into Hematoxylin and Eosin (H&E) stained images. Using aluminum chloride preprocessing for skin tissue, we enhance nuclei contrast in RCM images. We trained the model with digital H&E labels featuring two fluorescence channels, eliminating the need for image registration and providing pixel-level ground truth. Our contributions include proposing an optimal training strategy, conducting a comparative analysis demonstrating state-of-the-art performance, validating the model through an ablation study, and collecting perfectly matched input and ground truth images without registration. In-and-Out Net showcases promising results, offering a valuable tool for digital staining tasks and advancing the field of histological image analysis.

Deciphering Structural Complexity of Brain, Joint, and Muscle Tissues Using Fourier Ptychographic Scattered Light Microscopy

Fourier Ptychographic Microscopy (FPM) provides high-resolution imaging and morphological information over large fields of view, while Computational Scattered Light Imaging (ComSLI) excels at mapping interwoven fiber organization in unstained tissue sections. This study introduces Fourier Ptychographic Scattered Light Microscopy (FP-SLM), a new multi-modal approach that combines FPM and ComSLI analyses to create both high-resolution phase-contrast images and fiber orientation maps from a single measurement. The method is demonstrated on brain sections (frog, monkey) and sections from thigh muscle and knee (mouse). FP-SLM delivers high-resolution images while revealing fiber organization in nerve, muscle, tendon, cartilage, and bone tissues. The approach is validated by comparing the computed fiber orientations with those derived from structure tensor analysis of the high-resolution images. The comparison shows that FPM and ComSLI are compatible with each other and yield fully consistent results. Remarkably, this combination surpasses the sum of its parts, so that applying ComSLI analysis to FPM recordings and vice-versa outperforms both methods alone. This cross-analysis approach can be retrospectively applied to analyze any existing FPM or ComSLI dataset (acquired with LED array and low numerical aperture), significantly expanding the application range of both techniques and enhancing the study of complex tissue architectures in biomedical research.

Quadri-wave lateral shearing interferometry: a versatile tool for quantitative phase imaging

Quantitative phase imaging (QPI) has emerged as a powerful tool in label-free bioimaging, in situ microstructure characterization for advanced manufacturing, and high-speed imaging of material property changes. Among various QPI methods, quadri-wave lateral shearing interferometry (QWLSI) stands out for its unique advantages in compactness, robustness, and high temporal resolution, making it an ideal choice for a wide range of applications. The compact design of QWLSI allows for easy integration with existing microscopy systems, while its robustness is manifested in the ability to maintain precise interferometric sensitivity even in high-vibration environments. Moreover, QWLSI also enables single-shot measurements that facilitate the capture of fast dynamic processes. This paper provides an in-depth exploration of the technical aspects of QWLSI, focusing on the evolution of its optical system and the primary algorithms used in wavefront reconstruction. The review also showcases significant applications of QWLSI, with a particular emphasis on its contributions to biomedical imaging. By discussing the advantages, limitations, and potential future developments of QWLSI, this paper aims to provide a comprehensive overview of this powerful QPI technique and its impact on various research fields.

Kidney medicine meets computer vision: a bibliometric analysis

BACKGROUND AND OBJECTIVE

Rapid advances in computer vision (CV) have the potential to facilitate the examination, diagnosis, and treatment of diseases of the kidney. The bibliometric study aims to explore the research landscape and evolving research focus of the application of CV in kidney medicine research.

METHODS

The Web of Science Core Collection was utilized to identify publications related to the research or applications of CV technology in the field of kidney medicine from January 1, 1900, to December 31, 2022. We analyzed emerging research trends, highly influential publications and journals, prolific researchers, countries/regions, research institutions, co-authorship networks, and co-occurrence networks. Bibliographic information was analyzed and visualized using Python, Matplotlib, Seaborn, HistCite, and Vosviewer.

RESULTS

There was an increasing trend in the number of publications on CV-based kidney medicine research. These publications mainly focused on medical image processing, surgical procedures, medical image analysis/diagnosis, as well as the application and innovation of CV technology in medical imaging. The United States is currently the leading country in terms of the quantities of published articles and international collaborations, followed by China. Deep learning-based segmentation and machine learning-based texture analysis are the most commonly used techniques in this field. Regarding research hotspot trends, CV algorithms are shifting toward artificial intelligence, and research objects are expanding to encompass a wider range of kidney-related objects, with data dimensions used in research transitioning from 2D to 3D while simultaneously incorporating more diverse data modalities.

CONCLUSION

The present study provides a scientometric overview of the current progress in the research and application of CV technology in kidney medicine research. Through the use of bibliometric analysis and network visualization, we elucidate emerging trends, key sources, leading institutions, and popular topics. Our findings and analysis are expected to provide valuable insights for future research on the use of CV in kidney medicine research.

Code-Free Machine Learning Solutions for Microscopy Image Processing: Deep Learning

In recent years, there has been a significant expansion in the realm of processing microscopy images, thanks to the advent of machine learning techniques. These techniques offer diverse applications for image processing. Currently, numerous methods are used for processing microscopy images in the field of biology, ranging from conventional machine learning algorithms to sophisticated deep learning artificial neural networks with millions of parameters. However, a comprehensive grasp of the intricacies of these methods usually necessitates proficiency in programming and advanced mathematics. In our comprehensive review, we explore various widely used deep learning approaches tailored for the processing of microscopy images. Our emphasis is on algorithms that have gained popularity in the field of biology and have been adapted to cater to users lacking programming expertise. In essence, our target audience comprises biologists interested in exploring the potential of deep learning algorithms, even without programming skills. Throughout the review, we elucidate each algorithm's fundamental concepts and capabilities without delving into mathematical and programming complexities. Crucially, all the highlighted algorithms are accessible on open platforms without requiring code, and we provide detailed descriptions and links within our review. It's essential to recognize that addressing each specific problem demands an individualized approach. Consequently, our focus is not on comparing algorithms but on delineating the problems they are adept at solving. In practical scenarios, researchers typically select multiple algorithms suited to their tasks and experimentally determine the most effective one. It is worth noting that microscopy extends beyond the realm of biology; its applications span diverse fields such as geology and material science. Although our review predominantly centers on biomedical applications, the algorithms and principles outlined here are equally applicable to other scientific domains. Furthermore, a number of the proposed solutions can be modified for use in entirely distinct computer vision cases.

Impact of Stain Variation and Color Normalization for Prognostic Predictions in Pathology

In recent years, deep neural networks (DNNs) have demonstrated remarkable performance in pathology applications, potentially even outperforming expert pathologists due to their ability to learn subtle features from large datasets. One complication in preparing digital pathology datasets for DNN tasks is variation in tinctorial qualities. A common way to address this is to perform stain normalization on the images. In this study, we show that a well-trained DNN model trained on one batch of histological slides failed to generalize to another batch prepared at a different time from the same tissue blocks, even when stain normalization methods were applied. This study used sample data from a previously reported DNN that was able to identify patients with early stage non-small cell lung cancer (NSCLC) whose tumors did and did not metastasize, with high accuracy, based on training and then testing of digital images from H&E stained primary tumor tissue sections processed at the same time. In this study we obtained a new series of histologic slides from the adjacent recuts of same tissue blocks processed in the same lab but at a different time. We found that the DNN trained on the either batch of slides/images was unable to generalize and failed to predict progression in the other batch of slides/images (AUCcross-batch = 0.52 - 0.53 compared to AUCsame-batch = 0.74 - 0.81). The failure to generalize did not improve even when the tinctorial difference correction were made through either traditional color-tuning or stain normalization with the help of a Cycle Generative Adversarial Network (CycleGAN) process. This highlights the need to develop an entirely new way to process and collect consistent microscopy images from histologic slides that can be used to both train and allow for the general application of predictive DNN algorithms.

Virtual birefringence imaging and histological staining of amyloid deposits in label-free tissue using autofluorescence microscopy and deep learning

Systemic amyloidosis involves the deposition of misfolded proteins in organs/tissues, leading to progressive organ dysfunction and failure. Congo red is the gold-standard chemical stain for visualizing amyloid deposits in tissue, showing birefringence under polarization microscopy. However, Congo red staining is tedious and costly to perform, and prone to false diagnoses due to variations in amyloid amount, staining quality and manual examination of tissue under a polarization microscope. We report virtual birefringence imaging and virtual Congo red staining of label-free human tissue to show that a single neural network can transform autofluorescence images of label-free tissue into brightfield and polarized microscopy images, matching their histochemically stained versions. Blind testing with quantitative metrics and pathologist evaluations on cardiac tissue showed that our virtually stained polarization and brightfield images highlight amyloid patterns in a consistent manner, mitigating challenges due to variations in chemical staining quality and manual imaging processes in the clinical workflow.

Accelerating histopathology workflows with generative AI-based virtually multiplexed tumour profiling

Understanding the spatial heterogeneity of tumours and its links to disease initiation and progression is a cornerstone of cancer biology. Presently, histopathology workflows heavily rely on hematoxylin and eosin and serial immunohistochemistry staining, a cumbersome, tissue-exhaustive process that results in non-aligned tissue images. We propose the VirtualMultiplexer, a generative artificial intelligence toolkit that effectively synthesizes multiplexed immunohistochemistry images for several antibody markers (namely AR, NKX3.1, CD44, CD146, p53 and ERG) from only an input hematoxylin and eosin image. The VirtualMultiplexer captures biologically relevant staining patterns across tissue scales without requiring consecutive tissue sections, image registration or extensive expert annotations. Thorough qualitative and quantitative assessment indicates that the VirtualMultiplexer achieves rapid, robust and precise generation of virtually multiplexed imaging datasets of high staining quality that are indistinguishable from the real ones. The VirtualMultiplexer is successfully transferred across tissue scales and patient cohorts with no need for model fine-tuning. Crucially, the virtually multiplexed images enabled training a graph transformer that simultaneously learns from the joint spatial distribution of several proteins to predict clinically relevant endpoints. We observe that this multiplexed learning scheme was able to greatly improve clinical prediction, as corroborated across several downstream tasks, independent patient cohorts and cancer types. Our results showcase the clinical relevance of artificial intelligence-assisted multiplexed tumour imaging, accelerating histopathology workflows and cancer biology.

Virtual staining for histology by deep learning

In pathology and biomedical research, histology is the cornerstone method for tissue analysis. Currently, the histological workflow consumes plenty of chemicals, water, and time for staining procedures. Deep learning is now enabling digital replacement of parts of the histological staining procedure. In virtual staining, histological stains are created by training neural networks to produce stained images from an unstained tissue image, or through transferring information from one stain to another. These technical innovations provide more sustainable, rapid, and cost-effective alternatives to traditional histological pipelines, but their development is in an early phase and requires rigorous validation. In this review we cover the basic concepts of virtual staining for histology and provide future insights into the utilization of artificial intelligence (AI)-enabled virtual histology.

Feature Fusion GAN Based Virtual Staining on Plant Microscopy Images

Virtual staining of microscopy specimens using GAN-based methods could resolve critical concerns of manual staining process as displayed in recent studies on histopathology images. However, most of these works use basic-GAN framework ignoring microscopy image characteristics and their performance were evaluated based on structural and error statistics (SSIM and PSNR) between synthetic and ground-truth without considering any color space although virtual staining deals with color transformation. Besides, major aspects of staining, like color, contrast, focus, image-realness etc. were totally ignored. However, modifications of GAN architecture for virtual staining might be suitable by incorporating microscopy image features. Further, its successful implementation need to be examined by considering various aspects of staining process. Therefore, we designed, a new feature-fusion-GAN for virtual staining followed by performance assessment by framing a state-of-the-art multi-evaluation framework that includes numerous metrics in -qualitative (based on histogram-correlation of color and brightness); quantitative (SSIM and PSNR); focus aptitude (Brenner metrics and Spectral-Moments); and influence on perception (semantic perceptual influence score). For, experimental validation cell boundaries were highlighted by two different staining reagents, Safranin-O and Toluidine-Blue-O on plant microscopy images of potato tuber. We evaluated virtually stained image quality w.r.t ground-truth in RGB and YCbCr color spaces based on defined metrics and results are found very consistent. Further, impact of feature fusion has been demonstrated. Collectively, this study could be a baseline towards guiding architectural upgrading of deep pipelines for virtual staining of diverse microscopy modalities followed by future benchmark methodology or protocols.

Roadmap on computational methods in optical imaging and holography [invited]

Computational methods have been established as cornerstones in optical imaging and holography in recent years. Every year, the dependence of optical imaging and holography on computational methods is increasing significantly to the extent that optical methods and components are being completely and efficiently replaced with computational methods at low cost. This roadmap reviews the current scenario in four major areas namely incoherent digital holography, quantitative phase imaging, imaging through scattering layers, and super-resolution imaging. In addition to registering the perspectives of the modern-day architects of the above research areas, the roadmap also reports some of the latest studies on the topic. Computational codes and pseudocodes are presented for computational methods in a plug-and-play fashion for readers to not only read and understand but also practice the latest algorithms with their data. We believe that this roadmap will be a valuable tool for analyzing the current trends in computational methods to predict and prepare the future of computational methods in optical imaging and holography.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00340-024-08280-3.

In silico labeling in cell biology: Potential and limitations

In silico labeling is the computational cross-modality image translation where the output modality is a subcellular marker that is not specifically encoded in the input image, for example, in silico localization of organelles from transmitted light images. In principle, in silico labeling has the potential to facilitate rapid live imaging of multiple organelles with reduced photobleaching and phototoxicity, a technology enabling a major leap toward understanding the cell as an integrated complex system. However, five years have passed since feasibility was attained, without any demonstration of using in silico labeling to uncover new biological insight. In here, we discuss the current state of in silico labeling, the limitations preventing it from becoming a practical tool, and how we can overcome these limitations to reach its full potential.

Training immunophenotyping deep learning models with the same-section ground truth cell label derivation method improves virtual staining accuracy

Introduction

Deep learning (DL) models predicting biomarker expression in images of hematoxylin and eosin (H&E)-stained tissues can improve access to multi-marker immunophenotyping, crucial for therapeutic monitoring, biomarker discovery, and personalized treatment development. Conventionally, these models are trained on ground truth cell labels derived from IHC-stained tissue sections adjacent to H&E-stained ones, which might be less accurate than labels from the same section. Although many such DL models have been developed, the impact of ground truth cell label derivation methods on their performance has not been studied.

Methodology

In this study, we assess the impact of cell label derivation on H&E model performance, with CD3+ T-cells in lung cancer tissues as a proof-of-concept. We compare two Pix2Pix generative adversarial network (P2P-GAN)-based virtual staining models: one trained with cell labels obtained from the same tissue section as the H&E-stained section (the 'same-section' model) and one trained on cell labels from an adjacent tissue section (the 'serial-section' model).

Results

We show that the same-section model exhibited significantly improved prediction performance compared to the 'serial-section' model. Furthermore, the same-section model outperformed the serial-section model in stratifying lung cancer patients within a public lung cancer cohort based on survival outcomes, demonstrating its potential clinical utility.

Discussion

Collectively, our findings suggest that employing ground truth cell labels obtained through the same-section approach boosts immunophenotyping DL solutions.

Deep learning-based virtual H& E staining from label-free autofluorescence lifetime images

Label-free autofluorescence lifetime is a unique feature of the inherent fluorescence signals emitted by natural fluorophores in biological samples. Fluorescence lifetime imaging microscopy (FLIM) can capture these signals enabling comprehensive analyses of biological samples. Despite the fundamental importance and wide application of FLIM in biomedical and clinical sciences, existing methods for analysing FLIM images often struggle to provide rapid and precise interpretations without reliable references, such as histology images, which are usually unavailable alongside FLIM images. To address this issue, we propose a deep learning (DL)-based approach for generating virtual Hematoxylin and Eosin (H&E) staining. By combining an advanced DL model with a contemporary image quality metric, we can generate clinical-grade virtual H&E-stained images from label-free FLIM images acquired on unstained tissue samples. Our experiments also show that the inclusion of lifetime information, an extra dimension beyond intensity, results in more accurate reconstructions of virtual staining when compared to using intensity-only images. This advancement allows for the instant and accurate interpretation of FLIM images at the cellular level without the complexities associated with co-registering FLIM and histology images. Consequently, we are able to identify distinct lifetime signatures of seven different cell types commonly found in the tumour microenvironment, opening up new opportunities towards biomarker-free tissue histology using FLIM across multiple cancer types.

Quantitative phase imaging based on holography: trends and new perspectives

In 1948, Dennis Gabor proposed the concept of holography, providing a pioneering solution to a quantitative description of the optical wavefront. After 75 years of development, holographic imaging has become a powerful tool for optical wavefront measurement and quantitative phase imaging. The emergence of this technology has given fresh energy to physics, biology, and materials science. Digital holography (DH) possesses the quantitative advantages of wide-field, non-contact, precise, and dynamic measurement capability for complex-waves. DH has unique capabilities for the propagation of optical fields by measuring light scattering with phase information. It offers quantitative visualization of the refractive index and thickness distribution of weak absorption samples, which plays a vital role in the pathophysiology of various diseases and the characterization of various materials. It provides a possibility to bridge the gap between the imaging and scattering disciplines. The propagation of wavefront is described by the complex amplitude. The complex-value in the complex-domain is reconstructed from the intensity-value measurement by camera in the real-domain. Here, we regard the process of holographic recording and reconstruction as a transformation between complex-domain and real-domain, and discuss the mathematics and physical principles of reconstruction. We review the DH in underlying principles, technical approaches, and the breadth of applications. We conclude with emerging challenges and opportunities based on combining holographic imaging with other methodologies that expand the scope and utility of holographic imaging even further. The multidisciplinary nature brings technology and application experts together in label-free cell biology, analytical chemistry, clinical sciences, wavefront sensing, and semiconductor production.

Improvement of gram staining effect by ethanol pretreatment and quantization of staining image by unsupervised machine learning

In this study, we propose an Ethanol Pretreatment Gram staining method that significantly enhances the color contrast of the stain, thereby improving the accuracy of judgement, and demonstrated the effectiveness of the modification by eliminating unaided-eye observational errors with unsupervised machine learning image analysis. By comparing the traditional Gram staining method with the improved method on various bacterial samples, results showed that the improved method offers distinct color contrast. Using multimodal assessment strategies, including unaided-eye observation, manual image segmentation, and advanced unsupervised machine learning automatic image segmentation, the practicality of ethanol pretreatment on Gram staining was comprehensively validated. In our quantitative analysis, the application of the CIEDE2000, and CMC color difference standards confirmed the significant effect of the method in enhancing the discrimination of Gram staining.This study not only improved the efficacy of Gram staining, but also provided a more accurate and standardized strategy for analyzing Gram staining results, which might provide an useful analytical tool in microbiological diagnostics.

Multimodal 2D and 3D microscopic mapping of growth cartilage by computational imaging techniques - a short review including new research

Being able to image the microstructure of growth cartilage is important for understanding the onset and progression of diseases such as osteochondrosis and osteoarthritis, as well as for developing new treatments and implants. Studies of cartilage using conventional optical brightfield microscopy rely heavily on histological staining, where the added chemicals provide tissue-specific colours. Other microscopy contrast mechanisms include polarization, phase- and scattering contrast, enabling non-stained or 'label-free' imaging that significantly simplifies the sample preparation, thereby also reducing the risk of artefacts. Traditional high-performance microscopes tend to be both bulky and expensive.Computational imagingdenotes a range of techniques where computers with dedicated algorithms are used as an integral part of the image formation process. Computational imaging offers many advantages like 3D measurements, aberration correction and quantitative phase contrast, often combined with comparably cheap and compact hardware. X-ray microscopy is also progressing rapidly, in certain ways trailing the development of optical microscopy. In this study, we first briefly review the structures of growth cartilage and relevant microscopy characterization techniques, with an emphasis on Fourier ptychographic microscopy (FPM) and advanced x-ray microscopies. We next demonstrate with our own results computational imaging through FPM and compare the images with hematoxylin eosin and saffron (HES)-stained histology. Zernike phase contrast, and the nonlinear optical microscopy techniques of second harmonic generation (SHG) and two-photon excitation fluorescence (TPEF) are explored. Furthermore, X-ray attenuation-, phase- and diffraction-contrast computed tomography (CT) images of the very same sample are presented for comparisons. Future perspectives on the links to artificial intelligence, dynamic studies andin vivopossibilities conclude the article.

Lung Cancer Diagnosis on Virtual Histologically Stained Tissue Using Weakly Supervised Learning

Lung adenocarcinoma (LUAD) is the most common primary lung cancer and accounts for 40% of all lung cancer cases. The current gold standard for lung cancer analysis is based on the pathologists' interpretation of hematoxylin and eosin (H&E)-stained tissue slices viewed under a brightfield microscope or a digital slide scanner. Computational pathology using deep learning has been proposed to detect lung cancer on histology images. However, the histological staining workflow to acquire the H&E-stained images and the subsequent cancer diagnosis procedures are labor-intensive and time-consuming with tedious sample preparation steps and repetitive manual interpretation, respectively. In this work, we propose a weakly supervised learning method for LUAD classification on label-free tissue slices with virtual histological staining. The autofluorescence images of label-free tissue with histopathological information can be converted into virtual H&E-stained images by a weakly supervised deep generative model. For the downstream LUAD classification task, we trained the attention-based multiple-instance learning model with different settings on the open-source LUAD H&E-stained whole-slide images (WSIs) dataset from the Cancer Genome Atlas (TCGA). The model was validated on the 150 H&E-stained WSIs collected from patients in Queen Mary Hospital and Prince of Wales Hospital with an average area under the curve (AUC) of 0.961. The model also achieved an average AUC of 0.973 on 58 virtual H&E-stained WSIs, comparable to the results on 58 standard H&E-stained WSIs with an average AUC of 0.977. The attention heatmaps of virtual H&E-stained WSIs and ground-truth H&E-stained WSIs can indicate tumor regions of LUAD tissue slices. In conclusion, the proposed diagnostic workflow on virtual H&E-stained WSIs of label-free tissue is a rapid, cost effective, and interpretable approach to assist clinicians in postoperative pathological examinations. The method could serve as a blueprint for other label-free imaging modalities and disease contexts.

Digital histological staining of tissue slide images from optical coherence microscopy

The convergence of staining-free optical imaging and digital staining technologies has become a central focus in digital pathology, presenting significant advantages in streamlining specimen preparation and expediting the rapid acquisition of histopathological information. Despite the inherent merits of optical coherence microscopy (OCM) as a staining-free technique, its widespread application in observing histopathological slides has been constrained. This study introduces a novel approach by combining wide-field OCM with digital staining technology for the imaging of histopathological slides. Through the optimization of the histology slide production process satisfying the ground growth for digital staining as well as pronounced contrast for OCM imaging, successful imaging of various mouse tissues was achieved. Comparative analyses with conventional staining-based bright field images were executed to evaluate the proposed methodology's efficacy. Moreover, the study investigates the generalization of digital staining color appearance to ensure consistent histopathology, considering tissue-specific and thickness-dependent variations.

Perspective on quantitative phase imaging to improve precision cancer medicine

Significance

Quantitative phase imaging (QPI) offers a label-free approach to non-invasively characterize cellular processes by exploiting their refractive index based intrinsic contrast. QPI captures this contrast by translating refractive index associated phase shifts into intensity-based quantifiable data with nanoscale sensitivity. It holds significant potential for advancing precision cancer medicine by providing quantitative characterization of the biophysical properties of cells and tissue in their natural states.

Aim

This perspective aims to discuss the potential of QPI to increase our understanding of cancer development and its response to therapeutics. It also explores new developments in QPI methods towards advancing personalized cancer therapy and early detection.

Approach

We begin by detailing the technical advancements of QPI, examining its implementations across transmission and reflection geometries and phase retrieval methods, both interferometric and non-interferometric. The focus then shifts to QPI's applications in cancer research, including dynamic cell mass imaging for drug response assessment, cancer risk stratification, and in-vivo tissue imaging.

Results

QPI has emerged as a crucial tool in precision cancer medicine, offering insights into tumor biology and treatment efficacy. Its sensitivity to detecting nanoscale changes holds promise for enhancing cancer diagnostics, risk assessment, and prognostication. The future of QPI is envisioned in its integration with artificial intelligence, morpho-dynamics, and spatial biology, broadening its impact in cancer research.

Conclusions

QPI presents significant potential in advancing precision cancer medicine and redefining our approach to cancer diagnosis, monitoring, and treatment. Future directions include harnessing high-throughput dynamic imaging, 3D QPI for realistic tumor models, and combining artificial intelligence with multi-omics data to extend QPI's capabilities. As a result, QPI stands at the forefront of cancer research and clinical application in cancer care.

Automated Whole Slide Imaging for Label-Free Histology Using Photon Absorption Remote Sensing Microscopy

OBJECTIVE

Pathologists rely on histochemical stains to impart contrast in thin translucent tissue samples, revealing tissue features necessary for identifying pathological conditions. However, the chemical labeling process is destructive and often irreversible or challenging to undo, imposing practical limits on the number of stains that can be applied to the same tissue section. Here we present an automated label-free whole slide scanner using a PARS microscope designed for imaging thin, transmissible samples.

METHODS

Peak SNR and in-focus acquisitions are achieved across entire tissue sections using the scattering signal from the PARS detection beam to measure the optimal focal plane. Whole slide images (WSI) are seamlessly stitched together using a custom contrast leveling algorithm. Identical tissue sections are subsequently H&E stained and brightfield imaged. The one-to-one WSIs from both modalities are visually and quantitatively compared.

RESULTS

PARS WSIs are presented at standard 40x magnification in malignant human breast and skin samples. We show correspondence of subcellular diagnostic details in both PARS and H&E WSIs and demonstrate virtual H&E staining of an entire PARS WSI. The one-to-one WSI from both modalities show quantitative similarity in nuclear features and structural information.

CONCLUSION

PARS WSIs are compatible with existing digital pathology tools, and samples remain suitable for histochemical, immunohistochemical, and other staining techniques.

SIGNIFICANCE

This work is a critical advance for integrating label-free optical methods into standard histopathology workflows.

Quantitative phase imaging techniques for measuring scattering properties of cells and tissues: a review-part I

Significance

Quantitative phase imaging (QPI) techniques offer intrinsic information about the sample of interest in a label-free, noninvasive manner and have an enormous potential for wide biomedical applications with negligible perturbations to the natural state of the sample in vitro.

Aim

We aim to present an in-depth review of the scattering formulation of light-matter interactions as applied to biological samples such as cells and tissues, discuss the relevant quantitative phase measurement techniques, and present a summary of various reported applications.

Approach

We start with scattering theory and scattering properties of biological samples followed by an exploration of various microscopy configurations for 2D QPI for measurement of structure and dynamics.

Results

We reviewed 157 publications and presented a range of QPI techniques and discussed suitable applications for each. We also presented the theoretical frameworks for phase reconstruction associated with the discussed techniques and highlighted their domains of validity.

Conclusions

We provide detailed theoretical as well as system-level information for a wide range of QPI techniques. Our study can serve as a guideline for new researchers looking for an exhaustive literature review of QPI methods and relevant applications.

The virtual staining method by quantitative phase imaging for label free lymphocytes based on self-supervised iteration cycle-consistent adversarial networks

Quantitative phase imaging (QPI) provides 3D structural and morphological information for label free living cells. Unfortunately, this quantitative phase information cannot meet doctors' diagnostic requirements of the clinical "gold standard," which displays stained cells' pathological states based on 2D color features. To make QPI results satisfy the clinical "gold standard," the virtual staining method by QPI for label free lymphocytes based on self-supervised iteration Cycle-Consistent Adversarial Networks (CycleGANs) is proposed herein. The 3D phase information of QPI is, therefore, trained and transferred to a kind of 2D "virtual staining" image that is well in agreement with "gold standard" results. To solve the problem that unstained QPI and stained "gold standard" results cannot be obtained for the same label free living cell, the self-supervised iteration for the CycleGAN deep learning algorithm is designed to obtain a trained stained result as the ground truth for error evaluation. The structural similarity index of our virtual staining experimental results for 8756 lymphocytes is 0.86. Lymphocytes' area errors after converting to 2D virtual stained results from 3D phase information are less than 3.59%. The mean error of the nuclear to cytoplasmic ratio is 2.69%, and the color deviation from the "gold standard" is less than 6.67%.

Enhanced Multiscale Human Brain Imaging by Semi-supervised Digital Staining and Serial Sectioning Optical Coherence Tomography

A major challenge in neuroscience is to visualize the structure of the human brain at different scales. Traditional histology reveals micro- and meso-scale brain features, but suffers from staining variability, tissue damage and distortion that impedes accurate 3D reconstructions. Here, we present a new 3D imaging framework that combines serial sectioning optical coherence tomography (S-OCT) with a deep-learning digital staining (DS) model. We develop a novel semi-supervised learning technique to facilitate DS model training on weakly paired images. The DS model performs translation from S-OCT to Gallyas silver staining. We demonstrate DS on various human cerebral cortex samples with consistent staining quality. Additionally, we show that DS enhances contrast across cortical layer boundaries. Furthermore, we showcase geometry-preserving 3D DS on cubic-centimeter tissue blocks and visualization of meso-scale vessel networks in the white matter. We believe that our technique offers the potential for high-throughput, multiscale imaging of brain tissues and may facilitate studies of brain structures.

Coherent light scattering from cellular dynamics in living tissues

This review examines the biological physics of intracellular transport probed by the coherent optics of dynamic light scattering from optically thick living tissues. Cells and their constituents are in constant motion, composed of a broad range of speeds spanning many orders of magnitude that reflect the wide array of functions and mechanisms that maintain cellular health. From the organelle scale of tens of nanometers and upward in size, the motion inside living tissue is actively driven rather than thermal, propelled by the hydrolysis of bioenergetic molecules and the forces of molecular motors. Active transport can mimic the random walks of thermal Brownian motion, but mean-squared displacements are far from thermal equilibrium and can display anomalous diffusion through Lévy or fractional Brownian walks. Despite the average isotropic three-dimensional environment of cells and tissues, active cellular or intracellular transport of single light-scattering objects is often pseudo-one-dimensional, for instance as organelle displacement persists along cytoskeletal tracks or as membranes displace along the normal to cell surfaces, albeit isotropically oriented in three dimensions. Coherent light scattering is a natural tool to characterize such tissue dynamics because persistent directed transport induces Doppler shifts in the scattered light. The many frequency-shifted partial waves from the complex and dynamic media interfere to produce dynamic speckle that reveals tissue-scale processes through speckle contrast imaging and fluctuation spectroscopy. Low-coherence interferometry, dynamic optical coherence tomography, diffusing-wave spectroscopy, diffuse-correlation spectroscopy, differential dynamic microscopy and digital holography offer coherent detection methods that shed light on intracellular processes. In health-care applications, altered states of cellular health and disease display altered cellular motions that imprint on the statistical fluctuations of the scattered light. For instance, the efficacy of medical therapeutics can be monitored by measuring the changes they induce in the Doppler spectra of livingex vivocancer biopsies.

Towards ultrafast quantitative phase imaging via differentiable microscopy [Invited]

With applications ranging from metabolomics to histopathology, quantitative phase microscopy (QPM) is a powerful label-free imaging modality. Despite significant advances in fast multiplexed imaging sensors and deep-learning-based inverse solvers, the throughput of QPM is currently limited by the pixel-rate of the image sensors. Complementarily, to improve throughput further, here we propose to acquire images in a compressed form so that more information can be transferred beyond the existing hardware bottleneck of the image sensor. To this end, we present a numerical simulation of a learnable optical compression-decompression framework that learns content-specific features. The proposed differentiable quantitative phase microscopy (∂-QPM) first uses learnable optical processors as image compressors. The intensity representations produced by these optical processors are then captured by the imaging sensor. Finally, a reconstruction network running on a computer decompresses the QPM images post aquisition. In numerical experiments, the proposed system achieves compression of × 64 while maintaining the SSIM of ∼0.90 and PSNR of ∼30 dB on cells. The results demonstrated by our experiments open up a new pathway to QPM systems that may provide unprecedented throughput improvements.

Noninvasive Nonlinear Optical Computational Histology

Cancer remains a global health challenge, demanding early detection and accurate diagnosis for improved patient outcomes. An intelligent paradigm is introduced that elevates label-free nonlinear optical imaging with contrastive patch-wise learning, yielding stain-free nonlinear optical computational histology (NOCH). NOCH enables swift, precise diagnostic analysis of fresh tissues, reducing patient anxiety and healthcare costs. Nonlinear modalities are evaluated, including stimulated Raman scattering and multiphoton imaging, for their ability to enhance tumor microenvironment sensitivity, pathological analysis, and cancer examination. Quantitative analysis confirmed that NOCH images accurately reproduce nuclear morphometric features across different cancer stages. Key diagnostic features, such as nuclear morphology, size, and nuclear-cytoplasmic contrast, are well preserved. NOCH models also demonstrate promising generalization when applied to other pathological tissues. The study unites label-free nonlinear optical imaging with histopathology using contrastive learning to establish stain-free computational histology. NOCH provides a rapid, non-invasive, and precise approach to surgical pathology, holding immense potential for revolutionizing cancer diagnosis and surgical interventions.

Virtual histological staining of unlabeled autopsy tissue

Traditional histochemical staining of post-mortem samples often confronts inferior staining quality due to autolysis caused by delayed fixation of cadaver tissue, and such chemical staining procedures covering large tissue areas demand substantial labor, cost and time. Here, we demonstrate virtual staining of autopsy tissue using a trained neural network to rapidly transform autofluorescence images of label-free autopsy tissue sections into brightfield equivalent images, matching hematoxylin and eosin (H&E) stained versions of the same samples. The trained model can effectively accentuate nuclear, cytoplasmic and extracellular features in new autopsy tissue samples that experienced severe autolysis, such as COVID-19 samples never seen before, where the traditional histochemical staining fails to provide consistent staining quality. This virtual autopsy staining technique provides a rapid and resource-efficient solution to generate artifact-free H&E stains despite severe autolysis and cell death, also reducing labor, cost and infrastructure requirements associated with the standard histochemical staining.

Angio-Net: deep learning-based label-free detection and morphometric analysis of in vitro angiogenesis

Despite significant advancements in three-dimensional (3D) cell culture technology and the acquisition of extensive data, there is an ongoing need for more effective and dependable data analysis methods. These concerns arise from the continued reliance on manual quantification techniques. In this study, we introduce a microphysiological system (MPS) that seamlessly integrates 3D cell culture to acquire large-scale imaging data and employs deep learning-based virtual staining for quantitative angiogenesis analysis. We utilize a standardized microfluidic device to obtain comprehensive angiogenesis data. Introducing Angio-Net, a novel solution that replaces conventional immunocytochemistry, we convert brightfield images into label-free virtual fluorescence images through the fusion of SegNet and cGAN. Moreover, we develop a tool capable of extracting morphological blood vessel features and automating their measurement, facilitating precise quantitative analysis. This integrated system proves to be invaluable for evaluating drug efficacy, including the assessment of anticancer drugs on targets such as the tumor microenvironment. Additionally, its unique ability to enable live cell imaging without the need for cell fixation promises to broaden the horizons of pharmaceutical and biological research. Our study pioneers a powerful approach to high-throughput angiogenesis analysis, marking a significant advancement in MPS.

Beyond fluorescence: advances in computational label-free full specificity in 3D quantitative phase microscopy

Despite remarkable progresses in quantitative phase imaging (QPI) microscopes, their wide acceptance is limited due to the lack of specificity compared with the well-established fluorescence microscopy. In fact, the absence of fluorescent tag prevents to identify subcellular structures in single cells, making challenging the interpretation of label-free 2D and 3D phase-contrast data. Great effort has been made by many groups worldwide to address and overcome such limitation. Different computational methods have been proposed and many more are currently under investigation to achieve label-free microscopic imaging at single-cell level to recognize and quantify different subcellular compartments. This route promises to bridge the gap between QPI and FM for real-world applications.

Dual contrastive learning based image-to-image translation of unstained skin tissue into virtually stained H&E images

Staining is a crucial step in histopathology that prepares tissue sections for microscopic examination. Hematoxylin and eosin (H&E) staining, also known as basic or routine staining, is used in 80% of histopathology slides worldwide. To enhance the histopathology workflow, recent research has focused on integrating generative artificial intelligence and deep learning models. These models have the potential to improve staining accuracy, reduce staining time, and minimize the use of hazardous chemicals, making histopathology a safer and more efficient field. In this study, we introduce a novel three-stage, dual contrastive learning-based, image-to-image generative (DCLGAN) model for virtually applying an "H&E stain" to unstained skin tissue images. The proposed model utilizes a unique learning setting comprising two pairs of generators and discriminators. By employing contrastive learning, our model maximizes the mutual information between traditional H&E-stained and virtually stained H&E patches. Our dataset consists of pairs of unstained and H&E-stained images, scanned with a brightfield microscope at 20 × magnification, providing a comprehensive set of training and testing images for evaluating the efficacy of our proposed model. Two metrics, Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), were used to quantitatively evaluate virtual stained slides. Our analysis revealed that the average FID score between virtually stained and H&E-stained images (80.47) was considerably lower than that between unstained and virtually stained slides (342.01), and unstained and H&E stained (320.4) indicating a similarity virtual and H&E stains. Similarly, the mean KID score between H&E stained and virtually stained images (0.022) was significantly lower than the mean KID score between unstained and H&E stained (0.28) or unstained and virtually stained (0.31) images. In addition, a group of experienced dermatopathologists evaluated traditional and virtually stained images and demonstrated an average agreement of 78.8% and 90.2% for paired and single virtual stained image evaluations, respectively. Our study demonstrates that the proposed three-stage dual contrastive learning-based image-to-image generative model is effective in generating virtual stained images, as indicated by quantified parameters and grader evaluations. In addition, our findings suggest that GAN models have the potential to replace traditional H&E staining, which can reduce both time and environmental impact. This study highlights the promise of virtual staining as a viable alternative to traditional staining techniques in histopathology.

Noise learning of instruments for high-contrast, high-resolution and fast hyperspectral microscopy and nanoscopy

The low scattering efficiency of Raman scattering makes it challenging to simultaneously achieve good signal-to-noise ratio (SNR), high imaging speed, and adequate spatial and spectral resolutions. Here, we report a noise learning (NL) approach that estimates the intrinsic noise distribution of each instrument by statistically learning the noise in the pixel-spatial frequency domain. The estimated noise is then removed from the noisy spectra. This enhances the SNR by ca. 10 folds, and suppresses the mean-square error by almost 150 folds. NL allows us to improve the positioning accuracy and spatial resolution and largely eliminates the impact of thermal drift on tip-enhanced Raman spectroscopic nanoimaging. NL is also applicable to enhance SNR in fluorescence and photoluminescence imaging. Our method manages the ground truth spectra and the instrumental noise simultaneously within the training dataset, which bypasses the tedious labelling of huge dataset required in conventional deep learning, potentially shifting deep learning from sample-dependent to instrument-dependent.

Multi-channel feature extraction for virtual histological staining of photon absorption remote sensing images

Accurate and fast histological staining is crucial in histopathology, impacting diagnostic precision and reliability. Traditional staining methods are time-consuming and subjective, causing delays in diagnosis. Digital pathology plays a vital role in advancing and optimizing histology processes to improve efficiency and reduce turnaround times. This study introduces a novel deep learning-based framework for virtual histological staining using photon absorption remote sensing (PARS) images. By extracting features from PARS time-resolved signals using a variant of the K-means method, valuable multi-modal information is captured. The proposed multi-channel cycleGAN model expands on the traditional cycleGAN framework, allowing the inclusion of additional features. Experimental results reveal that specific combinations of features outperform the conventional channels by improving the labeling of tissue structures prior to model training. Applied to human skin and mouse brain tissue, the results underscore the significance of choosing the optimal combination of features, as it reveals a substantial visual and quantitative concurrence between the virtually stained and the gold standard chemically stained hematoxylin and eosin images, surpassing the performance of other feature combinations. Accurate virtual staining is valuable for reliable diagnostic information, aiding pathologists in disease classification, grading, and treatment planning. This study aims to advance label-free histological imaging and opens doors for intraoperative microscopy applications.

Intraoperative margin assessment for basal cell carcinoma with deep learning and histologic tumor mapping to surgical site

Successful treatment of solid cancers relies on complete surgical excision of the tumor either for definitive treatment or before adjuvant therapy. Intraoperative and postoperative radial sectioning, the most common form of margin assessment, can lead to incomplete excision and increase the risk of recurrence and repeat procedures. Mohs Micrographic Surgery is associated with complete removal of basal cell and squamous cell carcinoma through real-time margin assessment of 100% of the peripheral and deep margins. Real-time assessment in many tumor types is constrained by tissue size, complexity, and specimen processing / assessment time during general anesthesia. We developed an artificial intelligence platform to reduce the tissue preprocessing and histological assessment time through automated grossing recommendations, mapping and orientation of tumor to the surgical specimen. Using basal cell carcinoma as a model system, results demonstrate that this approach can address surgical laboratory efficiency bottlenecks for rapid and complete intraoperative margin assessment.

Time for a full digital approach in nephropathology: a systematic review of current artificial intelligence applications and future directions

INTRODUCTION

Artificial intelligence (AI) integration in nephropathology has been growing rapidly in recent years, facing several challenges including the wide range of histological techniques used, the low occurrence of certain diseases, and the need for data sharing. This narrative review retraces the history of AI in nephropathology and provides insights into potential future developments.

METHODS

Electronic searches in PubMed-MEDLINE and Embase were made to extract pertinent articles from the literature. Works about automated image analysis or the application of an AI algorithm on non-neoplastic kidney histological samples were included and analyzed to extract information such as publication year, AI task, and learning type. Prepublication servers and reviews were not included.

RESULTS

Seventy-six (76) original research articles were selected. Most of the studies were conducted in the United States in the last 7 years. To date, research has been mainly conducted on relatively easy tasks, like single-stain glomerular segmentation. However, there is a trend towards developing more complex tasks such as glomerular multi-stain classification.

CONCLUSION

Deep learning has been used to identify patterns in complex histopathology data and looks promising for the comprehensive assessment of renal biopsy, through the use of multiple stains and virtual staining techniques. Hybrid and collaborative learning approaches have also been explored to utilize large amounts of unlabeled data. A diverse team of experts, including nephropathologists, computer scientists, and clinicians, is crucial for the development of AI systems for nephropathology. Collaborative efforts among multidisciplinary experts result in clinically relevant and effective AI tools.

TIE-GANs: single-shot quantitative phase imaging using transport of intensity equation with integration of GANs

Significance

Artificial intelligence (AI) has become a prominent technology in computational imaging over the past decade. The expeditious and label-free characteristics of quantitative phase imaging (QPI) render it a promising contender for AI investigation. Though interferometric methodologies exhibit potential efficacy, their implementation involves complex experimental platforms and computationally intensive reconstruction procedures. Hence, non-interferometric methods, such as transport of intensity equation (TIE), are preferred over interferometric methods.

Aim

TIE method, despite its effectiveness, is tedious as it requires the acquisition of many images at varying defocus planes. The proposed methodology holds the ability to generate a phase image utilizing a single intensity image using generative adversarial networks (GANs). We present a method called TIE-GANs to overcome the multi-shot scheme of conventional TIE.

Approach

The present investigation employs the TIE as a QPI methodology, which necessitates reduced experimental and computational efforts. TIE is being used for the dataset preparation as well. The proposed method captures images from different defocus planes for training. Our approach uses an image-to-image translation technique to produce phase maps and is based on GANs. The main contribution of this work is the introduction of GANs with TIE (TIE:GANs) that can give better phase reconstruction results with shorter computation times. This is the first time the GANs is proposed for TIE phase retrieval.

Results

The characterization of the system was carried out with microbeads of 4    μ m size and structural similarity index (SSIM) for microbeads was found to be 0.98. We demonstrated the application of the proposed method with oral cells, which yielded a maximum SSIM value of 0.95. The key characteristics include mean squared error and peak-signal-to-noise ratio values of 140 and 26.42 dB for oral cells and 100 and 28.10 dB for microbeads.

Conclusions

The proposed methodology holds the ability to generate a phase image utilizing a single intensity image. Our method is feasible for digital cytology because of its reported high value of SSIM. Our approach can handle defocused images in such a way that it can take intensity image from any defocus plane within the provided range and able to generate phase map.

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

PhaseFIT: live-organoid phase-fluorescent image transformation via generative AI

Organoid models have provided a powerful platform for mechanistic investigations into fundamental biological processes involved in the development and function of organs. Despite the potential for image-based phenotypic quantification of organoids, their complex 3D structure, and the time-consuming and labor-intensive nature of immunofluorescent staining present significant challenges. In this work, we developed a virtual painting system, PhaseFIT (phase-fluorescent image transformation) utilizing customized and morphologically rich 2.5D intestinal organoids, which generate virtual fluorescent images for phenotypic quantification via accessible and low-cost organoid phase images. This system is driven by a novel segmentation-informed deep generative model that specializes in segmenting overlap and proximity between objects. The model enables an annotation-free digital transformation from phase-contrast to multi-channel fluorescent images. The virtual painting results of nuclei, secretory cell markers, and stem cells demonstrate that PhaseFIT outperforms the existing deep learning-based stain transformation models by generating fine-grained visual content. We further validated the efficiency and accuracy of PhaseFIT to quantify the impacts of three compounds on crypt formation, cell population, and cell stemness. PhaseFIT is the first deep learning-enabled virtual painting system focused on live organoids, enabling large-scale, informative, and efficient organoid phenotypic quantification. PhaseFIT would enable the use of organoids in high-throughput drug screening applications.

Roadmap on Label-Free Super-Resolution Imaging

Label-free super-resolution (LFSR) imaging relies on light-scattering processes in nanoscale objects without a need for fluorescent (FL) staining required in super-resolved FL microscopy. The objectives of this Roadmap are to present a comprehensive vision of the developments, the state-of-the-art in this field, and to discuss the resolution boundaries and hurdles which need to be overcome to break the classical diffraction limit of the LFSR imaging. The scope of this Roadmap spans from the advanced interference detection techniques, where the diffraction-limited lateral resolution is combined with unsurpassed axial and temporal resolution, to techniques with true lateral super-resolution capability which are based on understanding resolution as an information science problem, on using novel structured illumination, near-field scanning, and nonlinear optics approaches, and on designing superlenses based on nanoplasmonics, metamaterials, transformation optics, and microsphere-assisted approaches. To this end, this Roadmap brings under the same umbrella researchers from the physics and biomedical optics communities in which such studies have often been developing separately. The ultimate intent of this paper is to create a vision for the current and future developments of LFSR imaging based on its physical mechanisms and to create a great opening for the series of articles in this field.

Toward cell nuclei precision between OCT and H&E images translation using signal-to-noise ratio cycle-consistency

Medical image-to-image translation is often difficult and of limited effectiveness due to the differences in image acquisition mechanisms and the diverse structure of biological tissues. This work presents an unpaired image translation model between in-vivo optical coherence tomography (OCT) and ex-vivo Hematoxylin and eosin (H&E) stained images without the need for image stacking, registration, post-processing, and annotation. The model can generate high-quality and highly accurate virtual medical images, and is robust and bidirectional. Our framework introduces random noise to (1) blur redundant features, (2) defend against self-adversarial attacks, (3) stabilize inverse conversion, and (4) mitigate the impact of OCT speckles. We also demonstrate that our model can be pre-trained and then fine-tuned using images from different OCT systems in just a few epochs. Qualitative and quantitative comparisons with traditional image-to-image translation models show the robustness of our proposed signal-to-noise ratio (SNR) cycle-consistency method.