Stain-less staining for computed histopathology

Virtual histological staining of unlabeled autopsy tissue

Traditional histochemical staining of post-mortem samples often confronts inferior staining quality due to autolysis caused by delayed fixation of cadaver tissue, and such chemical staining procedures covering large tissue areas demand substantial labor, cost and time. Here, we demonstrate virtual staining of autopsy tissue using a trained neural network to rapidly transform autofluorescence images of label-free autopsy tissue sections into brightfield equivalent images, matching hematoxylin and eosin (H&E) stained versions of the same samples. The trained model can effectively accentuate nuclear, cytoplasmic and extracellular features in new autopsy tissue samples that experienced severe autolysis, such as COVID-19 samples never seen before, where the traditional histochemical staining fails to provide consistent staining quality. This virtual autopsy staining technique provides a rapid and resource-efficient solution to generate artifact-free H&E stains despite severe autolysis and cell death, also reducing labor, cost and infrastructure requirements associated with the standard histochemical staining.

Roadmap on optical sensors

Optical sensors and sensing technologies are playing a more and more important role in our modern world. From micro-probes to large devices used in such diverse areas like medical diagnosis, defence, monitoring of industrial and environmental conditions, optics can be used in a variety of ways to achieve compact, low cost, stand-off sensing with extreme sensitivity and selectivity. Actually, the challenges to the design and functioning of an optical sensor for a particular application requires intimate knowledge of the optical, material, and environmental properties that can affect its performance. This roadmap on optical sensors addresses different technologies and application areas. It is constituted by twelve contributions authored by world-leading experts, providing insight into the current state-of-the-art and the challenges their respective fields face. Two articles address the area of optical fibre sensors, encompassing both conventional and specialty optical fibres. Several other articles are dedicated to laser-based sensors, micro- and nano-engineered sensors, whispering-gallery mode and plasmonic sensors. The use of optical sensors in chemical, biological and biomedical areas is discussed in some other papers. Different approaches required to satisfy applications at visible, infrared and THz spectral regions are also discussed.

Accelerating Cancer Histopathology Workflows with Chemical Imaging and Machine Learning

Histopathology has remained a cornerstone for biomedical tissue assessment for over a century, with a resource-intensive workflow involving biopsy or excision, gross examination, sampling, tissue processing to snap frozen or formalin-fixed paraffin-embedded blocks, sectioning, staining, optical imaging, and microscopic assessment. Emerging chemical imaging approaches, including stimulated Raman scattering (SRS) microscopy, can directly measure inherent molecular composition in tissue (thereby dispensing with the need for tissue processing, sectioning, and using dyes) and can use artificial intelligence (AI) algorithms to provide high-quality images. Here we show the integration of SRS microscopy in a pathology workflow to rapidly record chemical information from minimally processed fresh-frozen prostate tissue. Instead of using thin sections, we record data from intact thick tissues and use optical sectioning to generate images from multiple planes. We use a deep learning–based processing pipeline to generate virtual hematoxylin and eosin images. Next, we extend the computational method to generate archival-quality images in minutes, which are equivalent to those obtained from hours/days-long formalin-fixed, paraffin-embedded processing. We assessed the quality of images from the perspective of enabling pathologists to make decisions, demonstrating that the virtual stained image quality was diagnostically useful and the interpathologist agreement on prostate cancer grade was not impacted. Finally, because this method does not wash away lipids and small molecules, we assessed the utility of lipid chemical composition in determining grade. Together, the combination of chemical imaging and AI provides novel capabilities for rapid assessments in pathology by reducing the complexity and burden of current workflows.

SIGNIFICANCE

Archival-quality (formalin-fixed paraffin-embedded), thin-section diagnostic images are obtained from thick-cut, fresh-frozen prostate tissues without dyes or stains to expedite cancer histopathology by combining SRS microscopy and machine learning.

Digital Histopathology by Infrared Spectroscopic Imaging

Infrared (IR) spectroscopic imaging records spatially resolved molecular vibrational spectra, enabling a comprehensive measurement of the chemical makeup and heterogeneity of biological tissues. Combining this novel contrast mechanism in microscopy with the use of artificial intelligence can transform the practice of histopathology, which currently relies largely on human examination of morphologic patterns within stained tissue. First, this review summarizes IR imaging instrumentation especially suited to histopathology, analyses of its performance, and major trends. Second, an overview of data processing methods and application of machine learning is given, with an emphasis on the emerging use of deep learning. Third, a discussion on workflows in pathology is provided, with four categories proposed based on the complexity of methods and the analytical performance needed. Last, a set of guidelines, termed experimental and analytical specifications for spectroscopic imaging in histopathology, are proposed to help standardize the diversity of approaches in this emerging area.

The effect of neural network architecture on virtual H&E staining: Systematic assessment of histological feasibility

Conventional histopathology has relied on chemical staining for over a century. The staining process makes tissue sections visible to the human eye through a tedious and labor-intensive procedure that alters the tissue irreversibly, preventing repeated use of the sample. Deep learning-based virtual staining can potentially alleviate these shortcomings. Here, we used standard brightfield microscopy on unstained tissue sections and studied the impact of increased network capacity on the resulting virtually stained H&E images. Using the generative adversarial neural network model pix2pix as a baseline, we observed that replacing simple convolutions with dense convolution units increased the structural similarity score, peak signal-to-noise ratio, and nuclei reproduction accuracy. We also demonstrated highly accurate reproduction of histology, especially with increased network capacity, and demonstrated applicability to several tissues. We show that network architecture optimization can improve the image translation accuracy of virtual H&E staining, highlighting the potential of virtual staining in streamlining histopathological analysis.

Deep learning-enabled virtual histological staining of biological samples

Histological staining is the gold standard for tissue examination in clinical pathology and life-science research, which visualizes the tissue and cellular structures using chromatic dyes or fluorescence labels to aid the microscopic assessment of tissue. However, the current histological staining workflow requires tedious sample preparation steps, specialized laboratory infrastructure, and trained histotechnologists, making it expensive, time-consuming, and not accessible in resource-limited settings. Deep learning techniques created new opportunities to revolutionize staining methods by digitally generating histological stains using trained neural networks, providing rapid, cost-effective, and accurate alternatives to standard chemical staining methods. These techniques, broadly referred to as virtual staining, were extensively explored by multiple research groups and demonstrated to be successful in generating various types of histological stains from label-free microscopic images of unstained samples; similar approaches were also used for transforming images of an already stained tissue sample into another type of stain, performing virtual stain-to-stain transformations. In this Review, we provide a comprehensive overview of the recent research advances in deep learning-enabled virtual histological staining techniques. The basic concepts and the typical workflow of virtual staining are introduced, followed by a discussion of representative works and their technical innovations. We also share our perspectives on the future of this emerging field, aiming to inspire readers from diverse scientific fields to further expand the scope of deep learning-enabled virtual histological staining techniques and their applications.

Clinical Spectroscopy: Lost in Translation?

This Focal Point Review paper discusses the developments of biomedical Raman and infrared spectroscopy, and the recent strive towards these technologies being regarded as reliable clinical tools. The promise of vibrational spectroscopy in the field of biomedical science, alongside the development of computational methods for spectral analysis, has driven a plethora of proof-of-concept studies which convey the potential of various spectroscopic approaches. Here we report a brief review of the literature published over the past few decades, with a focus on the current technical, clinical, and economic barriers to translation, namely the limitations of many of the early studies, and the lack of understanding of clinical pathways, health technology assessments, regulatory approval, clinical feasibility, and funding applications. The field of biomedical vibrational spectroscopy must acknowledge and overcome these hurdles in order to achieve clinical efficacy. Current prospects have been overviewed with comment on the advised future direction of spectroscopic technologies, with the aspiration that many of these innovative approaches can ultimately reach the frontier of medical diagnostics and many clinical applications.

Image-to-image translation of label-free molecular vibrational images for a histopathological review using the UNet+/seg-cGAN model

Translating images generated by label-free microscopy imaging, such as Coherent Anti-Stokes Raman Scattering (CARS), into more familiar clinical presentations of histopathological images will help the adoption of real-time, spectrally resolved label-free imaging in clinical diagnosis. Generative adversarial networks (GAN) have made great progress in image generation and translation, but have been criticized for lacking precision. In particular, GAN has often misinterpreted image information and identified incorrect content categories during image translation of microscopy scans. To alleviate this problem, we developed a new Pix2pix GAN model that simultaneously learns classifying contents in the images from a segmentation dataset during the image translation training. Our model integrates UNet+ with seg-cGAN, conditional generative adversarial networks with partial regularization of segmentation. Technical innovations of the UNet+/seg-cGAN model include: (1) replacing UNet with UNet+ as the Pix2pix cGAN's generator to enhance pattern extraction and richness of the gradient, and (2) applying the partial regularization strategy to train a part of the generator network as the segmentation sub-model on a separate segmentation dataset, thus enabling the model to identify correct content categories during image translation. The quality of histopathological-like images generated based on label-free CARS images has been improved significantly.

Quantum Cascade Laser-Based Infrared Imaging as a Label-Free and Automated Approach to Determine Mutations in Lung Adenocarcinoma

Therapeutic decisions in lung cancer critically depend on the determination of histologic types and oncogene mutations. Therefore, tumor samples are subjected to standard histologic and immunohistochemical analyses and examined for relevant mutations using comprehensive molecular diagnostics. In this study, an alternative diagnostic approach for automatic and label-free detection of mutations in lung adenocarcinoma tissue using quantum cascade laser-based infrared imaging is presented. For this purpose, a five-step supervised classification algorithm was developed, which was not only able to detect tissue types and tumor lesions, but also the tumor type and mutation status of adenocarcinomas. Tumor detection was verified on a data set of 214 patient samples with a specificity of 97% and a sensitivity of 95%. Furthermore, histology typing was verified on samples from 203 of the 214 patients with a specificity of 97% and a sensitivity of 94% for adenocarcinoma. The most frequently occurring mutations in adenocarcinoma (KRAS, EGFR, and TP53) were differentiated by this technique. Detection of mutations was verified in 60 patient samples from the data set with a sensitivity and specificity of 95% for each mutation. This demonstrates that quantum cascade laser infrared imaging can be used to analyze morphologic differences as well as molecular changes. Therefore, this single, one-step measurement provides comprehensive diagnostics of lung cancer histology types and most frequent mutations.

A versatile deep learning architecture for classification and label-free prediction of hyperspectral images

Hyperspectral imaging is a technique that provides rich chemical or compositional information not regularly available to traditional imaging modalities such as intensity imaging or color imaging based on the reflection, transmission, or emission of light. Analysis of hyperspectral imaging often relies on machine learning methods to extract information. Here, we present a new flexible architecture, the U-within-U-Net, that can perform classification, segmentation, and prediction of orthogonal imaging modalities on a variety of hyperspectral imaging techniques. Specifically, we demonstrate feature segmentation and classification on the Indian Pines hyperspectral dataset and simultaneous location prediction of multiple drugs in mass spectrometry imaging of rat liver tissue. We further demonstrate label-free fluorescence image prediction from hyperspectral stimulated Raman scattering microscopy images. The applicability of the U-within-U-Net architecture on diverse datasets with widely varying input and output dimensions and data sources suggest that it has great potential in advancing the use of hyperspectral imaging across many different application areas ranging from remote sensing, to medical imaging, to microscopy.

INFORM: INFrared-based ORganizational Measurements of tumor and its microenvironment to predict patient survival

The structure and organization of a tumor and its microenvironment are often associated with cancer outcomes due to spatially varying molecular composition and signaling. A persistent challenge is to use this physical and chemical spatial organization to understand cancer progression. Here, we present a high-definition infrared imaging-based organizational measurement framework (INFORM) that leverages intrinsic chemical contrast of tissue to label unique components of the tumor and its microenvironment. Using objective and automated computational methods, further, we determine organization characteristics important for prediction. We show that the tumor spatial organization assessed with this framework is predictive of overall survival in colon cancer that adds to capability from clinical variables such as stage and grade, approximately doubling the risk of death in high-risk individuals. Our results open an all-digital avenue for measuring and studying the association between tumor spatial organization and disease progression.

Three-dimensional deeply generated holography [Invited]

In this paper, we present a noniterative method for 3D computer-generated holography based on deep learning. A convolutional neural network is adapted for directly generating a hologram to reproduce a 3D intensity pattern in a given class. We experimentally demonstrated the proposed method with optical reproductions of multiple layers based on phase-only Fourier holography. Our method is noniterative, but it achieves a reproduction quality comparable with that of iterative methods for a given class.

Concurrent Vibrational Circular Dichroism Measurements with Infrared Spectroscopic Imaging

Vibrational circular dichroism (VCD) spectroscopy has emerged as a powerful platform to quantify chirality, a vital biological property that performs a pivotal role in the metabolism of life organisms. With a photoelastic modulator (PEM) integrated into an infrared spectrometer, the differential response of a sample to the direction of circularly polarized light can be used to infer conformation handedness. However, these optical components inherently exhibit chromatic behavior and are typically optimized at discrete spectral frequencies. Advancements of discrete frequency infrared (DFIR) spectroscopic microscopes in spectral image quality and data throughput are promising for use toward analytical VCD measurements. Utilizing the PEM advantages incorporated into a custom-built QCL microscope, we demonstrate a point scanning VCD instrument capable of acquiring spectra rapidly across all fingerprint region wavelengths in transmission configuration. Moreover, for the first time, we also demonstrate the VCD imaging performance of our instrument for site-specific chirality mapping of biological tissue samples. This study offers some insight into future possibilities of examining small, localized changes in tissue that have major implications for systemic diseases and their progression, while also laying the groundwork for additional modeling and validation in advancing the capability of VCD spectroscopy and imaging.

Emerging Advances to Transform Histopathology Using Virtual Staining

In an age where digitization is widespread in clinical and preclinical workflows, pathology is still predominantly practiced by microscopic evaluation of stained tissue specimens affixed on glass slides. Over the last decade, new high throughput digital scanning microscopes have ushered in the era of digital pathology that, along with recent advances in machine vision, have opened up new possibilities for Computer-Aided-Diagnoses. Despite these advances, the high infrastructural costs related to digital pathology and the perception that the digitization process is an additional and nondirectly reimbursable step have challenged its widespread adoption. Here, we discuss how emerging virtual staining technologies and machine learning can help to disrupt the standard histopathology workflow and create new avenues for the diagnostic paradigm that will benefit patients and healthcare systems alike via digital pathology.

Infrared spectral microscopy as a tool to monitor lung fibrosis development in a model system

Tissue fibrosis is a progressive and destructive disease process that can occur in many different organs including the liver, kidney, skin, and lungs. Fibrosis is typically initiated by inflammation as a result of chronic insults such as infection, chemicals and autoimmune diseases. Current approaches to examine organ fibrosis are limited to radiological and histological analyses. Infrared spectroscopic imaging offers a potential alternative approach to gain insight into biochemical changes associated with fibrosis progression. In this study, we demonstrate that IR imaging of a mouse model of pulmonary fibrosis can identify biochemical changes observed with fibrosis progression and the beginning of resolution using K-means analysis, spectral ratios and multivariate data analysis. This study demonstrates that IR imaging may be a useful approach to understand the biochemical events associated with fibrosis initiation, progression and resolution for both the clinical setting and for assessing novel anti-fibrotic drugs in a model system.

Label-free, automated classification of microsatellite status in colorectal cancer by infrared imaging

Challenging histopathological diagnostics in cancer include microsatellite instability-high (MSI-H) colorectal cancer (CRC), which occurs in 15% of early-stage CRC and is caused by a deficiency in the mismatch repair system. The diagnosis of MSI-H cannot be reliably achieved by visual inspection of a hematoxylin and eosin stained thin section alone, but additionally requires subsequent molecular analysis. Time- and sample-intensive immunohistochemistry with subsequent fragment length analysis is used. The aim of the presented feasibility study is to test the ability of quantum cascade laser (QCL)-based infrared (IR) imaging as an alternative diagnostic tool for MSI-H in CRC. We analyzed samples from 100 patients with sporadic CRC UICC stage II and III. Forty samples were used to develop the random forest classifier and 60 samples to verify the results on an independent blinded dataset. Specifically, 100% sensitivity and 93% specificity were achieved based on the independent 30 MSI-H- and 30 microsatellite stable (MSS)-patient validation cohort. This showed that QCL-based IR imaging is able to distinguish between MSI-H and MSS for sporadic CRC - a question that goes beyond morphological features - based on the use of spatially resolved infrared spectra used as biomolecular fingerprints.

Label-free hematology analysis using deep-ultraviolet microscopy

Hematological analysis, via a complete blood count (CBC) and microscopy, is critical for screening, diagnosing, and monitoring blood conditions and diseases but requires complex equipment, multiple chemical reagents, laborious system calibration and procedures, and highly trained personnel for operation. Here we introduce a hematological assay based on label-free molecular imaging with deep-ultraviolet microscopy that can provide fast quantitative information of key hematological parameters to facilitate and improve hematological analysis. We demonstrate that this label-free approach yields 1) a quantitative five-part white blood cell differential, 2) quantitative red blood cell and hemoglobin characterization, 3) clear identification of platelets, and 4) detailed subcellular morphology. Analysis of tens of thousands of live cells is achieved in minutes without any sample preparation. Finally, we introduce a pseudocolorization scheme that accurately recapitulates the appearance of cells under conventional staining protocols for microscopic analysis of blood smears and bone marrow aspirates. Diagnostic efficacy is evaluated by a panel of hematologists performing a blind analysis of blood smears from healthy donors and thrombocytopenic and sickle cell disease patients. This work has significant implications toward simplifying and improving CBC and blood smear analysis, which is currently performed manually via bright-field microscopy, and toward the development of a low-cost, easy-to-use, and fast hematological analyzer as a point-of-care device and for low-resource settings.

Deeply coded aperture for lensless imaging

In this Letter, we present a method for jointly designing a coded aperture and a convolutional neural network for reconstructing an object from a single-shot lensless measurement. The coded aperture and the reconstruction network are connected with a deep learning framework in which the coded aperture is placed as a first convolutional layer. Our co-optimization method was experimentally demonstrated with a fully convolutional network, and its performance was compared to a coded aperture with a modified uniformly redundant array.

Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue

Histological staining is a vital step in diagnosing various diseases and has been used for more than a century to provide contrast in tissue sections, rendering the tissue constituents visible for microscopic analysis by medical experts. However, this process is time consuming, labour intensive, expensive and destructive to the specimen. Recently, the ability to virtually stain unlabelled tissue sections, entirely avoiding the histochemical staining step, has been demonstrated using tissue-stain-specific deep neural networks. Here, we present a new deep-learning-based framework that generates virtually stained images using label-free tissue images, in which different stains are merged following a micro-structure map defined by the user. This approach uses a single deep neural network that receives two different sources of information as its input: (1) autofluorescence images of the label-free tissue sample and (2) a "digital staining matrix", which represents the desired microscopic map of the different stains to be virtually generated in the same tissue section. This digital staining matrix is also used to virtually blend existing stains, digitally synthesizing new histological stains. We trained and blindly tested this virtual-staining network using unlabelled kidney tissue sections to generate micro-structured combinations of haematoxylin and eosin (H&E), Jones' silver stain, and Masson's trichrome stain. Using a single network, this approach multiplexes the virtual staining of label-free tissue images with multiple types of stains and paves the way for synthesizing new digital histological stains that can be created in the same tissue cross section, which is currently not feasible with standard histochemical staining methods.

Infrared Spectroscopic Imaging Visualizes a Prognostic Extracellular Matrix-Related Signature in Breast Cancer

Molecular analysis techniques such as gene expression analysis and proteomics have contributed greatly to our understanding of cancer heterogeneity. In prior studies, gene expression analysis was shown to stratify patient outcome on the basis of tumor-microenvironment associated genes. A specific gene expression profile, referred to as ECM3 (Extracellular Matrix Cluster 3), indicated poorer survival in patients with grade III tumors. In this work, we aimed to visualize the downstream effects of this gene expression profile onto the tissue, thus providing a spatial context to altered gene expression profiles. Using infrared spectroscopic imaging, we identified spectral patterns specific to the ECM3 gene expression profile, achieving a high spectral classification performance of 0.87 as measured by the area under the curve of the receiver operating characteristic curve. On a patient level, we correctly identified 20 out of 22 ECM3 group patients and 19 out of 20 non-ECM3 group patients by using this spectroscopic imaging-based classifier. By comparing pixels that were identified as ECM3 or non-ECM3 with H&E and IHC images, we were also able to observe an association between tissue morphology and the gene expression clusters, showing the ability of our method to capture broad outcome associated features from infrared images.

All-digital histopathology by infrared-optical hybrid microscopy

Optical microscopy for biomedical samples requires expertise in staining to visualize structure and composition. Midinfrared (mid-IR) spectroscopic imaging offers label-free molecular recording and virtual staining by probing fundamental vibrational modes of molecular components. This quantitative signal can be combined with machine learning to enable microscopy in diverse fields from cancer diagnoses to forensics. However, absorption of IR light by common optical imaging components makes mid-IR light incompatible with modern optical microscopy and almost all biomedical research and clinical workflows. Here we conceptualize an IR-optical hybrid (IR-OH) approach that sensitively measures molecular composition based on an optical microscope with wide-field interferometric detection of absorption-induced sample expansion. We demonstrate that IR-OH exceeds state-of-the-art IR microscopy in coverage (10-fold), spatial resolution (fourfold), and spectral consistency (by mitigating the effects of scattering). The combined impact of these advances allows full slide infrared absorption images of unstained breast tissue sections on a visible microscope platform. We further show that automated histopathologic segmentation and generation of computationally stained (stainless) images is possible, resolving morphological features in both color and spatial detail comparable to current pathology protocols but without stains or human interpretation. IR-OH is compatible with clinical and research pathology practice and could make for a cost-effective alternative to conventional stain-based protocols for stainless, all-digital pathology.

Is Infrared Spectroscopy Ready for the Clinic?

Fourier transform-infrared spectroscopy (FT-IR) represents an attractive molecular diagnostic modality for translation to the clinic, where comprehensive chemical profiling of biological samples may revolutionize a myriad of pathways in clinical settings. Principally, FT-IR provides a rapid, cost-effective platform to obtain a molecular fingerprint of clinical samples based on vibrational transitions of chemical bonds upon interaction with infrared light. To date, considerable research activities have demonstrated competitive to superior performance of FT-IR strategies in comparison to conventional techniques, with particular promise for earlier, accessible disease diagnostics, thereby improving patient outcomes. However, amidst the changing healthcare landscape in times of aging populations and increased prevalence of cancer and chronic disease, routine adoption of FT-IR within clinical laboratories has remained elusive. Hence, this perspective shall outline the significant clinical potential of FT-IR diagnostics and subsequently address current barriers to translation from the perspective of all stakeholders, in the context of biofluid, histopathology, cytology, microbiology, and biomarker discovery frameworks. Thereafter, future perspectives of FT-IR for healthcare will be discussed, with consideration of recent technological advances that may facilitate future clinical translation.

Fast stimulated Raman and second harmonic generation imaging for intraoperative gastro-intestinal cancer detection

Conventional haematoxylin, eosin and saffron (HES) histopathology, currently the 'gold-standard' for pathological diagnosis of cancer, requires extensive sample preparations that are achieved within time scales that are not compatible with intra-operative situations where quick decisions must be taken. Providing to pathologists a close to real-time technology revealing tissue structures at the cellular level with HES histologic quality would provide an invaluable tool for surgery guidance with evident clinical benefit. Here, we specifically develop a stimulated Raman imaging based framework that demonstrates gastro-intestinal (GI) cancer detection of unprocessed human surgical specimens. The generated stimulated Raman histology (SRH) images combine chemical and collagen information to mimic conventional HES histopathology staining. We report excellent agreements between SRH and HES images acquire on the same patients for healthy, pre-cancerous and cancerous colon and pancreas tissue sections. We also develop a novel fast SRH imaging modality that captures at the pixel level all the information necessary to provide instantaneous SRH images. These developments pave the way for instantaneous label free GI histology in an intra-operative context.

Hyperspectral Tissue Image Segmentation Using Semi-Supervised NMF and Hierarchical Clustering

Hyperspectral imaging (HSI) of tissue samples in the mid-infrared (mid-IR) range provides spectro-chemical and tissue structure information at sub-cellular spatial resolution. Disease states can be directly assessed by analyzing the mid-IR spectra of different cell types (e.g., epithelial cells) and sub-cellular components (e.g., nuclei), provided that we can accurately classify the pixels belonging to these components. The challenge is to extract information from hundreds of noisy mid-IR bands at each pixel, where each band is not very informative in itself, making annotations of unstained tissue HSI images particularly tricky. Because the tissue structure is not necessarily identical between the two sections, only a few regions in unstained HSI image can be annotated with high confidence, even when serial (or adjacent) hematoxylin and eosin stained section is used as a visual guide. In order to completely use both labeled and unlabeled pixels in training images, we have developed an HSI pixel classification method that uses semi-supervised learning for both spectral dimension reduction and hierarchical pixel clustering. Compared to the supervised classifiers, the proposed method was able to account for the vast differences in the spectra of sub-cellular components of the same cell type and to achieve an F1 score of 71.18% on twofold cross-validation across 20 tissue images. To generate further interest in this promising modality, we have released our source code and also showed that disease classification is straightforward after HSI image segmentation.

Digital Staining of High-Definition Fourier Transform Infrared (FT-IR) Images Using Deep Learning

Histological stains, such as hematoxylin and eosin (H&E), are routinely used in clinical diagnosis and research. While these labels offer a high degree of specificity, throughput is limited by the need for multiple samples. Traditional histology stains, such as immunohistochemical labels, also rely only on protein expression and cannot quantify small molecules and metabolites that may aid in diagnosis. Finally, chemical stains and dyes permanently alter the tissue, making downstream analysis impossible. Fourier transform infrared (FT-IR) spectroscopic imaging has shown promise for label-free characterization of important tissue phenotypes and can bypass the need for many chemical labels. Fourier transform infrared classification commonly leverages supervised learning, requiring human annotation that is tedious and prone to errors. One alternative is digital staining, which leverages machine learning to map IR spectra to a corresponding chemical stain. This replaces human annotation with computer-aided alignment. Previous work relies on alignment of adjacent serial tissue sections. Since the tissue samples are not identical at the cellular level, this technique cannot be applied to high-definition FT-IR images. In this paper, we demonstrate that cellular-level mapping can be accomplished using identical samples for both FT-IR and chemical labels. In addition, higher-resolution results can be achieved using a deep convolutional neural network that integrates spatial and spectral features.

Deep learning for FTIR histology: leveraging spatial and spectral features with convolutional neural networks

Current methods for cancer detection rely on tissue biopsy, chemical labeling/staining, and examination of the tissue by a pathologist. Though these methods continue to remain the gold standard, they are non-quantitative and susceptible to human error. Fourier transform infrared (FTIR) spectroscopic imaging has shown potential as a quantitative alternative to traditional histology. However, identification of histological components requires reliable classification based on molecular spectra, which are susceptible to artifacts introduced by noise and scattering. Several tissue types, particularly in heterogeneous tissue regions, tend to confound traditional classification methods. Convolutional neural networks (CNNs) are the current state-of-the-art in image classification, providing the ability to learn spatial characteristics of images. In this paper, we demonstrate that CNNs with architectures designed to process both spectral and spatial information can significantly improve classifier performance over per-pixel spectral classification. We report classification results after applying CNNs to data from tissue microarrays (TMAs) to identify six major cellular and acellular constituents of tissue, namely adipocytes, blood, collagen, epithelium, necrosis, and myofibroblasts. Experimental results show that the use of spatial information in addition to the spectral information brings significant improvements in the classifier performance and allows classification of cellular subtypes, such as adipocytes, that exhibit minimal chemical information but have distinct spatial characteristics. This work demonstrates the application and efficiency of deep learning algorithms in improving the diagnostic techniques in clinical and research activities related to cancer.

SIproc: an open-source biomedical data processing platform for large hyperspectral images

There has recently been significant interest within the vibrational spectroscopy community to apply quantitative spectroscopic imaging techniques to histology and clinical diagnosis. However, many of the proposed methods require collecting spectroscopic images that have a similar region size and resolution to the corresponding histological images. Since spectroscopic images contain significantly more spectral samples than traditional histology, the resulting data sets can approach hundreds of gigabytes to terabytes in size. This makes them difficult to store and process, and the tools available to researchers for handling large spectroscopic data sets are limited. Fundamental mathematical tools, such as MATLAB, Octave, and SciPy, are extremely powerful but require that the data be stored in fast memory. This memory limitation becomes impractical for even modestly sized histological images, which can be hundreds of gigabytes in size. In this paper, we propose an open-source toolkit designed to perform out-of-core processing of hyperspectral images. By taking advantage of graphical processing unit (GPU) computing combined with adaptive data streaming, our software alleviates common workstation memory limitations while achieving better performance than existing applications.

Multimodal Chemical Analysis of the Brain by High Mass Resolution Mass Spectrometry and Infrared Spectroscopic Imaging

The brain functions through chemical interactions between many different cell types, including neurons and glia. Acquiring comprehensive information on complex, heterogeneous systems requires multiple analytical tools, each of which have unique chemical specificity and spatial resolution. Multimodal imaging generates complementary chemical information via spatially localized molecular maps, ideally from the same sample, but requires method enhancements that span from data acquisition to interpretation. We devised a protocol for performing matrix-assisted laser desorption/ionization (MALDI)-Fourier transform ion cyclotron resonance-mass spectrometry imaging (MSI), followed by infrared (IR) spectroscopic imaging on the same specimen. Multimodal measurements from the same tissue provide precise spatial alignment between modalities, enabling more advanced image processing such as image fusion and sharpening. Performing MSI first produces higher quality data from each technique compared to performing IR imaging before MSI. The difference is likely due to fixing the tissue section during MALDI matrix removal, thereby preventing analyte degradation occurring during IR imaging from an unfixed specimen. Leveraging the unique capabilities of each modality, we utilized pan sharpening of MS (mass spectrometry) ion images with selected bands from IR spectroscopy and midlevel data fusion. In comparison to sharpening with histological images, pan sharpening can employ a plethora of IR bands, producing sharpened MS images while retaining the fidelity of the initial ion images. Using Laplacian pyramid sharpening, we determine the localization of several lipids present within the hippocampus with high mass accuracy at 5 μm pixel widths. Further, through midlevel data fusion of the imaging data sets combined with k-means clustering, the combined data set discriminates between additional anatomical structures unrecognized by the individual imaging approaches. Significant differences between molecular ion abundances are detected between relevant structures within the hippocampus, such as the CA1 and CA3 regions. Our methodology provides high quality multiplex and multimodal chemical imaging of the same tissue sample, enabling more advanced data processing and analysis routines.

Simultaneous cancer and tumor microenvironment subtyping using confocal infrared microscopy for all-digital molecular histopathology

Cancer alters both the morphological and the biochemical properties of multiple cell types in a tissue. Generally, the morphology of epithelial cells is practical for routine disease diagnoses. Here, infrared spectroscopic imaging biochemically characterizes breast cancer, both epithelial cells and the tumor-associated microenvironment. Unfortunately, conventional spectral analyses are slow. Hence, we designed and built a laser confocal microscope that demonstrates a high signal-to-noise ratio for confident diagnoses. The instrument cuts down imaging time from days to minutes, making the technology feasible for research and clinical translation. Finally, automated human breast cancer biopsy imaging is reported in ∼1 hour, paving the way for routine research into the total tumor (epithelial plus microenvironment) properties and rapid, label-free diagnoses.

Histopathology based on spatial patterns of epithelial cells is the gold standard for clinical diagnoses and research in carcinomas; although known to be important, the tissue microenvironment is not readily used due to complex and subjective interpretation with existing tools. Here, we demonstrate accurate subtyping from molecular properties of epithelial cells using emerging high-definition Fourier transform infrared (HD FT-IR) spectroscopic imaging combined with machine learning algorithms. In addition to detecting four epithelial subtypes, we simultaneously delineate three stromal subtypes that characterize breast tumors. While FT-IR imaging data enable fully digital pathology with rich information content, the long spectral scanning times required for signal averaging and processing make the technology impractical for routine research or clinical use. Hence, we developed a confocal design in which refractive IR optics are designed to provide high-definition, rapid spatial scanning and discrete spectral tuning using a quantum cascade laser (QCL) source. This instrument provides simultaneously high resolving power (2-μm pixel size) and high signal-to-noise ratio (SNR) (>1,300), providing a speed increase of ∼50-fold for obtaining classified results compared with present imaging spectrometers. We demonstrate spectral fidelity and interinstrument operability of our developed instrument by accurate analysis of a 100-case breast tissue set that was analyzed in a day, considerably speeding research. Clinical breast biopsies typical of a patients’ caseload are analyzed in ∼1 hour. This study paves the way for comprehensive tumor-microenvironment analyses in feasible time periods, presenting a critical step in practical label-free molecular histopathology.

Selecting optimal features from Fourier transform infrared spectroscopy for discrete-frequency imaging

Tissue histology utilizing chemical and immunohistochemical labels plays an important role in biomedicine and disease diagnosis. Recent research suggests that mid-infrared (IR) spectroscopic imaging may augment histology by providing quantitative molecular information. One of the major barriers to this approach is long acquisition time using Fourier-transform infrared (FTIR) spectroscopy. Recent advances in discrete frequency sources, particularly quantum cascade lasers (QCLs), may mitigate this problem by allowing selective sampling of the absorption spectrum. However, DFIR imaging only provides a significant advantage when the number of spectral samples is minimized, requiring a priori knowledge of important spectral features. In this paper, we demonstrate the use of a GPU-based genetic algorithm (GA) using linear discriminant analysis (LDA) for DFIR feature selection. Our proposed method relies on pre-acquired broadband FTIR images for feature selection. Based on user-selected criteria for classification accuracy, our algorithm provides a minimal set of features that can be used with DFIR in a time-frame more practical for clinical diagnosis.

PSIP1/p75 promotes tumorigenicity in breast cancer cells by promoting the transcription of cell cycle genes

Breast cancer (BC) is a highly heterogeneous disease, both at the pathological and molecular level, and several chromatin-associated proteins play crucial roles in BC initiation and progression. Here, we demonstrate the role of PSIP1 (PC4 and SF2 interacting protein)/p75 (LEDGF) in BC progression. PSIP1/p75, previously identified as a chromatin-adaptor protein, is found to be upregulated in basal-like/triple negative breast cancer (TNBC) patient samples and cell lines. Immunohistochemistry in tissue arrays showed elevated levels of PSIP1 in metastatic invasive ductal carcinoma. Survival data analyses revealed that the levels of PSIP1 showed a negative association with TNBC patient survival. Depletion of PSIP1/p75 significantly reduced the tumorigenicity and metastatic properties of TNBC cell lines while its over-expression promoted tumorigenicity. Further, gene expression studies revealed that PSIP1 regulates the expression of genes controlling cell-cycle progression, cell migration and invasion. Finally, by interacting with RNA polymerase II, PSIP1/p75 facilitates the association of RNA pol II to the promoter of cell cycle genes and thereby regulates their transcription. Our findings demonstrate an important role of PSIP1/p75 in TNBC tumorigenicity by promoting the expression of genes that control the cell cycle and tumor metastasis.

Emerging Themes in Image Informatics and Molecular Analysis for Digital Pathology

Pathology is essential for research in disease and development, as well as for clinical decision making. For more than 100 years, pathology practice has involved analyzing images of stained, thin tissue sections by a trained human using an optical microscope. Technological advances are now driving major changes in this paradigm toward digital pathology (DP). The digital transformation of pathology goes beyond recording, archiving, and retrieving images, providing new computational tools to inform better decision making for precision medicine. First, we discuss some emerging innovations in both computational image analytics and imaging instrumentation in DP. Second, we discuss molecular contrast in pathology. Molecular DP has traditionally been an extension of pathology with molecularly specific dyes. Label-free, spectroscopic images are rapidly emerging as another important information source, and we describe the benefits and potential of this evolution. Third, we describe multimodal DP, which is enabled by computational algorithms and combines the best characteristics of structural and molecular pathology. Finally, we provide examples of application areas in telepathology, education, and precision medicine. We conclude by discussing challenges and emerging opportunities in this area.

Accurate and interpretable classification of microspectroscopy pixels using artificial neural networks

This paper addresses the problem of classifying materials from microspectroscopy at a pixel level. The challenges lie in identifying discriminatory spectral features and obtaining accurate and interpretable models relating spectra and class labels. We approach the problem by designing a supervised classifier from a tandem of Artificial Neural Network (ANN) models that identify relevant features in raw spectra and achieve high classification accuracy. The tandem of ANN models is meshed with classification rule extraction methods to lower the model complexity and to achieve interpretability of the resulting model. The contribution of the work is in designing each ANN model based on the microspectroscopy hypothesis about a discriminatory feature of a certain target class being composed of a linear combination of spectra. The novelty lies in meshing ANN and decision rule models into a tandem configuration to achieve accurate and interpretable classification results. The proposed method was evaluated using a set of broadband coherent anti-Stokes Raman scattering (BCARS) microscopy cell images (600 000  pixel-level spectra) and a reference four-class rule-based model previously created by biochemical experts. The generated classification rule-based model was on average 85% accurate measured by the DICE pixel label similarity metric, and on average 96% similar to the reference rules measured by the vector cosine metric.

Towards Translation of Discrete Frequency Infrared Spectroscopic Imaging for Digital Histopathology of Clinical Biopsy Samples

Fourier transform infrared (FT-IR) spectroscopic imaging has been widely tested as a tool for stainless digital histology of biomedical specimens, including for the identification of infiltration and fibrosis in endomyocardial biopsy samples to assess transplant rejection. A major barrier in clinical translation has been the slow speed of imaging. To address this need, we tested and report here the viability of using high speed discrete frequency infrared (DFIR) imaging to obtain stain-free biochemical imaging in cardiovascular samples collected from patients. Images obtained by this method were classified with high accuracy by a Bayesian classification algorithm trained on FT-IR imaging data as well as on DFIR data. A single spectral feature correlated with instances of fibrosis, as identified by the pathologist, highlights the advantage of the DFIR imaging approach for rapid detection. The speed of digital pathologic recognition was at least 16 times faster than the fastest FT-IR imaging instrument. These results indicate that a fast, on-site identification of fibrosis using IR imaging has potential for real time assistance during surgeries. Further, the work describes development and applications of supervised classifiers on DFIR imaging data, comparing classifiers developed on FT-IR and DFIR imaging modalities and identifying specific spectral features for accurate identification of fibrosis. This addresses a topic of much debate on the use of training data and cross-modality validity of IR measurements. Together, the work is a step toward addressing a clinical diagnostic need at acquisition time scales that make IR imaging technology practical for medical use.

Development of a practical spatial-spectral analysis protocol for breast histopathology using Fourier transform infrared spectroscopic imaging

Breast cancer screening provides sensitive tumor identification, but low specificity implies that a vast majority of biopsies are not ultimately diagnosed as cancer. Automated techniques to evaluate biopsies can prevent errors, reduce pathologist workload and provide objective analysis. Fourier transform infrared (FT-IR) spectroscopic imaging provides both molecular signatures and spatial information that may be applicable for pathology. Here, we utilize both the spectral and spatial information to develop a combined classifier that provides rapid tissue assessment. First, we evaluated the potential of IR imaging to provide a diagnosis using spectral data alone. While highly accurate histologic [epithelium, stroma] recognition could be achieved, the same was not possible for disease [cancer, no-cancer] due to the diversity of spectral signals. Hence, we employed spatial data, developing and evaluating increasingly complex models, to detect cancers. Sub-mm tumors could be very confidently predicted as indicated by the quantitative measurement of accuracy via receiver operating characteristic (ROC) curve analyses. The developed protocol was validated with a small set and statistical performance used to develop a model that predicts study design for a large scale, definitive validation. The results of evaluation on different instruments, at higher noise levels, under a coarser spectral resolution and two sampling modes [transmission and transflection], indicate that the protocol is highly accurate under a variety of conditions. The study paves the way to validating IR imaging for rapid breast tumor detection, its statistical validation and potential directions for optimization of the speed and sampling for clinical deployment.

Third-harmonic generation imaging of breast tissue biopsies

We demonstrate for the first time the imaging of unstained breast tissue biopsies using third-harmonic generation (THG) microscopy. As a label-free imaging technique, THG microscopy is compared to phase contrast and polarized light microscopy which are standard imaging methods for breast tissues. A simple feature detection algorithm is applied to detect tumour-associated lymphocyte rich regions in unstained breast biopsy tissue and compared with corresponding regions identified by a pathologist from bright-field images of hematoxylin and eosin stained breast tissue. Our results suggest that THG imaging holds potential as a complementary technique for analysing breast tissue biopsies.

Vibrational spectroscopy and imaging for concurrent cellular trafficking of co-localized doxorubicin and deuterated phospholipid vesicles

Simultaneous tracking of nanoparticles and encapsulated payload is of great importance and visualizing their activity is arduous. Here we use vibrational spectroscopy to study the in vitro tracking of co-localized lipid nanoparticles and encapsulated drug employing a model system derived from doxorubicin-encapsulated deuterated phospholipid (dodecyl phosphocholine-d38) single tailed phospholipid vesicles.

A label-free approach by infrared spectroscopic imaging for interrogating the biochemistry of diabetic nephropathy progression

Routine histology, the current gold standard, involves staining for specific biomolecules. However, untapped biochemical information in tissue can be gathered using biochemical imaging. Infrared spectroscopy is an emerging modality that allows label-free chemical imaging to derive biochemical information (such as protein, lipids, DNA, collagen) from tissues. Here we employed this technology in order to better predict the development of diabetic nephropathy. Using human primary kidney biopsies or nephrectomies, we obtained tissue from 4 histologically normal kidneys, 4 histologically normal kidneys from diabetic subjects, and 5 kidneys with evidence of diabetic nephropathy. A biochemical signature of diabetic nephropathy was derived that enabled prediction of nephropathy based on the ratio of only 2 spectral frequencies. Nonetheless, using the entire spectrum of biochemical information, we were able to detect renal disease with near-perfect accuracy. Additionally, study of sequential protocol biopsies from 3 transplanted kidneys showed biochemical changes even prior to clinical manifestation of diabetic nephropathy. Thus, infrared imaging can identify critical biochemical alterations that precede morphologic changes, potentially allowing for earlier intervention.

Fully automated registration of vibrational microspectroscopic images in histologically stained tissue sections

Background

In recent years, hyperspectral microscopy techniques such as infrared or Raman microscopy have been applied successfully for diagnostic purposes. In many of the corresponding studies, it is common practice to measure one and the same sample under different types of microscopes. Any joint analysis of the two image modalities requires to overlay the images, so that identical positions in the sample are located at the same coordinate in both images. This step, commonly referred to as image registration, has typically been performed manually in the lack of established automated computational registration tools.

Results

We propose a corresponding registration algorithm that addresses this registration problem, and demonstrate the robustness of our approach in different constellations of microscopes. First, we deal with subregion registration of Fourier Transform Infrared (FTIR) microscopic images in whole-slide histopathological staining images. Second, we register FTIR imaged cores of tissue microarrays in their histopathologically stained counterparts, and finally perform registration of Coherent anti-Stokes Raman spectroscopic (CARS) images within histopathological staining images.

Conclusions

Our validation involves a large variety of samples obtained from colon, bladder, and lung tissue on three different types of microscopes, and demonstrates that our proposed method works fully automated and highly robust in different constellations of microscopes involving diverse types of tissue samples.

Electronic supplementary material

The online version of this article (doi:10.1186/s12859-015-0804-9) contains supplementary material, which is available to authorized users.

High definition infrared spectroscopic imaging for lymph node histopathology

Chemical imaging is a rapidly emerging field in which molecular information within samples can be used to predict biological function and recognize disease without the use of stains or manual identification. In Fourier transform infrared (FT-IR) spectroscopic imaging, molecular absorption contrast provides a large signal relative to noise. Due to the long mid-IR wavelengths and sub-optimal instrument design, however, pixel sizes have historically been much larger than cells. This limits both the accuracy of the technique in identifying small regions, as well as the ability to visualize single cells. Here we obtain data with micron-sized sampling using a tabletop FT-IR instrument, and demonstrate that the high-definition (HD) data lead to accurate identification of multiple cells in lymph nodes that was not previously possible. Highly accurate recognition of eight distinct classes - naïve and memory B cells, T cells, erythrocytes, connective tissue, fibrovascular network, smooth muscle, and light and dark zone activated B cells was achieved in healthy, reactive, and malignant lymph node biopsies using a random forest classifier. The results demonstrate that cells currently identifiable only through immunohistochemical stains and cumbersome manual recognition of optical microscopy images can now be distinguished to a similar level through a single IR spectroscopic image from a lymph node biopsy.

Extracting knowledge from chemical imaging data using computational algorithms for digital cancer diagnosis

Fourier transform infrared (FTIR) spectroscopic imaging is an emerging microscopy modality for clinical histopathologic diagnoses as well as for biomedical research. Spectral data recorded in this modality are indicative of the underlying, spatially resolved biochemical composition but need computerized algorithms to digitally recognize and transform this information to a diagnostic tool to identify cancer or other physiologic conditions. Statistical pattern recognition forms the backbone of these recognition protocols and can be used for highly accurate results. Aided by biochemical correlations with normal and diseased states and the power of modern computer-aided pattern recognition, this approach is capable of combating many standing questions of traditional histology-based diagnosis models. For example, a simple diagnostic test can be developed to determine cell types in tissue. As a more advanced application, IR spectral data can be integrated with patient information to predict risk of cancer, providing a potential road to precision medicine and personalized care in cancer treatment. The IR imaging approach can be implemented to complement conventional diagnoses, as the samples remain unperturbed and are not destroyed. Despite high potential and utility of this approach, clinical implementation has not yet been achieved due to practical hurdles like speed of data acquisition and lack of optimized computational procedures for extracting clinically actionable information rapidly. The latter problem has been addressed by developing highly efficient ways to process IR imaging data but remains one that has considerable scope for progress. Here, we summarize the major issues and provide practical considerations in implementing a modified Bayesian classification protocol for digital molecular pathology. We hope to familiarize readers with analysis methods in IR imaging data and enable researchers to develop methods that can lead to the use of this promising technique for digital diagnosis of cancer.