Estimation of Tissue Optical Parameters with Hyperspectral Imaging and Spectral Unmixing

Detection and margin assessment of thyroid carcinoma with microscopic hyperspectral imaging using transformer networks

Significance

Hyperspectral imaging (HSI) is an emerging imaging modality for oncological applications and can improve cancer detection with digital pathology.

Aim

The study aims to highlight the increased accuracy and sensitivity of detecting the margin of thyroid carcinoma in hematoxylin and eosin (H&E)-stained histological slides using HSI and data augmentation methods.

Approach

Using an automated microscopic imaging system, we captured 2599 hyperspectral images from 65 H&E-stained human thyroid slides. Images were then preprocessed into 153,906 image patches of dimension 250 × 250 × 84   pixels . We modified the TimeSformer network architecture, which used alternating spectral attention and spatial attention layers. We implemented several data augmentation methods for HSI based on the RandAugment algorithm. We compared the performances of TimeSformer on HSI against the performances of pretrained ConvNext and pretrained vision transformers (ViT) networks on red, green, and blue (RGB) images. Finally, we applied attention unrolling techniques on the trained TimeSformer network to identify the biological features to which the network paid attention.

Results

In the testing dataset, TimeSformer achieved an accuracy of 90.87%, a weightedF 1 score of 89.79%, a sensitivity of 91.50%, and an area under the receiving operator characteristic curve (AU-ROC) score of 97.04%. Additionally, TimeSformer produced thyroid carcinoma tumor margins with an average Jaccard score of 0.76 mm. Without data augmentation, TimeSformer achieved an accuracy of 88.23%, a weightedF 1 score of 86.46%, a sensitivity of 85.53%, and an AU-ROC score of 94.94%. In comparison, the ViT network achieved an 89.98% accuracy, an 88.14% weightedF 1 score, an 84.77% sensitivity, and a 96.17% AU-ROC. Our visualization results showed that the network paid attention to biological features.

Conclusions

The TimeSformer model trained with hyperspectral histological data consistently outperformed conventional RGB-based models, highlighting the superiority of HSI in this context. Our proposed augmentation methods improved the accuracy, theF 1 score, and the sensitivity score.

Learnable real-time inference of molecular composition from diffuse spectroscopy of brain tissue

Significance

Diffuse optical modalities such as broadband near-infrared spectroscopy (bNIRS) and hyperspectral imaging (HSI) represent a promising alternative for low-cost, non-invasive, and fast monitoring of living tissue. Particularly, the possibility of extracting the molecular composition of the tissue from the optical spectra deems the spectroscopy techniques as a unique diagnostic tool.

Aim

No established method exists to streamline the inference of the biochemical composition from the optical spectrum for real-time applications such as surgical monitoring. We analyze a machine learning technique for inference of changes in the molecular composition of brain tissue.

Approach

We propose modifications to the existing learnable methodology based on the Beer-Lambert law. We evaluate the method's applicability to linear and nonlinear formulations of this physical law. The approach is tested on data obtained from the bNIRS- and HSI-based monitoring of brain tissue.

Results

The results demonstrate that the proposed method enables real-time molecular composition inference while maintaining the accuracy of traditional methods. Preliminary findings show that Beer-Lambert law-based spectral unmixing allows contrasting brain anatomy semantics such as the vessel tree and tumor area.

Conclusion

We present a data-driven technique for inferring molecular composition change from diffuse spectroscopy of brain tissue, potentially enabling intra-operative monitoring.

An Ensemble Learning Method for Detection of Head and Neck Squamous Cell Carcinoma Using Polarized Hyperspectral Microscopic Imaging

Head and neck squamous cell carcinoma (HNSCC) has a high mortality rate. In this study, we developed a Stokes-vector-derived polarized hyperspectral imaging (PHSI) system for H&E-stained pathological slides with HNSCC and built a dataset to develop a deep learning classification method based on convolutional neural networks (CNN). We use our polarized hyperspectral microscope to collect the four Stokes parameter hypercubes (S0, S1, S2, and S3) from 56 patients and synthesize pseudo-RGB images using a transformation function that approximates the human eye's spectral response to visual stimuli. Each image is divided into patches. Data augmentation is applied using rotations and flipping. We create a four-branch model architecture where each branch is trained on one Stokes parameter individually, then we freeze the branches and fine-tune the top layers of our model to generate final predictions. Our results show high accuracy, sensitivity, and specificity, indicating that our model performed well on our dataset. Future works can improve upon these results by training on more varied data, classifying tumors based on their grade, and introducing more recent architectural techniques.

Polarized Hyperspectral Microscopic Imaging for White Blood Cells on Wright-Stained Blood Smear Slides

White blood cells, also called leukocytes, are hematopoietic cells of the immune system that are involved in protecting the body against both infectious diseases and foreign materials. The abnormal development and uncontrolled proliferation of these cells can lead to devastating cancers. Their timely recognition in the peripheral blood is critical to diagnosis and treatment. In this study, we developed a microscopic imaging system for improving the visualization of white blood cells on Wright's stained blood smear slides, with two different setups: polarized light imaging and polarized hyperspectral imaging. Based on the polarized light imaging setup, we collected the RGB images of Stokes vector parameters (S0, S1, S2, and S3) of five types of white blood cells (neutrophil, eosinophil, basophil, lymphocyte, and monocyte), and calculated the Stokes vector derived parameters: the degree of polarization (DOP), the degree of linear polarization (DOLP), and the degree of circular polarization (DOCP)). We also calculated Stokes vector data based on the polarized hyperspectral imaging setup. The preliminary results demonstrate that Stokes vector derived parameters (DOP, DOLP, and DOCP) could improve the visualization of granules in granulocytes (neutrophils, eosinophils, and basophils). Furthermore, Stokes vector derived parameters (DOP, DOLP, and DOCP) could improve the visualization of surface structures (protein patterns) of lymphocytes enabling subclassification of lymphocyte subpopulations. Finally, S2, S3, and DOCP could enhance the morphologic visualization of monocyte nucleus. We also demonstrated that the polarized hyperspectral imaging setup could provide complementary spectral information to the spatial information on different Stokes vector parameters of white blood cells. This work demonstrates that polarized light imaging & polarized hyperspectral imaging has the potential to become a strong imaging tool in the diagnosis of disorders arising from white blood cells.

Polarized hyperspectral microscopic imaging for collagen visualization on pathologic slides of head and neck squamous cell carcinoma

We developed a polarized hyperspectral microscope to collect four types of Stokes vector data cubes (S0, S1, S2, and S3) of the pathologic slides with head and neck squamous cell carcinoma (HNSCC). Our system consists of an optical light microscope with a movable stage, two polarizers, two liquid crystal variable retarders (LCVRs), and a SnapScan hyperspectral camera. The polarizers and LCVRs work in tandem with the hyperspectral camera to acquire polarized hyperspectral images. Synthetic pseudo-RGB images are generated from the four Stokes vector data cubes based on a transformation function similar to the spectral response of human eye for the visualization of hyperspectral images. Collagen is the most abundant extracellular matrix (ECM) protein in the human body. A major focus of studying the ECM in tumor microenvironment is the role of collagen in both normal and abnormal function. Collagen tends to accumulate in and around tumors during cancer development and growth. In this study, we acquired images from normal regions containing normal cells and collagen fibers and from tumor regions containing cancerous squamous cells and collagen fibers on HNSCC pathologic slides. The preliminary results demonstrated that our customized polarized hyperspectral microscope is able to improve the visualization of collagen on HNSCC pathologic slides under different situations, including thick fibers of normal stroma, thin fibers of normal stroma, fibers of normal muscle cells, fibers accumulated in tumors, fibers accumulated around tumors. Our preliminary results also demonstrated that the customized polarized hyperspectral microscope is capable of extracting the spectral signatures of collagen based on Stokes vector parameters and can have various applications in pathology and oncology.

Automatic detection of head and neck squamous cell carcinoma on pathologic slides using polarized hyperspectral imaging and deep learning

The study is to incorporate polarized hyperspectral imaging (PHSI) with deep learning for automatic detection of head and neck squamous cell carcinoma (SCC) on hematoxylin and eosin (H&E) stained tissue slides. A polarized hyperspectral imaging microscope had been developed in our group. In this paper, we firstly collected the Stokes vector data cubes (S0, S1, S2, and S3) of histologic slides from 17 patients with SCC by the PHSI microscope, under the wavelength range from 467 nm to 750 nm. Secondly, we generated the synthetic RGB images from the original Stokes vector data cubes. Thirdly, we cropped the synthetic RGB images into image patches at the image size of 96×96 pixels, and then set up a ResNet50-based convolutional neural network (CNN) to classify the image patches of the four Stokes vector parameters (S0, S1, S2, and S3) by application of transfer learning. To test the performances of the model, each time we trained the model based on the image patches (S0, S1, S2, and S3) of 16 patients out of 17 patients, and used the trained model to calculate the testing accuracy based on the image patches of the rest 1 patient (S0, S1, S2, and S3). We repeated the process for 6 times and obtained 24 testing accuracies (S0, S1, S2, and S3) from 6 different patients out of the 17 patients. The preliminary results showed that the average testing accuracy (84.2%) on S3 outperformed the average testing accuracy (83.5%) on S0. Furthermore, 4 of 6 testing accuracies of S3 (96.0%, 87.3%, 82.8%, and 86.7%) outperformed the testing accuracies of S0 (93.3%, 85.2%, 80.2%, and 79.0%). The study demonstrated the potential of using polarized hyperspectral imaging and deep learning for automatic detection of head and neck SCC on pathologic slides.

Automatic detection of head and neck squamous cell carcinoma on pathologic slides using polarized hyperspectral imaging and machine learning

The aim of this study is to incorporate polarized hyperspectral imaging (PHSI) with machine learning for automatic detection of head and neck squamous cell carcinoma (SCC) on hematoxylin and eosin (H&E) stained tissue slides. A polarized hyperspectral imaging microscope had been developed in our group. In this paper, we imaged 20 H&E stained tissue slides from 10 patients with SCC of the larynx by the PHSI microscope. Several machine learning algorithms, including support vector machine (SVM), random forest, Gaussian naive Bayes, and logistic regression, were applied to the collected image data for the automatic detection of SCC on the H&E stained tissue slides. The performance of these methods was compared among the collected PHSI data, the pseudo-RGB images generated from the PHSI data, and the PHSI data after applying the principal component analysis (PCA) transformation. The results suggest that SVM is a superior classifier for the classification task based on the PHSI data cubes compared to the other three classifiers. The incorporate of four Stokes vector parameters improved the classification accuracy. Finally, the PCA transformed image data did not improve the accuracy as it might lose some important information from the original PHSI data. The preliminary results show that polarized hyperspectral imaging can have many potential applications in digital pathology.

Classification of Hyperspectral In Vivo Brain Tissue Based on Linear Unmixing

Hyperspectral imaging is a multidimensional optical technique with the potential of providing fast and accurate tissue classification. The main challenge is the adequate processing of the multidimensional information usually linked to long processing times and significant computational costs, which require expensive hardware. In this study, we address the problem of tissue classification for intraoperative hyperspectral images of in vivo brain tissue. For this goal, two methodologies are introduced that rely on a blind linear unmixing (BLU) scheme for practical tissue classification. Both methodologies identify the characteristic end-members related to the studied tissue classes by BLU from a training dataset and classify the pixels by a minimum distance approach. The proposed methodologies are compared with a machine learning method based on a supervised support vector machine (SVM) classifier. The methodologies based on BLU achieve speedup factors of ~459× and ~429× compared to the SVM scheme, while keeping constant and even slightly improving the classification performance.

Assessment of carboxyhemoglobin content in the blood with high accuracy: wavelength range optimization for nonlinear curve fitting of optical spectra

The impact of carbon monoxide (CO) gas on the human organism is very dangerous. The affinity of CO to hemoglobin is considerably higher than that of oxygen. Thus, the interaction of CO with the blood results in a higher content of carboxyhemoglobin (HbCO) in red blood cells (RBCs) and correspondingly in tissue hypoxia. The disruption in the organism depends on the HbCO content in the blood. To assess any complications in the body at a given moment due to CO exposure and predict future consequences, it is necessary to measure the dynamics of hemoglobin derivative concentrations simultaneously. However, measuring HbCO and other derivatives in RBCs without hemolysis accurately is complicated due to the strong intercollinearity between the molar absorptivities of hemoglobin derivatives and superposition of absorption and scattering spectra. In the present study, to quantitatively assess the contents of the hemoglobin derivatives in the blood after exposure to CO, improved accuracy is achieved by optimizing the wavelength range used for the nonlinear curve fitting of optical spectra. Experimental spectra were measured in the wavelength range Δλ=500−700nm. For each experimental curve, it was established the value of optimal interval Δλopt for which the correlation coefficient between experimental data and corresponding points of the theoretical fitting curve was the maximum in the wavelength range Δλtyp=535−580nm, which contains the typical absorption peaks for HbO2, Hb, and HbCO. The concentrations obtained based on such fitting curves were considered to be highly accurate. The quantitative assessment enabled the determination of theHbCO nonlinear increase with the time of CO exposure in the in vitro experiment and the study of the dynamics of hemoglobin derivative transformations during blood incubation.

Development of a new polarized hyperspectral imaging microscope

In this study, we proposed and designed a transmission mode polarized hyperspectral imaging microscope (PHSIM). The hyperspectral imaging (HSI) component is based on the snapscan with a hyperspectral camera. The HSI wavelength range is from 467-700 nm. Polarized light imaging is realized by the integration of two polarizers and two liquid crystal variable retarders (LCVR), which is capable of full Stokes polarimetric imaging. The new imaging device was tested for the detection of squamous cell carcinoma (SCC) in H&E stained oral tissue slides of 8 patients. One normal area and one cancerous area on each slide are selected to make the comparison. The preliminary results indicated that the spectral curves of the Stokes vector parameters (S0, S1, S2, S3) of the normal area on the H&E stained oral tissue slides are different from those of SCC in certain wavelength range. Further work is required to apply the new polarized hyperspectral imaging microscope to a large number of patient samples and to test the PHSIM system in different cancer types.

Retinal oximetry and fractal analysis of capillary maps in sickle cell disease patients and matched healthy volunteers

PURPOSE

Fractal analysis can be used to quantitatively analyze the retinal microvasculature and might be a suitable method to quantify retinal capillary changes in sickle cell disease (SCD) patients. Retinal oximetry measurements might function as a proxy for the pathophysiology of cerebrovascular diseases. Moreover, hypoxia has an important role in the pathophysiology of diabetic and other retinopathies. However, little is known about the oximetry around the macula in SCD patients. With this study, we explored the feasibility to perform these quantified measurements in SCD patients.

METHODS

Retinal microvascular and oximetry measurements were performed in eight SCD patients and eight healthy matched controls. Oximetry pictures and non-invasive capillary perfusion maps (nCPM) were obtained by the retinal function imager. Measurements were conducted twice on two different study days. Measured variables included monofractal dimension (Dbox), relative saturation, deoxygenated hemoglobin (deoxyHb), and oxygenated hemoglobin (oxyHb) concentration.

RESULTS

No statistically significant differences in vessel density were found in the different annular zones (large vessels, p = 0.66; small vessels, p = 0.66) and anatomical quadrants (large vessels, p = 0.74; small vessels, p = 0.72). Furthermore, no significant between-group differences were found in the other different anatomical quadrants and annular zones around the fovea for relative saturation levels and deoxygenated Hb. However, the oxyHb levels were significantly lower in SCD patients, compared with those in matched controls in the temporal quadrants (p = 0.04; p = 0.02) and the superior nasal quadrant (p = 0.05).

CONCLUSIONS

Our study demonstrated the feasibility of multispectral imaging to measure retinal changes in oxygenation in both SCD patients and matched volunteers. The results suggest that in SCD patients before any structural microvascular changes in the central retina are present, functional abnormalities can be observed with abnormal oximetry measurements.

Feasibility of hyperspectral analysis for discrimination of rabbit liver VX2 tumor

BACKGROUND

Hepatocellular carcinoma is one of the most common malignant tumors worldwide. Currently, the most accurate diagnosis imaging modality for hepatocellular carcinoma is enhanced magnetic resonance imaging. However, it is still difficult to distinguish cirrhosis lesions, and novel diagnosis modalities are still needed.

AIM

To investigate the feasibility of hyperspectral analysis for discrimination of rabbit liver VX2 tumor.

METHODS

In this study, a rabbit liver VX2 tumor model was established. After laparotomy, under direct view, VX2 tumor tissue and normal liver tissue were subjected to hyperspectral analysis.

RESULTS

The spectral signature of the liver tumor was clearly distinguishable from that of the normal tissue, simply from the original spectral curves. Specifically, two absorption peaks at 600-900 nm wavelength in normal tissue disappeared but a new reflection peak appeared in the tumor. The average optical reflection at the whole waveband of 400-1800 nm in liver tumor was higher than that of the normal tissue.

CONCLUSION

Hyperspectral analysis can differentiate rabbit VX2 tumors. Further research will continue to perform hyperspectral imaging to obtain more information for differentiation of liver cancer from normal tissue.

Hyperspectral Imaging of Neoplastic Progression in a Mouse Model of Oral Carcinogenesis

Hyperspectral imaging (HSI) is an emerging modality for medical applications and holds great potential for noninvasive early detection of cancer. It has been reported that early cancer detection can improve the survival and quality of life of head and neck cancer patients. In this paper, we explored the possibility of differentiating between premalignant lesions and healthy tongue tissue using hyperspectral imaging in a chemical induced oral cancer animal model. We proposed a novel classification algorithm for cancer detection using hyperspectral images. The method detected the dysplastic tissue with an average area under the curve (AUC) of 0.89. The hyperspectral imaging and classification technique may provide a new tool for oral cancer detection.

A Minimum Spanning Forest-Based Method for Noninvasive Cancer Detection With Hyperspectral Imaging

GOAL

The purpose of this paper is to develop a classification method that combines both spectral and spatial information for distinguishing cancer from healthy tissue on hyperspectral images in an animal model.

METHODS

An automated algorithm based on a minimum spanning forest (MSF) and optimal band selection has been proposed to classify healthy and cancerous tissue on hyperspectral images. A support vector machine classifier is trained to create a pixel-wise classification probability map of cancerous and healthy tissue. This map is then used to identify markers that are used to compute mutual information for a range of bands in the hyperspectral image and thus select the optimal bands. An MSF is finally grown to segment the image using spatial and spectral information.

CONCLUSION

The MSF based method with automatically selected bands proved to be accurate in determining the tumor boundary on hyperspectral images.

SIGNIFICANCE

Hyperspectral imaging combined with the proposed classification technique has the potential to provide a noninvasive tool for cancer detection.

Framework for hyperspectral image processing and quantification for cancer detection during animal tumor surgery

Hyperspectral imaging (HSI) is an imaging modality that holds strong potential for rapid cancer detection during image-guided surgery. But the data from HSI often needs to be processed appropriately in order to extract the maximum useful information that differentiates cancer from normal tissue. We proposed a framework for hyperspectral image processing and quantification, which includes a set of steps including image preprocessing, glare removal, feature extraction, and ultimately image classification. The framework has been tested on images from mice with head and neck cancer, using spectra from 450- to 900-nm wavelength. The image analysis computed Fourier coefficients, normalized reflectance, mean, and spectral derivatives for improved accuracy. The experimental results demonstrated the feasibility of the hyperspectral image processing and quantification framework for cancer detection during animal tumor surgery, in a challenging setting where sensitivity can be low due to a modest number of features present, but potential for fast image classification can be high. This HSI approach may have potential application in tumor margin assessment during image-guided surgery, where speed of assessment may be the dominant factor.