A Polarization-Imaging-Based Machine Learning Framework for Quantitative Pathological Diagnosis of Cervical Precancerous Lesions

Machine learning based local recurrence prediction in colorectal cancer using polarized light imaging

Significance

Current treatment for stage III colorectal cancer (CRC) patients involves surgery that may not be sufficient in many cases, requiring additional adjuvant systemic therapy. Identification of this latter cohort that is likely to recur following surgery is key to better personalized therapy selection, but there is a lack of proper quantitative assessment tools for potential clinical adoption.

Aim

The purpose of this study is to employ Mueller matrix (MM) polarized light microscopy in combination with supervised machine learning (ML) to quantitatively analyze the prognostic value of peri-tumoral collagen in CRC in relation to 5-year local recurrence (LR).

Approach

A simple MM microscope setup was used to image surgical resection samples acquired from stage III CRC patients. Various potential biomarkers of LR were derived from MM elements via decomposition and transformation operations. These were used as features by different supervised ML models to distinguish samples from patients that locally recurred 5 years later from those that did not.

Results

Using the top five most prognostic polarimetric biomarkers ranked by their relevant feature importances, the best-performing XGBoost model achieved a patient-level accuracy of 86%. When the patient pool was further stratified, 96% accuracy was achieved within a tumor-stage-III sub-cohort.

Conclusions

ML-aided polarimetric analysis of collagenous stroma may provide prognostic value toward improving the clinical management of CRC patients.

Optical Sensor System for 3D Jones Matrix Reconstruction of Optical Anisotropy Maps of Self-Assembled Polycrystalline Soft Matter Films

Our work uses a polarization matrix formalism to analyze and algorithmically represent optical anisotropy by open dehydration of blood plasma films. Analytical relations for Jones matrix reconstruction of optical birefringence maps of protein crystal networks of dehydrated biofluid films are found. A technique for 3D step-by-step measurement of the distributions of the elements of the Jones matrix or Jones matrix images (JMI) of the optically birefringent structure of blood plasma films (BPF) has been created. Correlation between JMI maps and corresponding birefringence images of dehydrated BPF and saliva films (SF) obtained from donors and prostate cancer patients was determined. Within the framework of statistical analysis of layer-by-layer optical birefringence maps, the parameters most sensitive to pathological changes in the structure of dehydrated films were found to be the central statistical moments of the 1st to 4th orders. We physically substantiated and experimentally determined the sensitivity of the method of 3D polarization scanning technique of BPF and SF preparations in the diagnosis of endometriosis of uterine tissue.

Artificial intelligence-based risk stratification, accurate diagnosis and treatment prediction in gynecologic oncology

As data-driven science, artificial intelligence (AI) has paved a promising path toward an evolving health system teeming with thrilling opportunities for precision oncology. Notwithstanding the tremendous success of oncological AI in such fields as lung carcinoma, breast tumor and brain malignancy, less attention has been devoted to investigating the influence of AI on gynecologic oncology. Hereby, this review sheds light on the ever-increasing contribution of state-of-the-art AI techniques to the refined risk stratification and whole-course management of patients with gynecologic tumors, in particular, cervical, ovarian and endometrial cancer, centering on information and features extracted from clinical data (electronic health records), cancer imaging including radiological imaging, colposcopic images, cytological and histopathological digital images, and molecular profiling (genomics, transcriptomics, metabolomics and so forth). However, there are still noteworthy challenges beyond performance validation. Thus, this work further describes the limitations and challenges faced in the real-word implementation of AI models, as well as potential solutions to address these issues.

Deep-learning-based cross-modality translation from Stokes image to bright-field contrast

Significance

Mueller matrix (MM) microscopy has proven to be a powerful tool for probing microstructural characteristics of biological samples down to subwavelength scale. However, in clinical practice, doctors usually rely on bright-field microscopy images of stained tissue slides to identify characteristic features of specific diseases and make accurate diagnosis. Cross-modality translation based on polarization imaging helps to improve the efficiency and stability in analyzing sample properties from different modalities for pathologists.

Aim

In this work, we propose a computational image translation technique based on deep learning to enable bright-field microscopy contrast using snapshot Stokes images of stained pathological tissue slides. Taking Stokes images as input instead of MM images allows the translated bright-field images to be unaffected by variations of light source and samples.

Approach

We adopted CycleGAN as the translation model to avoid requirements on co-registered image pairs in the training. This method can generate images that are equivalent to the bright-field images with different staining styles on the same region.

Results

Pathological slices of liver and breast tissues with hematoxylin and eosin staining and lung tissues with two types of immunohistochemistry staining, i.e., thyroid transcription factor-1 and Ki-67, were used to demonstrate the effectiveness of our method. The output results were evaluated by four image quality assessment methods.

Conclusions

By comparing the cross-modality translation performance with MM images, we found that the Stokes images, with the advantages of faster acquisition and independence from light intensity and image registration, can be well translated to bright-field images.

Harnessing the power of optical microscopy for visualization and analysis of histopathological images

Histopathology is the foundation and gold standard for identifying diseases, and precise quantification of histopathological images can provide the pathologist with objective clues to make a more convincing diagnosis. Optical microscopy (OM), an important branch of optical imaging technology that provides high-resolution images of tissue cytology and structural morphology, has been used in the diagnosis of histopathology and evolved into a new disciplinary direction of optical microscopic histopathology (OMH). There are a number of ex-vivo studies providing applicability of different OMH approaches, and a transfer of these techniques toward in vivo diagnosis is currently in progress. Furthermore, combined with advanced artificial intelligence algorithms, OMH allows for improved diagnostic reliability and convenience due to the complementarity of retrieval information. In this review, we cover recent advances in OMH, including the exploration of new techniques in OMH as well as their applications, and look ahead to new challenges in OMH. These typical application examples well demonstrate the application potential and clinical value of OMH techniques in histopathological diagnosis.

A generalized AI method for pathology cancer diagnosis and prognosis prediction based on transfer learning and hierarchical split

Objective.This study aims to propose a generalized AI method for pathology cancer diagnosis and prognosis prediction based on transfer learning and hierarchical split.Approach.We present a neural network framework for cancer diagnosis and prognosis prediction in pathological images. To enhance the network's depth and width, we employ a hierarchical split block (HS-Block) to create an AI-aided diagnosis system suitable for semi-supervised clinical settings with limited labeled samples and cross-domain tasks. By incorporating a lightweight convolution unit based on the HS-Block, we improve the feature information extraction capabilities of a regular network (RegNet). Additionally, we integrate a Convolutional Block Attention Module into the first and last convolutions to optimize the extraction of global features and local details. To address limited sample labels, we employ a dual-transfer learning (DTL) mechanism named DTL-HS-Regnet, enabling semi-supervised learning in clinical settings.Main results.Our proposed DTL-HS-Regnet model outperforms other advanced deep-learning models in three different types of cancer diagnosis tasks. It demonstrates superior feature extraction ability, achieving an average sensitivity, specificity, accuracy, and F1 score of 0.9987, 1.0000, 1.0000 and 0.9992, respectively. Furthermore, we evaluate the model's capability to directly extract prognosis prediction information from pathological images by constructing patient cohorts. The results show that the correlation between DTL-HS-Regnet predictions and the presence of cancer-associated fibroblasts is comparable to that of pathologists.Significance.Our proposed AI method offers a generalized approach for cancer diagnosis and prognosis prediction in pathology. The outstanding performance of the DTL-HS-Regnet model demonstrates its potential for improving current practices in image digital pathology, expanding the boundaries of cancer treatment in two critical areas.

Joint Diagnostic Method of Tumor Tissue Based on Hyperspectral Spectral-Spatial Transfer Features

In order to improve the clinical application of hyperspectral technology in the pathological diagnosis of tumor tissue, a joint diagnostic method based on spectral-spatial transfer features was established by simulating the actual clinical diagnosis process and combining micro-hyperspectral imaging with large-scale pathological data. In view of the limited sample volume of medical hyperspectral data, a multi-data transfer model pre-trained on conventional pathology datasets was applied to the classification task of micro-hyperspectral images, to explore the differences in spectral-spatial transfer features in the wavelength of 410-900 nm between tumor tissues and normal tissues. The experimental results show that the spectral-spatial transfer convolutional neural network (SST-CNN) achieved a classification accuracy of 95.46% for the gastric cancer dataset and 95.89% for the thyroid cancer dataset, thus outperforming models trained on single conventional digital pathology and single hyperspectral data. The joint diagnostic method established based on SST-CNN can complete the interpretation of a section of data in 3 min, thus providing a new technical solution for the rapid diagnosis of pathology. This study also explored problems involving the correlation between tumor tissues and typical spectral-spatial features, as well as the efficient transformation of conventional pathological and transfer spectral-spatial features, which solidified the theoretical research on hyperspectral pathological diagnosis.

Mueller matrix imaging of pathological slides with plastic coverslips

Mueller matrix microscopy is capable of polarization characterization of pathological samples and polarization imaging based digital pathology. In recent years, hospitals are replacing glass coverslips with plastic coverslips for automatic preparations of dry and clean pathological slides with less slide-sticking and air bubbles. However, plastic coverslips are usually birefringent and introduce polarization artifacts in Mueller matrix imaging. In this study, a spatial frequency based calibration method (SFCM) is used to remove such polarization artifacts. The polarization information of the plastic coverslips and the pathological tissues are separated by the spatial frequency analysis, then the Mueller matrix images of pathological tissues are restored by matrix inversions. By cutting two adjacent lung cancer tissue slides, we prepare paired samples of very similar pathological structures but one with a glass coverslip and the other with a plastic coverslip. Comparisons between Mueller matrix images of the paired samples show that SFCM can effectively remove the artifacts due to plastic coverslip.

Dual Polarization Modality Fusion Network for Assisting Pathological Diagnosis

Polarization imaging is sensitive to sub-wavelength microstructures of various cancer tissues, providing abundant optical characteristics and microstructure information of complex pathological specimens. However, how to reasonably utilize polarization information to strengthen pathological diagnosis ability remains a challenging issue. In order to take full advantage of pathological image information and polarization features of samples, we propose a dual polarization modality fusion network (DPMFNet), which consists of a multi-stream CNN structure and a switched attention fusion module for complementarily aggregating the features from different modality images. Our proposed switched attention mechanism could obtain the joint feature embeddings by switching the attention map of different modality images to improve their semantic relatedness. By including a dual-polarization contrastive training scheme, our method can synthesize and align the interaction and representation of two polarization features. Experimental evaluations on three cancer datasets show the superiority of our method in assisting pathological diagnosis, especially in small datasets and low imaging resolution cases. Grad-CAM visualizes the important regions of the pathological images and the polarization images, indicating that the two modalities play different roles and allow us to give insightful corresponding explanations and analysis on cancer diagnosis conducted by the DPMFNet. This technique has potential to facilitate the performance of pathological aided diagnosis and broaden the current digital pathology boundary based on pathological image features.

A generalized framework of feature learning enhanced convolutional neural network for pathology-image-oriented cancer diagnosis

In this paper, a feature learning enhanced convolutional neural network (FLE-CNN) is proposed for cancer detection from histopathology images. To build a highly generalized computer-aided diagnosis (CAD) system, an information refinement unit employing depth- and point-wise convolutions is meticulously designed, where a dual-domain attention mechanism is adopted to focus primarily on the important areas. By deploying a residual fusion unit, context information is further integrated to extract highly discriminative features with strong representation ability. Experimental results demonstrate the merits of the proposed FLE-CNN in terms of feature extraction, which has achieved average sensitivity, specificity, precision, accuracy and F1 score of 0.9992, 0.9998, 0.9992, 0.9997 and 0.9992 in a five-class cancer detection task, and in comparison to some other advanced deep learning models, above indicators have been improved by 1.23%, 0.31%, 1.24%, 0.5% and 1.26%, respectively. Moreover, the proposed FLE-CNN provides satisfactory results in three important diagnosis, which further validates that FLE-CNN is a competitive CAD model with high generalization ability.

Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review

Transparency in Machine Learning (ML), often also referred to as interpretability or explainability, attempts to reveal the working mechanisms of complex models. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e., a relationship between algorithm and users. Thus, prototyping and user evaluations are critical to attaining solutions that afford transparency. Following human-centered design principles in highly specialized and high stakes domains, such as medical image analysis, is challenging due to the limited access to end users and the knowledge imbalance between those users and ML designers. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature from 2012 to 2021 in PubMed, EMBASE, and Compendex databases. We identified 2508 records and 68 articles met the inclusion criteria. Current techniques in transparent ML are dominated by computational feasibility and barely consider end users, e.g. clinical stakeholders. Despite the different roles and knowledge of ML developers and end users, no study reported formative user research to inform the design and development of transparent ML models. Only a few studies validated transparency claims through empirical user evaluations. These shortcomings put contemporary research on transparent ML at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research, we introduce the INTRPRT guideline, a design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests human-centered design principles, recommending formative user research as the first step to understand user needs and domain requirements. Following these guidelines increases the likelihood that the algorithms afford transparency and enable stakeholders to capitalize on the benefits of transparent ML.

Polarimetric biomarkers of peri-tumoral stroma can correlate with 5-year survival in patients with left-sided colorectal cancer

Using a novel variant of polarized light microscopy for high-contrast imaging and quantification of unstained histology slides, the current study assesses the prognostic potential of peri-tumoral collagenous stroma architecture in 32 human stage III colorectal cancer (CRC) patient samples. We analyze three distinct polarimetrically-derived images and their associated texture features, explore different unsupervised clustering algorithm models to group the data, and compare the resultant groupings with patient survival. The results demonstrate an appreciable total accuracy of ~ 78% with significant separation (p < 0.05) across all approaches for the binary classification of 5-year patient survival outcomes. Surviving patients preferentially belonged to Cluster 1 irrespective of model approach, suggesting similar stromal microstructural characteristics in this sub-population. The results suggest that polarimetrically-derived stromal biomarkers may possess prognostic value that could improve clinical management/treatment stratification in CRC patients.

Analyzing the Influence of Imaging Resolution on Polarization Properties of Scattering Media Obtained From Mueller Matrix

The Mueller matrix contains abundant micro- and even nanostructural information of media. Especially, it can be used as a powerful tool to characterize anisotropic structures quantitatively, such as the particle size, density, and orientation information of fibers in the sample. Compared with unpolarized microscopic imaging techniques, Mueller matrix microscopy can also obtain some essential structural information about the sample from the derived parameters images at low resolution. Here, to analyze the comprehensive effects of imaging resolution on polarization properties obtained from the Mueller matrix, we, first, measure the microscopic Mueller matrices of unstained rat dorsal skin tissue slices rich in collagen fibers using a series of magnifications or numerical aperture (NA) values of objectives. Then, the first-order moments and image texture parameters are quantified and analyzed in conjunction with the polarization parameter images. The results show that the Mueller matrix polar decomposition parameters diattenuation D, linear retardance δ, and depolarization Δ images obtained using low NA objective retain most of the structural information of the sample and can provide fast imaging speed. In addition, the scattering phase function analysis and Monte Carlo simulation based on the cylindrical scatterers reveal that the diattenuation parameter D images with different imaging resolutions are expected to be used to distinguish among the fibrous scatterers in the medium with different particle sizes. This study provides a criterion to decide which structural information can be accurately and rapidly obtained using a transmission Mueller matrix microscope with low NA objectives to assist pathological diagnosis and other applications.

Optimal Configurations of Mueller Polarimeter for Gaussian–Poisson Mixed Noise

The accuracy of the Mueller polarimeter is usually affected by Gaussian–Poisson mixed noise, and by optimizing the instrument matrices of polarization state generator and polarization state analyzer in the measurement system, the estimation variance caused by Gaussian noise can be suppressed, and the estimation variance caused by Poisson noise can be made independent of the sample. However, the optimization procedure usually targets only the numerical value of the instrument matrix without considering how to configure the measurement system to achieve the optimal instrument matrix. In this paper, we investigate how to make the measurement system optimal for different measurement systems by combining geometric optimization on the Poincaré sphere and finally propose a series of measurement configurations for different applications.

Deep learning for denoising in a Mueller matrix microscope

The Mueller matrix microscope is a powerful tool for characterizing the microstructural features of a complex biological sample. Performance of a Mueller matrix microscope usually relies on two major specifications: measurement accuracy and acquisition time, which may conflict with each other but both contribute to the complexity and expenses of the apparatus. In this paper, we report a learning-based method to improve both specifications of a Mueller matrix microscope using a rotating polarizer and a rotating waveplate polarization state generator. Low noise data from long acquisition time are used as the ground truth. A modified U-Net structured network incorporating channel attention effectively reduces the noise in lower quality Mueller matrix images obtained with much shorter acquisition time. The experimental results show that using high quality Mueller matrix data as ground truth, such a learning-based method can achieve both high measurement accuracy and short acquisition time in polarization imaging.

Polarization-based probabilistic discriminative model for quantitative characterization of cancer cells

We propose a polarization-based probabilistic discriminative model for deriving a set of new sigmoid-transformed polarimetry feature parameters, which not only enables accurate and quantitative characterization of cancer cells at pixel level, but also accomplish the task with a simple and stable model. By taking advantages of polarization imaging techniques, these parameters enable a low-magnification and wide-field imaging system to separate the types of cells into more specific categories that previously were distinctive under high magnification. Instead of blindly choosing the model, the L0 regularization method is used to obtain the simplified and stable polarimetry feature parameter. We demonstrate the model viability by using the pathological tissues of breast cancer and liver cancer, in each of which there are two derived parameters that can characterize the cells and cancer cells respectively with satisfactory accuracy and sensitivity. The stability of the final model opens the possibility for physical interpretation and analysis. This technique may bypass the typically labor-intensive and subjective tumor evaluating system, and could be used as a blueprint for an objective and automated procedure for cancer cell screening.

Polarization imaging-based radiomics approach for the staging of liver fibrosis

Mueller matrix imaging contains abundant biological microstructure information and has shown promising potential in clinical applications. Compared with the ordinary unpolarized light microscopy that relies on the spatial resolution to reveal detailed histological features, Mueller matrix imaging encodes rich information on the microstructures even at low-resolution and wide-field conditions. Accurate staging of liver fibrosis is essential for the therapeutic diagnosis and prognosis of chronic liver diseases. In the clinic, pathologists commonly use semiquantitative numerical scoring systems to determine the stages of liver fibrosis based on the visualization of stained characteristic morphological changes, which require skilled staining technicians and well-trained pathologists. A polarization imaging based quantitative diagnostic method can help to reduce the time-consuming multiple staining processes and provide quantitative information to facilitate the accurate staging of liver fibrosis. In this study, we report a polarization imaging based radiomics approach to provide quantitative diagnostic features for the staging of liver fibrosis. Comparisons between polarization image features under a 4× objective lens with H&E image features under 4×, 10×, 20×, and 40× objective lenses were performed to highlight the superiority of the high dimensional polarization image features in the characterization of the histological microstructures of liver fibrosis tissues at low-resolution and wide-field conditions.

Artificial intelligence for nuclear medicine in oncology

As in all other medical fields, artificial intelligence (AI) is increasingly being used in nuclear medicine for oncology. There are many articles that discuss AI from the viewpoint of nuclear medicine, but few focus on nuclear medicine from the viewpoint of AI. Nuclear medicine images are characterized by their low spatial resolution and high quantitativeness. It is noted that AI has been used since before the emergence of deep learning. AI can be divided into three categories by its purpose: (1) assisted interpretation, i.e., computer-aided detection (CADe) or computer-aided diagnosis (CADx). (2) Additional insight, i.e., AI provides information beyond the radiologist's eye, such as predicting genes and prognosis from images. It is also related to the field called radiomics/radiogenomics. (3) Augmented image, i.e., image generation tasks. To apply AI to practical use, harmonization between facilities and the possibility of black box explanations need to be resolved.