Automated classification of hepatocellular carcinoma differentiation using multiphoton microscopy and deep learning

Label-free imaging diagnosis and collagen-optical evaluation of endometrioid adenocarcinoma with multiphoton microscopy

The assessment of tumor grade and pathological stage plays a pivotal role in determining the treatment strategy and predicting the prognosis of endometrial cancer. In this study, we employed multiphoton microscopy (MPM) to establish distinctive optical pathological signatures specific to endometrioid adenocarcinoma (EAC), while also assessing the diagnostic sensitivity, specificity, and accuracy of MPM for this particular malignancy. The MPM technique exhibits robust capability in discriminating between benign hyperplasia and various grades of cancer tissue, with statistically significant differences observed in nucleocytoplasmic ratio and second harmonic generation/two-photon excited fluorescence intensity. Moreover, by utilizing semi-automated image analysis, we identified notable disparities in six collagen signatures between benign and malignant endometrial stroma. Our study demonstrates that MPM can differentiate between benign endometrial hyperplasia and EAC without labels, while also quantitatively assessing changes in the tumor microenvironment by analyzing collagen signatures in the endometrial stromal tissue.

Review of the application of the most current sophisticated image processing methods for the skin cancer diagnostics purposes

This paper presents the most current and innovative solutions applying modern digital image processing methods for the purpose of skin cancer diagnostics. Skin cancer is one of the most common types of cancers. It is said that in the USA only, one in five people will develop skin cancer and this trend is constantly increasing. Implementation of new, non-invasive methods plays a crucial role in both identification and prevention of skin cancer occurrence. Early diagnosis and treatment are needed in order to decrease the number of deaths due to this disease. This paper also contains some information regarding the most common skin cancer types, mortality and epidemiological data for Poland, Europe, Canada and the USA. It also covers the most efficient and modern image recognition methods based on the artificial intelligence applied currently for diagnostics purposes. In this work, both professional, sophisticated as well as inexpensive solutions were presented. This paper is a review paper and covers the period of 2017 and 2022 when it comes to solutions and statistics. The authors decided to focus on the latest data, mostly due to the rapid technology development and increased number of new methods, which positively affects diagnosis and prognosis.

The Use of Artificial Intelligence in the Liver Histopathology Field: A Systematic Review

Digital pathology (DP) has begun to play a key role in the evaluation of liver specimens. Recent studies have shown that a workflow that combines DP and artificial intelligence (AI) applied to histopathology has potential value in supporting the diagnosis, treatment evaluation, and prognosis prediction of liver diseases. Here, we provide a systematic review of the use of this workflow in the field of hepatology. Based on the PRISMA 2020 criteria, a search of the PubMed, SCOPUS, and Embase electronic databases was conducted, applying inclusion/exclusion filters. The articles were evaluated by two independent reviewers, who extracted the specifications and objectives of each study, the AI tools used, and the results obtained. From the 266 initial records identified, 25 eligible studies were selected, mainly conducted on human liver tissues. Most of the studies were performed using whole-slide imaging systems for imaging acquisition and applying different machine learning and deep learning methods for image pre-processing, segmentation, feature extractions, and classification. Of note, most of the studies selected demonstrated good performance as classifiers of liver histological images compared to pathologist annotations. Promising results to date bode well for the not-too-distant inclusion of these techniques in clinical practice.

Prognostic significance of collagen signatures at breast tumor boundary obtained by combining multiphoton imaging and imaging analysis

PURPOSE

Collagen features in breast tumor microenvironment is closely associated with the prognosis of patients. We aim to explore the prognostic significance of collagen features at breast tumor border by combining multiphoton imaging and imaging analysis.

METHODS

We used multiphoton microscopy (MPM) to label-freely image human breast tumor samples and then constructed an automatic classification model based on deep learning to identify collagen signatures from multiphoton images. We recognized three kinds of collagen signatures at tumor boundary (CSTB I-III) in a small-scale, and furthermore obtained a CSTB score for each patient based on the combined CSTB I-III by using the ridge regression analysis. The prognostic performance of CSTB score is assessed by the area under the receiver operating characteristic curve (AUC), Cox proportional hazard regression analysis, as well as Kaplan-Meier survival analysis.

RESULTS

As an independent prognostic factor, statistical results reveal that the prognostic performance of CSTB score is better than that of the clinical model combining three independent prognostic indicators, molecular subtype, tumor size, and lymph nodal metastasis (AUC, Training dataset: 0.773 vs. 0.749; External validation: 0.753 vs. 0.724; HR, Training dataset: 4.18 vs. 3.92; External validation: 4.98 vs. 4.16), and as an auxiliary indicator, it can greatly improve the accuracy of prognostic prediction. And furthermore, a nomogram combining the CSTB score with the clinical model is established for prognosis prediction and clinical decision making.

CONCLUSION

This standardized and automated imaging prognosticator may convince pathologists to adopt it as a prognostic factor, thereby customizing more effective treatment plans for patients.

Current Status and Analysis of Machine Learning in Hepatocellular Carcinoma

Hepatocellular carcinoma (HCC) is a common tumor. Although the diagnosis and treatment of HCC have made great progress, the overall prognosis remains poor. As the core component of artificial intelligence, machine learning (ML) has developed rapidly in the past decade. In particular, ML has become widely used in the medical field, and it has helped in the diagnosis and treatment of cancer. Different algorithms of ML have different roles in diagnosis, treatment, and prognosis. This article reviews recent research, explains the application of different ML models in HCC, and provides suggestions for follow-up research.

Toward next-generation endoscopes integrating biomimetic video systems, nonlinear optical microscopy, and deep learning

According to the World Health Organization, the proportion of the world's population over 60 years will approximately double by 2050. This progressive increase in the elderly population will lead to a dramatic growth of age-related diseases, resulting in tremendous pressure on the sustainability of healthcare systems globally. In this context, finding more efficient ways to address cancers, a set of diseases whose incidence is correlated with age, is of utmost importance. Prevention of cancers to decrease morbidity relies on the identification of precursor lesions before the onset of the disease, or at least diagnosis at an early stage. In this article, after briefly discussing some of the most prominent endoscopic approaches for gastric cancer diagnostics, we review relevant progress in three emerging technologies that have significant potential to play pivotal roles in next-generation endoscopy systems: biomimetic vision (with special focus on compound eye cameras), non-linear optical microscopies, and Deep Learning. Such systems are urgently needed to enhance the three major steps required for the successful diagnostics of gastrointestinal cancers: detection, characterization, and confirmation of suspicious lesions. In the final part, we discuss challenges that lie en route to translating these technologies to next-generation endoscopes that could enhance gastrointestinal imaging, and depict a possible configuration of a system capable of (i) biomimetic endoscopic vision enabling easier detection of lesions, (ii) label-free in vivo tissue characterization, and (iii) intelligently automated gastrointestinal cancer diagnostic.

Dictionary learning LASSO for feature selection with application to hepatocellular carcinoma grading using contrast enhanced magnetic resonance imaging

Introduction

The successful use of machine learning (ML) for medical diagnostic purposes has prompted myriad applications in cancer image analysis. Particularly for hepatocellular carcinoma (HCC) grading, there has been a surge of interest in ML-based selection of the discriminative features from high-dimensional magnetic resonance imaging (MRI) radiomics data. As one of the most commonly used ML-based selection methods, the least absolute shrinkage and selection operator (LASSO) has high discriminative power of the essential feature based on linear representation between input features and output labels. However, most LASSO methods directly explore the original training data rather than effectively exploiting the most informative features of radiomics data for HCC grading. To overcome this limitation, this study marks the first attempt to propose a feature selection method based on LASSO with dictionary learning, where a dictionary is learned from the training features, using the Fisher ratio to maximize the discriminative information in the feature.

Methods

This study proposes a LASSO method with dictionary learning to ensure the accuracy and discrimination of feature selection. Specifically, based on the Fisher ratio score, each radiomic feature is classified into two groups: the high-information and the low-information group. Then, a dictionary is learned through an optimal mapping matrix to enhance the high-information part and suppress the low discriminative information for the task of HCC grading. Finally, we select the most discrimination features according to the LASSO coefficients based on the learned dictionary.

Results and discussion

The experimental results based on two classifiers (KNN and SVM) showed that the proposed method yielded accuracy gains, compared favorably with another 5 state-of-the-practice feature selection methods.

A Deep-Learning-Computed Cancer Score for the Identification of Human Hepatocellular Carcinoma Area Based on a Six-Colour Multiplex Immunofluorescence Panel

Liver cancer is one of the most frequently diagnosed and fatal cancers worldwide, with hepatocellular carcinoma (HCC) being the most common primary liver cancer. Hundreds of studies involving thousands of patients have now been analysed across different cancer types, including HCC, regarding the effects of immune infiltrates on the prognosis of cancer patients. However, for these analyses, an unambiguous delineation of the cancer area is paramount, which is difficult due to the strong heterogeneity and considerable inter-operator variability induced by qualitative visual assessment and manual assignment. Nowadays, however, multiplex analyses allow the simultaneous evaluation of multiple protein markers, which, in conjunction with recent machine learning approaches, may offer great potential for the objective, enhanced identification of cancer areas with further in situ analysis of prognostic immune parameters. In this study, we, therefore, used an exemplary five-marker multiplex immunofluorescence panel of commonly studied markers for prognosis (CD3 T, CD4 T helper, CD8 cytotoxic T, FoxP3 regulatory T, and PD-L1) and DAPI to assess which analytical approach is best suited to combine morphological and immunohistochemical data into a cancer score to identify the cancer area that best matches an independent pathologist’s assignment. For each cell, a total of 68 individual cell features were determined, which were used as input for 4 different approaches for computing a cancer score: a correlation-based selection of individual cell features, a MANOVA-based selection of features, a multilayer perceptron, and a convolutional neural network (a U-net). Accuracy was used to evaluate performance. With a mean accuracy of 75%, the U-net was best capable of identifying the cancer area. Although individual cell features showed a strong heterogeneity between patients, the spatial representations obtained with the computed cancer scores delineate HCC well from non-cancer liver tissues. Future analyses with larger sample sizes will help to improve the model and enable direct, in-depth investigations of prognostic parameters, ultimately enabling precision medicine.

Label-free multiphoton microscopy enables histopathological assessment of colorectal liver metastases and supports automated classification of neoplastic tissue

As the state of resection margins is an important prognostic factor after extirpation of colorectal liver metastases, surgeons aim to obtain negative margins, sometimes elaborated by resections of the positive resection plane after intraoperative frozen sections. However, this is time consuming and results sometimes remain unclear during surgery. Label-free multimodal multiphoton microscopy (MPM) is an optical technique that retrieves morpho-chemical information avoiding all staining and that can potentially be performed in real-time. Here, we investigated colorectal liver metastases and hepatic tissue using a combination of three endogenous nonlinear signals, namely: coherent anti-Stokes Raman scattering (CARS) to visualize lipids, two-photon excited fluorescence (TPEF) to visualize cellular patterns, and second harmonic generation (SHG) to visualize collagen fibers. We acquired and analyzed over forty thousand MPM images of metastatic and normal liver tissue of 106 patients. The morphological information with biochemical specificity produced by MPM allowed discriminating normal liver from metastatic tissue and discerning the tumor borders on cryosections as well as formalin-fixed bulk tissue. Furthermore, automated tissue type classification with a correct rate close to 95% was possible using a simple approach based on discriminant analysis of texture parameters. Therefore, MPM has the potential to increase the precision of resection margins in hepatic surgery of metastases without prolonging surgical intervention.

Human Hepatocellular Carcinoma Classification from H&E Stained Histopathology Images with 3D Convolutional Neural Networks and Focal Loss Function

This paper proposes a new Hepatocellular Carcinoma (HCC) classification method utilizing a hyperspectral imaging system (HSI) integrated with a light microscope. Using our custom imaging system, we have captured 270 bands of hyperspectral images of healthy and cancer tissue samples with HCC diagnosis from a liver microarray slide. Convolutional Neural Networks with 3D convolutions (3D-CNN) have been used to build an accurate classification model. With the help of 3D convolutions, spectral and spatial features within the hyperspectral cube are incorporated to train a strong classifier. Unlike 2D convolutions, 3D convolutions take the spectral dimension into account while automatically collecting distinctive features during the CNN training stage. As a result, we have avoided manual feature engineering on hyperspectral data and proposed a compact method for HSI medical applications. Moreover, the focal loss function, utilized as a CNN cost function, enables our model to tackle the class imbalance problem residing in the dataset effectively. The focal loss function emphasizes the hard examples to learn and prevents overfitting due to the lack of inter-class balancing. Our empirical results demonstrate the superiority of hyperspectral data over RGB data for liver cancer tissue classification. We have observed that increased spectral dimension results in higher classification accuracy. Both spectral and spatial features are essential in training an accurate learner for cancer tissue classification.

Classification of multi-differentiated liver cancer pathological images based on deep learning attention mechanism

PURPOSE

Liver cancer is one of the most common malignant tumors in the world, ranking fifth in malignant tumors. The degree of differentiation can reflect the degree of malignancy. The degree of malignancy of liver cancer can be divided into three types: poorly differentiated, moderately differentiated, and well differentiated. Diagnosis and treatment of different levels of differentiation are crucial to the survival rate and survival time of patients. As the gold standard for liver cancer diagnosis, histopathological images can accurately distinguish liver cancers of different levels of differentiation. Therefore, the study of intelligent classification of histopathological images is of great significance to patients with liver cancer. At present, the classification of histopathological images of liver cancer with different degrees of differentiation has disadvantages such as time-consuming, labor-intensive, and large manual investment. In this context, the importance of intelligent classification of histopathological images is obvious.

METHODS

Based on the development of a complete data acquisition scheme, this paper applies the SENet deep learning model to the intelligent classification of all types of differentiated liver cancer histopathological images for the first time, and compares it with the four deep learning models of VGG16, ResNet50, ResNet_CBAM, and SKNet. The evaluation indexes adopted in this paper include confusion matrix, Precision, recall, F1 Score, etc. These evaluation indexes can be used to evaluate the model in a very comprehensive and accurate way.

RESULTS

Five different deep learning classification models are applied to collect the data set and evaluate model. The experimental results show that the SENet model has achieved the best classification effect with an accuracy of 95.27%. The model also has good reliability and generalization ability. The experiment proves that the SENet deep learning model has a good application prospect in the intelligent classification of histopathological images.

CONCLUSIONS

This study also proves that deep learning has great application value in solving the time-consuming and laborious problems existing in traditional manual film reading, and it has certain practical significance for the intelligent classification research of other cancer histopathological images.

Fusing pre-trained convolutional neural networks features for multi-differentiated subtypes of liver cancer on histopathological images

Liver cancer is a malignant tumor with high morbidity and mortality, which has a tremendous negative impact on human survival. However, it is a challenging task to recognize tens of thousands of histopathological images of liver cancer by naked eye, which poses numerous challenges to inexperienced clinicians. In addition, factors such as long time-consuming, tedious work and huge number of images impose a great burden on clinical diagnosis. Therefore, our study combines convolutional neural networks with histopathology images and adopts a feature fusion approach to help clinicians efficiently discriminate the differentiation types of primary hepatocellular carcinoma histopathology images, thus improving their diagnostic efficiency and relieving their work pressure. In this study, for the first time, 73 patients with different differentiation types of primary liver cancer tumors were classified. We performed an adequate classification evaluation of liver cancer differentiation types using four pre-trained deep convolutional neural networks and nine different machine learning (ML) classifiers on a dataset of liver cancer histopathology images with multiple differentiation types. And the test set accuracy, validation set accuracy, running time with different strategies, precision, recall and F1 value were used for adequate comparative evaluation. Proved by experimental results, fusion networks (FuNet) structure is a good choice, which covers both channel attention and spatial attention, and suppresses channel interference with less information. Meanwhile, it can clarify the importance of each spatial location by learning the weights of different locations in space, then apply it to the study of classification of multi-differentiated types of liver cancer. In addition, in most cases, the Stacking-based integrated learning classifier outperforms other ML classifiers in the classification task of multi-differentiation types of liver cancer with the FuNet fusion strategy after dimensionality reduction of the fused features by principle component analysis (PCA) features, and a satisfactory result of 72.46% is achieved in the test set, which has certain practicality.

Deep Learning-Based Universal Expert-Level Recognizing Pathological Images of Hepatocellular Carcinoma and Beyond

Background and Aims

We aim to develop a diagnostic tool for pathological-image classification using transfer learning that can be applied to diverse tumor types.

Methods

Microscopic images of liver tissue with and without hepatocellular carcinoma (HCC) were used to train and validate the classification framework based on a convolutional neural network. To evaluate the universal classification performance of the artificial intelligence (AI) framework, histological images from colorectal tissue and the breast were collected. Images for the training and validation sets were obtained from the Xiamen Hospital of Traditional Chinese Medicine, and those for the test set were collected from Zhongshan Hospital Xiamen University. The accuracy, sensitivity, and specificity values for the proposed framework were reported and compared with those of human image interpretation.

Results

In the human–machine comparisons, the sensitivity, and specificity for the AI algorithm were 98.0, and 99.0%, whereas for the human experts, the sensitivity ranged between 86.0 and 97.0%, while the specificity ranged between 91.0 and 100%. Based on transfer learning, the accuracies of the AI framework in classifying colorectal carcinoma and breast invasive ductal carcinoma were 96.8 and 96.0%, respectively.

Conclusion

The performance of the proposed AI framework in classifying histological images with HCC was comparable to the classification performance achieved by human experts, indicating that extending the proposed AI’s application to diagnoses and treatment recommendations is a promising area for future investigation.

Role of three-dimensional printing and artificial intelligence in the management of hepatocellular carcinoma: Challenges and opportunities

Hepatocellular carcinoma (HCC) constitutes the fifth most frequent malignancy worldwide and the third most frequent cause of cancer-related deaths. Currently, treatment selection is based on the stage of the disease. Emerging fields such as three-dimensional (3D) printing, 3D bioprinting, artificial intelligence (AI), and machine learning (ML) could lead to evidence-based, individualized management of HCC. In this review, we comprehensively report the current applications of 3D printing, 3D bioprinting, and AI/ML-based models in HCC management; we outline the significant challenges to the broad use of these novel technologies in the clinical setting with the goal of identifying means to overcome them, and finally, we discuss the opportunities that arise from these applications. Notably, regarding 3D printing and bioprinting-related challenges, we elaborate on cost and cost-effectiveness, cell sourcing, cell viability, safety, accessibility, regulation, and legal and ethical concerns. Similarly, regarding AI/ML-related challenges, we elaborate on intellectual property, liability, intrinsic biases, data protection, cybersecurity, ethical challenges, and transparency. Our findings show that AI and 3D printing applications in HCC management and healthcare, in general, are steadily expanding; thus, these technologies will be integrated into the clinical setting sooner or later. Therefore, we believe that physicians need to become familiar with these technologies and prepare to engage with them constructively.

The application of artificial intelligence in hepatology: A systematic review

The integration of human and artificial intelligence (AI) in medicine has only recently begun but it has already become obvious that intelligent systems can dramatically improve the management of liver diseases. Big data made it possible to envisage transformative developments of the use of AI for diagnosing, predicting prognosis and treating liver diseases, but there is still a lot of work to do. If we want to achieve the 21^st century digital revolution, there is an urgent need for specific national and international rules, and to adhere to bioethical parameters when collecting data. Avoiding misleading results is essential for the effective use of AI. A crucial question is whether it is possible to sustain, technically and morally, the process of integration between man and machine. We present a systematic review on the applications of AI to hepatology, highlighting the current challenges and crucial issues related to the use of such technologies.

Deep learning in hepatocellular carcinoma: Current status and future perspectives

Hepatocellular carcinoma (HCC) is among the leading causes of cancer incidence and death. Despite decades of research and development of new treatment options, the overall outcomes of patients with HCC continue to remain poor. There are areas of unmet need in risk prediction, early diagnosis, accurate prognostication, and individualized treatments for patients with HCC. Recent years have seen an explosive growth in the application of artificial intelligence (AI) technology in medical research, with the field of HCC being no exception. Among the various AI-based machine learning algorithms, deep learning algorithms are considered state-of-the-art techniques for handling and processing complex multimodal data ranging from routine clinical variables to high-resolution medical images. This article will provide a comprehensive review of the recently published studies that have applied deep learning for risk prediction, diagnosis, prognostication, and treatment planning for patients with HCC.

Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data

INTRODUCTION

Hepatocellular carcinoma is the prevalent primary liver cancer, a silent disease that killed 782,000 worldwide in 2018. Multimodal deep learning is the application of deep learning techniques, fusing more than one data modality as the model's input.

PURPOSE

A computer-aided diagnosis system for hepatocellular carcinoma developed with multimodal deep learning approaches could use multiple data modalities as recommended by clinical guidelines, and enhance the robustness and the value of the second-opinion given to physicians. This article describes the process of creation and evaluation of an algorithm for computer-aided diagnosis of hepatocellular carcinoma developed with multimodal deep learning techniques fusing preprocessed computed-tomography images with structured data from patient Electronic Health Records.

RESULTS

The classification performance achieved by the proposed algorithm in the test dataset was: accuracy = 86.9%, precision = 89.6%, recall = 86.9% and F-Score = 86.7%. These classification performance metrics are closer to the state-of-the-art in this area and were achieved with data modalities which are cheaper than traditional Magnetic Resonance Imaging approaches, enabling the use of the proposed algorithm by low and mid-sized healthcare institutions.

CONCLUSION

The classification performance achieved with the multimodal deep learning algorithm is higher than human specialists diagnostic performance using only CT for diagnosis. Even though the results are promising, the multimodal deep learning architecture used for hepatocellular carcinoma prediction needs more training and test processes using different datasets before the use of the proposed algorithm by physicians in real healthcare routines. The additional training aims to confirm the classification performance achieved and enhance the model's robustness.

3D convolutional neural networks for stalled brain capillary detection

Adequate blood supply is critical for normal brain function. Brain vasculature dysfunctions, including stalled blood flow in cerebral capillaries, are associated with cognitive decline and pathogenesis in Alzheimer's disease. Recent advances in imaging technology enabled generation of high-quality 3D images that can be used to visualize stalled blood vessels. However, localization of stalled vessels in 3D images is often required as the first step for downstream analysis. When performed manually, this process is tedious, time-consuming, and error-prone. Here, we describe a deep learning-based approach for automatic detection of stalled capillaries in brain images based on 3D convolutional neural networks. Our approach includes custom 3D data augmentations and a weights transfer method that re-uses weights from 2D models pre-trained on natural images for initialization of 3D networks. We used an ensemble of several 3D models to produce the winning submission to the "Clog Loss: Advance Alzheimer's Research with Stall Catchers" machine learning competition that challenged the participants with classifying blood vessels in 3D image stacks as stalled or flowing. In this setting, our approach outperformed other methods and demonstrated state-of-the-art results, achieving 85% Matthews correlation coefficient, 85% sensitivity, and 99.3% specificity. The source code for our solution is publicly available.

Genetic syndromes screening by facial recognition technology: VGG-16 screening model construction and evaluation

Background

Many genetic syndromes (GSs) have distinct facial dysmorphism, and facial gestalts can be used as a diagnostic tool for recognizing a syndrome. Facial recognition technology has advanced in recent years, and the screening of GSs by facial recognition technology has become feasible. This study constructed an automatic facial recognition model for the identification of children with GSs.

Results

A total of 456 frontal facial photos were collected from 228 children with GSs and 228 healthy children in Guangdong Provincial People's Hospital from Jun 2016 to Jan 2021. Only one frontal facial image was selected for each participant. The VGG-16 network (named after its proposal lab, Visual Geometry Group from Oxford University) was pretrained by transfer learning methods, and a facial recognition model based on the VGG-16 architecture was constructed. The performance of the VGG-16 model was evaluated by five-fold cross-validation. Comparison of VGG-16 model to five physicians were also performed. The VGG-16 model achieved the highest accuracy of 0.8860 ± 0.0211, specificity of 0.9124 ± 0.0308, recall of 0.8597 ± 0.0190, F1-score of 0.8829 ± 0.0215 and an area under the receiver operating characteristic curve of 0.9443 ± 0.0276 (95% confidence interval: 0.9210–0.9620) for GS screening, which was significantly higher than that achieved by human experts.

Conclusions

This study highlighted the feasibility of facial recognition technology for GSs identification. The VGG-16 recognition model can play a prominent role in GSs screening in clinical practice.

Analysis on the Characterization of Multiphoton Microscopy Images for Malignant Neoplastic Colon Lesion Detection under Deep Learning Methods

BACKGROUND

Colorectal cancer has a high incidence rate worldwide, with over 1.8 million new cases and 880,792 deaths in 2018. Fortunately, its early detection significantly increases the survival rate, reaching a cure rate of 90% when diagnosed at a localized stage. Colonoscopy is the gold standard technique for detection and removal of colorectal lesions with potential to evolve into cancer. When polyps are found in a patient, the current procedure is their complete removal. However, in this process, gastroenterologists cannot assure complete resection and clean margins which are given by the histopathology analysis of the removed tissue, which is performed at laboratory.

AIMS

In this paper, we demonstrate the capabilities of multiphoton microscopy (MPM) technology to provide imaging biomarkers that can be extracted by deep learning techniques to identify malignant neoplastic colon lesions and distinguish them from healthy, hyperplastic, or benign neoplastic tissue, without the need for histopathological staining.

MATERIALS AND METHODS

To this end, we present a novel MPM public dataset containing 14,712 images obtained from 42 patients and grouped into 2 classes. A convolutional neural network is trained on this dataset and a spatially coherent predictions scheme is applied for performance improvement.

RESULTS

We obtained a sensitivity of 0.8228 ± 0.1575 and a specificity of 0.9114 ± 0.0814 on detecting malignant neoplastic lesions. We also validated this approach to estimate the self-confidence of the network on its own predictions, obtaining a mean sensitivity of 0.8697 and a mean specificity of 0.9524 with the 18.67% of the images classified as uncertain.

CONCLUSIONS

This work lays the foundations for performing in vivo optical colon biopsies by combining this novel imaging technology together with deep learning algorithms, hence avoiding unnecessary polyp resection and allowing in situ diagnosis assessment.

Evolution of the liver biopsy and its future

Liver biopsies are commonly used to evaluate a wide variety of medical disorders, including neoplasms and post-transplant complications. However, its use is being impacted by improved clinical diagnosis of disorders, and non-invasive methods for evaluating liver tissue and as a result the indications of a liver biopsy have undergone major changes in the last decade. The evolution of highly effective treatments for some of the common indications for liver biopsy in the last decade (e.g., viral hepatitis B and C) has led to a decline in the number of liver biopsies in recent years. At the same time, the emergence of better technologies for histologic evaluation, tissue content analysis and genomics are among the many new and exciting developments in the field that hold great promise for the future and are going to shape the indications for a liver biopsy in the future. Recent advances in slide scanners now allow creation of "digital/virtual" slides that have image of the entire tissue section present in a slide [whole slide imaging (WSI)]. WSI can now be done very rapidly and at very high resolution, allowing its use in routine clinical practice. In addition, a variety of technologies have been developed in recent years that use different light sources and/or microscopes allowing visualization of tissues in a completely different way. One such technique that is applicable to liver specimens combines multiphoton microscopy (MPM) with advanced clearing and fluorescent stains known as Clearing Histology with MultiPhoton Microscopy (CHiMP). Although it has not yet been extensively validated, the technique has the potential to decrease inefficiency, reduce artifacts, and increase data while being readily integrable into clinical workflows. Another technology that can provide rapid and in-depth characterization of thousands of molecules in a tissue sample, including liver tissues, is matrix assisted laser desorption/ionization (MALDI) mass spectrometry. MALDI has already been applied in a clinical research setting with promising diagnostic and prognostic capabilities, as well as being able to elucidate mechanisms of liver diseases that may be targeted for the development of new therapies. The logical next step in huge data sets obtained from such advanced analysis of liver tissues is the application of machine learning (ML) algorithms and application of artificial intelligence (AI), for automated generation of diagnoses and prognoses. This review discusses the evolving role of liver biopsies in clinical practice over the decades, and describes newer technologies that are likely to have a significant impact on how they will be used in the future.

Machine Learning Aided Photonic Diagnostic System for Minimally Invasive Optically Guided Surgery in the Hepatoduodenal Area

Abdominal cancer is a widely prevalent group of tumours with a high level of mortality if diagnosed at a late stage. Although the cancer death rates have in general declined over the past few decades, the mortality from tumours in the hepatoduodenal area has significantly increased in recent years. The broader use of minimal access surgery (MAS) for diagnostics and treatment can significantly improve the survival rate and quality of life of patients after surgery. This work aims to develop and characterise an appropriate technical implementation for tissue endogenous fluorescence (TEF) and assess the efficiency of machine learning methods for the real-time diagnosis of tumours in the hepatoduodenal area. In this paper, we present the results of the machine learning approach applied to the optically guided MAS. We have elaborated tissue fluorescence approach with a fibre-optic probe to record the TEF and blood perfusion parameters during MAS in patients with cancers in the hepatoduodenal area. The measurements from the laser Doppler flowmetry (LDF) channel were used as a sensor of the tissue vitality to reduce variability in TEF data. Also, we evaluated how the blood perfusion oscillations are changed in the tumour tissue. The evaluated amplitudes of the cardiac (0.6-1.6 Hz) and respiratory (0.2-0.6 Hz) oscillations was significantly higher in intact tissues (p < 0.001) compared to the cancerous ones, while the myogenic (0.2-0.06 Hz) oscillation did not demonstrate any statistically significant difference. Our results demonstrate that a fibre-optic TEF probe accompanied with ML algorithms such as k-Nearest Neighbours or AdaBoost is highly promising for the real-time in situ differentiation between cancerous and healthy tissues by detecting the information about the tissue type that is encoded in the fluorescence spectrum. Also, we show that the detection can be supplemented and enhanced by parallel collection and classification of blood perfusion oscillations.

Classification and mutation prediction based on histopathology H&E images in liver cancer using deep learning

Hepatocellular carcinoma (HCC) is the most common subtype of liver cancer, and assessing its histopathological grade requires visual inspection by an experienced pathologist. In this study, the histopathological H&E images from the Genomic Data Commons Databases were used to train a neural network (inception V3) for automatic classification. According to the evaluation of our model by the Matthews correlation coefficient, the performance level was close to the ability of a 5-year experience pathologist, with 96.0% accuracy for benign and malignant classification, and 89.6% accuracy for well, moderate, and poor tumor differentiation. Furthermore, the model was trained to predict the ten most common and prognostic mutated genes in HCC. We found that four of them, including CTNNB1, FMN2, TP53, and ZFX4, could be predicted from histopathology images, with external AUCs from 0.71 to 0.89. The findings demonstrated that convolutional neural networks could be used to assist pathologists in the classification and detection of gene mutation in liver cancer.

Deep learning a boon for biophotonics?

This review covers original articles using deep learning in the biophotonic field published in the last years. In these years deep learning, which is a subset of machine learning mostly based on artificial neural network geometries, was applied to a number of biophotonic tasks and has achieved state-of-the-art performances. Therefore, deep learning in the biophotonic field is rapidly growing and it will be utilized in the next years to obtain real-time biophotonic decision-making systems and to analyze biophotonic data in general. In this contribution, we discuss the possibilities of deep learning in the biophotonic field including image classification, segmentation, registration, pseudostaining and resolution enhancement. Additionally, we discuss the potential use of deep learning for spectroscopic data including spectral data preprocessing and spectral classification. We conclude this review by addressing the potential applications and challenges of using deep learning for biophotonic data.

Dense-UNet: a novel multiphoton in vivo cellular image segmentation model based on a convolutional neural network

Background

Multiphoton microscopy (MPM) offers a feasible approach for the biopsy in clinical medicine, but it has not been used in clinical applications due to the lack of efficient image processing methods, especially the automatic segmentation technology. Segmentation technology is still one of the most challenging assignments of the MPM imaging technique.

Methods

The MPM imaging segmentation model based on deep learning is one of the most effective methods to address this problem. In this paper, the practicability of using a convolutional neural network (CNN) model to segment the MPM image of skin cells in vivo was explored. A set of MPM in vivo skin cells images with a resolution of 128×128 was successfully segmented under the Python environment with TensorFlow. A novel deep-learning segmentation model named Dense-UNet was proposed. The Dense-UNet, which is based on U-net structure, employed the dense concatenation to deepen the depth of the network architecture and achieve feature reuse. This model included four expansion modules (each module consisted of four down-sampling layers) to extract features.

Results

Sixty training images were taken from the dorsal forearm using a femtosecond Ti:Sa laser running at 735 nm. The resolution of the images is 128×128 pixels. Experimental results confirmed that the accuracy of Dense-UNet (92.54%) was higher than that of U-Net (88.59%), with a significantly lower loss value of 0.1681. The 90.60% Dice coefficient value of Dense-UNet outperformed U-Net by 11.07%. The F1-Score of Dense-UNet, U-Net, and Seg-Net was 93.35%, 90.02%, and 85.04%, respectively.

Conclusions

The deepened down-sampling path improved the ability of the model to capture cellular fined-detailed boundary features, while the symmetrical up-sampling path provided a more accurate location based on the test result. These results were the first time that the segmentation of MPM in vivo images had been adopted by introducing a deep CNN to bridge this gap in Dense-UNet technology. Dense-UNet has reached ultramodern performance for MPM images, especially for in vivo images with low resolution. This implementation supplies an automatic segmentation model based on deep learning for high-precision segmentation of MPM images in vivo.

Multiphoton microscopy of the dermoepidermal junction and automated identification of dysplastic tissues with deep learning

Histopathological image analysis performed by a trained expert is currently regarded as the gold-standard for the diagnostics of many pathologies, including cancers. However, such approaches are laborious, time consuming and contain a risk for bias or human error. There is thus a clear need for faster, less intrusive and more accurate diagnostic solutions, requiring also minimal human intervention. Multiphoton microscopy (MPM) can alleviate some of the drawbacks specific to traditional histopathology by exploiting various endogenous optical signals to provide virtual biopsies that reflect the architecture and composition of tissues, both in-vivo or ex-vivo. Here we show that MPM imaging of the dermoepidermal junction (DEJ) in unstained fixed tissues provides useful cues for a histopathologist to identify the onset of non-melanoma skin cancers. Furthermore, we show that MPM images collected on the DEJ, besides being easy to interpret by a trained specialist, can be automatically classified into healthy and dysplastic classes with high precision using a Deep Learning method and existing pre-trained convolutional neural networks. Our results suggest that deep learning enhanced MPM for in-vivo skin cancer screening could facilitate timely diagnosis and intervention, enabling thus more optimal therapeutic approaches.

Deep-UV excitation fluorescence microscopy for detection of lymph node metastasis using deep neural network

Deep-UV (DUV) excitation fluorescence microscopy has potential to provide rapid diagnosis with simple technique comparing to conventional histopathology based on hematoxylin and eosin (H&E) staining. We established a fluorescent staining protocol for DUV excitation fluorescence imaging that has enabled clear discrimination of nucleoplasm, nucleolus, and cytoplasm. Fluorescence images of metastasis-positive/-negative lymph nodes of gastric cancer patients were used for patch-based training with a deep neural network (DNN) based on Inception-v3 architecture. The performance on small patches of the fluorescence images was comparable with that of H&E images. Gradient-weighted class activation mapping analysis revealed the areas where the trained model identified metastatic lesions in the images containing cancer cells. We extended the method to large-size image analysis enabling accurate detection of metastatic lesions. We discuss usefulness of DUV excitation fluorescence imaging with the aid of DNN analysis, which is promising for assisting pathologists in assessment of lymph node metastasis.

Recent advances in multiphoton microscopy combined with nanomaterials in the field of disease evolution and clinical applications to liver cancer

Multiphoton microscopy (MPM) is expected to become a powerful clinical tool, with its unique advantages of being label-free, high resolution, deep imaging depth, low light photobleaching and low phototoxicity. Nanomaterials, with excellent physical and chemical properties, are biocompatible and easy to prepare and functionalize. The addition of nanomaterials exactly compensates for some defects of MPM, such as the weak endogenous signal strength, limited imaging materials, insufficient imaging depth and lack of therapeutic effects. Therefore, combining MPM with nanomaterials is a promising biomedical imaging method. Here, we mainly review the principle of MPM and its application in liver cancer, especially in disease evolution and clinical applications, including monitoring tumor progression, diagnosing tumor occurrence, detecting tumor metastasis, and evaluating cancer therapy response. Then, we introduce the latest advances in the combination of MPM with nanomaterials, including the MPM imaging of gold nanoparticles (AuNPs) and carbon dots (CDs). Finally, we also propose the main challenges and future research directions of MPM technology in HCC.

Multiphoton microscopy in surgical oncology- a systematic review and guide for clinical translatability

BACKGROUND

Multiphoton microscopy (MPM) facilitates three-dimensional, high-resolution functional imaging of unlabeled tissues in vivo and ex vivo. This systematic review discusses the diagnostic value, advantages and challenges in the practical use of MPM in surgical oncology.

METHOD AND FINDINGS

A Medline search was conducted in April 2019. Fifty-three original research papers investigating MPM compared to standard histology in human patients with solid tumors were identified. A qualitative synopsis and meta-analysis of 14 blinded studies was performed. Risk of bias and applicability were evaluated. MPM can image fresh, frozen or fixed tissues up to a depth 1000 μm in the z-plane. Best results including functional imaging and virtual histochemistry are obtained by in vivo imaging or scanning fresh tissue immediately after excision. Two-photon excited fluorescence by natural fluorophores of the cytoplasm and second harmonic generation signals by fluorophores of the extracellular matrix can be scanned simultaneously, providing high resolution optical histochemistry comparable to standard histology. Functional parameters like fluorescence lifetime imaging or optical redox ratio provide additional objective information. A major concern is inability to visualize the nucleus. However, in a subpopulation analysis of 440 specimens, MPM yielded a sensitivity of 94%, specificity of 96% and accuracy of 95% for the detection of malignant tissue.

CONCLUSION

MPM is a promising emerging technique in surgical oncology. Ex vivo imaging has high sensitivity, specificity and accuracy for the detection of tumor cells. For broad clinical application in vivo, technical challenges need to be resolved.

Classification and Segmentation of Hyperspectral Data of Hepatocellular Carcinoma Samples Using 1-D Convolutional Neural Network

Pathological diagnosis plays an important role in the diagnosis and treatment of hepatocellular carcinoma (HCC). The traditional method of pathological diagnosis of most cancers requires freezing, slicing, hematoxylin and eosin staining, and manual analysis, limiting the speed of the diagnosis process. In this study, we designed a one-dimensional convolutional neural network to classify the hyperspectral data of HCC sample slices acquired by our hyperspectral imaging system. A weighted loss function was employed to promote the performance of the model. The proposed method allows us to accelerate the diagnosis process of identifying tumor tissues. Our deep learning model achieved good performance on our data set with sensitivity, specificity, and area under receiver operating characteristic curve of 0.871, 0.888, and 0.950, respectively. Meanwhile, our deep learning model outperformed the other machine learning methods assessed on our data set. The proposed method is a potential tool for the label-free and real-time pathologic diagnosis. © 2019 International Society for Advancement of Cytometry.