Real-time diagnosis and visualization of tumor margins in excised breast specimens using fluorescence lifetime imaging and machine learning

Dual excitation spectral autofluorescence lifetime and reflectance imaging for fast macroscopic characterization of tissues

Advancements in optical imaging techniques have revolutionized the field of biomedical research, allowing for the comprehensive characterization of tissues and their underlying biological processes. Yet, there is still a lack of tools to provide quantitative and objective characterization of tissues that can aid clinical assessment in vivo to enhance diagnostic and therapeutic interventions. Here, we present a clinically viable fiber-based imaging system combining time-resolved spectrofluorimetry and reflectance spectroscopy to achieve fast multiparametric macroscopic characterization of tissues. An essential feature of the setup is its ability to perform dual wavelength excitation in combination with recording time-resolved fluorescence data in several spectral intervals. Initial validation of this bimodal system was carried out in freshly resected human colorectal cancer specimens, where we demonstrated the ability of the system to differentiate normal from malignant tissues based on their autofluorescence and reflectance properties. To further highlight the complementarity of autofluorescence and reflectance measurements and demonstrate viability in a clinically relevant scenario, we also collected in vivo data from the skin of a volunteer. Altogether, integration of these modalities in a single platform can offer multidimensional characterization of tissues, thus facilitating a deeper understanding of biological processes and potentially advancing diagnostic and therapeutic approaches in various medical applications.

New approaches for low phototoxicity imaging of living cells and tissues

Fluorescence microscopy is a powerful tool used in scientific and medical research, but it is inextricably linked to phototoxicity. Neglecting phototoxicity can lead to erroneous or inconclusive results. Recently, several reports have addressed this issue, but it is still underestimated by many researchers, even though it can lead to cell death. Phototoxicity can be reduced by appropriate microscopic techniques and carefully designed experiments. This review focuses on recent strategies to reduce phototoxicity in microscopic imaging of living cells and tissues. We describe digital image processing and new hardware solutions. We point out new modifications of microscopy methods and hope that this review will interest microscopy hardware engineers. Our aim is to underscore the challenges and potential solutions integral to the design of microscopy systems. Simultaneously, we intend to engage biologists, offering insight into the latest technological advancements in imaging that can enhance their understanding and practice.

Coupling a recurrent neural network to SPAD TCSPC systems for real-time fluorescence lifetime imaging

Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. Inspired by the concept of Edge Artificial Intelligence (Edge AI), we propose a robust approach that enables fast FLI with no degradation of accuracy. This approach couples a recurrent neural network (RNN), which is trained to estimate the fluorescence lifetime directly from raw timestamps without building histograms, to SPAD TCSPC systems, thereby drastically reducing transfer data volumes and hardware resource utilization, and enabling real-time FLI acquisition. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in the presence of background noise by a large margin. To explore the ultimate limits of the approach, we derive the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. To demonstrate real-time operation, we build a FLI microscope based on an existing SPAD TCSPC system comprising a 32[Formula: see text]32 SPAD sensor named Piccolo. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on the Xilinx Kintex-7 FPGA that controls the Piccolo. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications, including biological imaging, biomedical diagnostics, and fluorescence-assisted surgery, etc.

Applications of machine learning in time-domain fluorescence lifetime imaging: a review

Many medical imaging modalities have benefited from recent advances in Machine Learning (ML), specifically in deep learning, such as neural networks. Computers can be trained to investigate and enhance medical imaging methods without using valuable human resources. In recent years, Fluorescence Lifetime Imaging (FLIm) has received increasing attention from the ML community. FLIm goes beyond conventional spectral imaging, providing additional lifetime information, and could lead to optical histopathology supporting real-time diagnostics. However, most current studies do not use the full potential of machine/deep learning models. As a developing image modality, FLIm data are not easily obtainable, which, coupled with an absence of standardisation, is pushing back the research to develop models which could advance automated diagnosis and help promote FLIm. In this paper, we describe recent developments that improve FLIm image quality, specifically time-domain systems, and we summarise sensing, signal-to-noise analysis and the advances in registration and low-level tracking. We review the two main applications of ML for FLIm: lifetime estimation and image analysis through classification and segmentation. We suggest a course of action to improve the quality of ML studies applied to FLIm. Our final goal is to promote FLIm and attract more ML practitioners to explore the potential of lifetime imaging.

Rapid azoospermia classification by stimulated Raman scattering and second harmonic generation microscopy

Disease diagnosis and classification pose significant challenges due to the limited capabilities of traditional methods to obtain molecular information with spatial distribution. Optical imaging techniques, utilizing (auto)fluorescence and nonlinear optical signals, introduce new dimensions for biomarkers exploration that can improve diagnosis and classification. Nevertheless, these signals often cover only a limited number of species, impeding a comprehensive assessment of the tissue microenvironment, which is crucial for effective disease diagnosis and therapy. To address this challenge, we developed a multimodal platform, termed stimulated Raman scattering and second harmonic generation microscopy (SRASH), capable of simultaneously providing both chemical bonds and structural information of tissues. Applying SRASH imaging to azoospermia patient samples, we successfully identified lipids, protein, and collagen contrasts, unveiling molecular and structural signatures for non-obstructive azoospermia. This achievement is facilitated by LiteBlendNet-Dx (LBNet-Dx), our diagnostic algorithm, which achieved an outstanding 100% sample-level accuracy in classifying azoospermia, surpassing conventional imaging modalities. As a label-free technique, SRASH imaging eliminates the requirement for sample pre-treatment, demonstrating great potential for clinical translation and enabling molecular imaging-based diagnosis and therapy.

Macroscopic temporally and spectrally resolved fluorescence imaging enhanced by laser-wavelength multiplexing

We present a laser scanning system for macroscopic samples that records fully resolved decay curves in individual pixels, resolves the images in 16 wavelength channels, and records simultaneously at several laser wavelengths. By using confocal detection, the system delivers images that are virtually free of lateral scattering and out-of-focus haze. Image formats can be up to 256 × 256 pixels and up to 1024 time channels. We demonstrate the performance of the system both on model experiments with fluorescent micro-beads and on the tumor model in the living mice.

Probing organoid metabolism using fluorescence lifetime imaging microscopy (FLIM): The next frontier of drug discovery and disease understanding

Organoid models have been used to address important questions in developmental and cancer biology, tissue repair, advanced modelling of disease and therapies, among other bioengineering applications. Such 3D microenvironmental models can investigate the regulation of cell metabolism, and provide key insights into the mechanisms at the basis of cell growth, differentiation, communication, interactions with the environment and cell death. Their accessibility and complexity, based on 3D spatial and temporal heterogeneity, make organoids suitable for the application of novel, dynamic imaging microscopy methods, such as fluorescence lifetime imaging microscopy (FLIM) and related decay time-assessing readouts. Several biomarkers and assays have been proposed to study cell metabolism by FLIM in various organoid models. Herein, we present an expert-opinion discussion on the principles of FLIM and PLIM, instrumentation and data collection and analysis protocols, and general and emerging biosensor-based approaches, to highlight the pioneering work being performed in this field.

Automatic diagnosis and classification of breast surgical samples with dynamic full-field OCT and machine learning

Purpose

The adoption of emerging imaging technologies in the medical community is often hampered when they provide a new unfamiliar contrast that requires experience to be interpreted. Dynamic full-field optical coherence tomography (D-FF-OCT) microscopy is such an emerging technique. It provides fast, high-resolution images of excised tissues with a contrast comparable to H&E histology but without any tissue preparation and alteration.

Approach

We designed and compared two machine learning approaches to support interpretation of D-FF-OCT images of breast surgical specimens and thus provide tools to facilitate medical adoption. We conducted a pilot study on 51 breast lumpectomy and mastectomy surgical specimens and more than 1000 individual 1.3 × 1.3    mm 2 images and compared with standard H&E histology diagnosis.

Results

Using our automatic diagnosis algorithms, we obtained an accuracy above 88% at the image level (1.3 × 1.3    mm 2 ) and above 96% at the specimen level (above cm 2 ).

Conclusions

Altogether, these results demonstrate the high potential of D-FF-OCT coupled to machine learning to provide a rapid, automatic, and accurate histopathology diagnosis with minimal sample alteration.

Stomach tissue classification using autofluorescence spectroscopy and machine learning

BACKGROUND AND OBJECTIVES

Determination of stomach tumor location and invasion depth requires delineation of gastric histological structure, which has hitherto been widely accomplished by histochemical staining. In recent years, alternative histochemical evaluation methods have been pursued to accelerate intraoperative diagnosis, often by bypassing the time-consuming step of dyeing. Owing to strong endogenous signals from coenzymes, metabolites, and proteins, autofluorescence spectroscopy is a favorable candidate technique to achieve this aim.

MATERIALS AND METHODS

We investigated stomach tissue slices and block specimens using a fast fluorescence imaging scanner. To obtain histological information from broad and structureless fluorescence spectra, we analyzed tens of thousands of spectra with multiple machine-learning algorithms and built a tissue classification model trained with dissected gastric tissues.

RESULTS

A machine-learning-based spectro-histological model was built based on the autofluorescence spectra measured from stomach tissue samples with delineated and validated histological structures. The scores from a principal components analysis were employed as input features, and prediction accuracy was confirmed to be 92.0%, 90.1%, and 91.4% for mucosa, submucosa, and muscularis propria, respectively. We investigated the tissue samples in both sliced and block forms using a fast fluorescence imaging scanner.

CONCLUSION

We successfully demonstrated differentiation of multiple tissue layers of well-defined specimens with the guidance of a histologist. Our spectro-histology classification model is applicable to histological prediction for both tissue blocks and slices, even though only sliced samples were trained.

High Dynamic Range Dual-Modal White Light Imaging Improves the Accuracy of Tumor Bed Sampling After Neoadjuvant Therapy for Breast Cancer

OBJECTIVES

Accurate evaluation of residual cancer burden remains challenging because of the lack of appropriate techniques for tumor bed sampling. This study evaluated the application of a white light imaging system to help pathologists differentiate the components and location of tumor bed in specimens.

METHODS

The high dynamic range dual-mode white light imaging (HDR-DWI) system was developed to capture antiglare reflection and multiexposure HDR transmission images. It was tested in 60 specimens of modified radical mastectomy after neoadjuvant therapy. We observed the differential transmittance among tumor tissue, fibrosis tissue, and adipose tissue.

RESULTS

The sensitivity and specificity of HDR-DWI were compared with x-ray or visual examination to determine whether HDR-DWI was superior in identifying tumor beds. We found that tumor tissue had lower transmittance (0.12 ± 0.03) than fibers (0.15 ± 0.04) and fats (0.27 ± 0.07) (P < .01).

CONCLUSIONS

HDR-DWI was more sensitive in identifying fiber and tumor tissues than cabinet x-ray and visual observation (P < .01). In addition, HDR-DWI could identify more fibrosis areas than the currently used whole slide imaging did in 12 samples (12/60). We have determined that HDR-DWI can provide more in-depth tumor bed information than x-ray and visual examination do, which will help prevent diagnostic errors in tumor bed sampling.

NIRDye 812: A molecular platform tailored for multimodal bioimaging applications of targeted fluorescence- and photoacoustic-guided surgery

The primary treatment for malignant tumors remains to be surgical removal of the diseased tissue. The presence or absence of residual diseased tissue at the tumor margin is the strongest predictor of postoperative prognosis and recurrence. Accordingly, reliance on the ability of surgeons to visually distinguish diseased tissue from healthy tissue unambiguously in real time is crucial. Near infrared-I (NIRI) fluorescence-emitting targeting biomolecular constructs such as anticancer antibody-fluorophore conjugates, namely cetuximab-IRDye® 800CW (CTB-IRDye® 800CW), are FDA-approved for clinical trial usage in the fluorescence-guided resection of diseased tissue due to affording improved direct visualization of tumor tissue when compared to the use of either the unaided eye under standard white light illumination (WLI) surgical techniques or non-targeting fluorophores. Unfortunately, though helpful, CTB-IRDye® 800CW affords limited (i) identification of diseased tissue and (ii) tumor margin delineation, because the immunoconjugate generates suboptimal tumor-to-background ratios (TBRs) as a result of its spectral/photophysical profiles poorly aligning with the fixed optical windows of pre-/clinical setups. As such, CTB-IRDye® 800CW is more prone to affording incomplete resection compared to if TBRs were higher due to otherwise. To aid in accurately identifying deep-seated diseased tissue, photoacoustic (PA) tomography has been implemented alongside CTB-IRDye® 800CW to achieve PA signals that could result in higher TBRs. However, in clinical trial practice, using IRDye® 800CW for PA imaging also yields subpar TBRs due to it affording low PA signals. To overcome such limitations, we developed NIRDye 812, a structurally-modified topological equivalent of IRDye® 800CW, to confer it the capability to yield both higher TBRs and superior PA signal than that of the equivalent CTB-conjugate and fluorophore IRDye® 800CW itself, respectively. To do so, we substituted the oxygen atom at its meso-position with a sulfur atom. CTB-NIRDye 812 demonstrated a red-shifted absorption wavelength at 796 nm and a peak NIR-I fluorescence emission wavelength at 820 nm, which better dovetails with the fixed windows of preinstalled fixed emission filters within commercial pre-/clinical NIR-I fluorescence imaging instruments. Overall, CTB-NIRDye 812 provided a ∼ 2-fold increase in TBRs compared to those of CTB-IRDye® 800CW in vivo. Also, NIRDye 812 displayed an ∼60% higher PA signal than that of IRDye® 800CW. Collectively, we achieved our goal of improving upon the spectral/photophysical and PA properties of IRDye® 800CW via introducing a subtle modification to its electronic core such that its CTB immunoconjugate could potentially allow for fast track or breakthrough designation by the FDA due to its near-identical structure displaying considerably improved efficacy.

Suppression of natural lens fluorescence in fundus autofluorescence measurements: review of hardware solutions

Fluorescence lifetime imaging ophthalmoscopy (FLIO), a technique for investigating metabolic changes in the eye ground, can reveal the first signs of diseases related to metabolism. The fluorescence of the natural lens overlies the fundus fluorescence. Although the influence of natural lens fluorescence can be somewhat decreased with mathematical models, excluding this influence during the measurement by using hardware enables more exact estimation of the fundus fluorescence. Here, we analyze four 1-photon excitation hardware solutions to suppress the influence of natural lens fluorescence: aperture stop separation, confocal scanning laser ophthalmoscopy, combined confocal scanning laser ophthalmoscopy and aperture stop separation, and dual point confocal scanning laser ophthalmoscopy. The effect of each principle is demonstrated in examples. The best suppression is provided by the dual point principle, realized with a confocal scanning laser ophthalmoscope. In this case, in addition to the fluorescence of the whole eye, the fluorescence of the anterior part of the eye is detected from a non-excited spot of the fundus. The intensity and time-resolved fluorescence spectral data of the fundus are derived through the subtraction of the simultaneously measured fluorescence of the excited and non-excited spots. Advantages of future 2-photon fluorescence excitation are also discussed. This study provides the first quantitative evaluation of hardware principles to suppress the fluorescence of the natural lens during measurements of fundus autofluorescence.

Intraoperative delineation of p16+ oropharyngeal carcinoma of unknown primary origin with fluorescence lifetime imaging: Preliminary report

BACKGROUND

This study evaluated whether fluorescence lifetime imaging (FLIm), coupled with standard diagnostic workups, could enhance primary lesion detection in patients with p16+ head and neck squamous cell carcinoma of the unknown primary (HNSCCUP).

METHODS

FLIm was integrated into transoral robotic surgery to acquire optical data on six HNSCCUP patients' oropharyngeal tissues. An additional 55-patient FLIm dataset, comprising conventional primary tumors, trained a machine learning classifier; the output predicted the presence and location of HNSCCUP for the six patients. Validation was performed using histopathology.

RESULTS

Among the six HNSCCUP patients, p16+ occult primary was surgically identified in three patients, whereas three patients ultimately had no identifiable primary site in the oropharynx. FLIm correctly detected HNSCCUP in all three patients (ROC-AUC: 0.90 ± 0.06), and correctly predicted benign oropharyngeal tissue for the remaining three patients. The mean sensitivity was 95% ± 3.5%, and specificity 89% ± 12.7%.

CONCLUSIONS

FLIm may be a useful diagnostic adjunct for detecting HNSCCUP.

Nuclear‐medicine probes: where we are and where we are going

Nuclear medicine probes turned into the key for the identification and precise location of sentinel lymph nodes and other occult lesions (i.e., tumors) by using the systemic administration of radiotracers. Intraoperative nuclear probes are key in the surgical management of some malignancies as well as in the determination of positive surgical margins, thus reducing the extent and potential surgery morbidity.

Depending on their application, nuclear probes are classified into two main categories, namely, counting and imaging. Although counting probes present a simple design, are handheld (to be moved rapidly), and provide only acoustic signals when detecting radiation, imaging probes, also known as cameras, are more hardware‐complex and also able to provide images but at the cost of an increased intervention time as displacing the camera has to be done slowly.

This review article begins with an introductory section to highlight the relevance of nuclear‐based probes and their components as well as the main differences between ionization‐ (semiconductor) and scintillation‐based probes. Then, the most significant performance parameters of the probe are reviewed (i.e., sensitivity, contrast, count rate capabilities, shielding, energy, and spatial resolution), as well as the different types of probes based on the target radiation nature, namely: gamma (γ), beta (β) (positron and electron), and Cherenkov. Various available intraoperative nuclear probes are finally compared in terms of performance to discuss the state‐of‐the‐art of nuclear medicine probes.

The manuscript concludes by discussing the ideal probe design and the aspects to be considered when selecting nuclear‐medicine probes.

Spatial analysis of photoplethysmography in cutaneous squamous cell carcinoma

The primary treatment of the common malignancy squamous cell carcinoma is surgical removal. In this process, sufficient tissue removal is balanced against unnecessary mutilation. We recently presented a remote photoplethysmography algorithm, which revealed significant differences between processed video recordings of cancer biopsy areas and surrounding tissue. The aim of this study was to investigate whether spatial analyses of photoplethysmography data correlate with post-excision pathological analyses and thus have potential to assist in tumour delineation. Based on high speed video recordings of 11 patients with squamous cell carcinoma, we examined different parameters derived from temporal remote photoplethysmography variations. Signal characteristics values in sites matching histological sections were compared with pathological measures. Values were ranked and statistically tested with a Kendall correlation analysis. A moderate, negative correlation was found between signal oscillations and the width and transversal area of squamous cell carcinoma in the frequencies below 1 Hz and specifically from 0.02 to 0.15 Hz. We have presented a correlation between frequency content and prevalence of cancer based on regular video recordings of squamous cell carcinoma. We believe this is supported by published findings on malignant melanoma. Our findings indicate that photoplethysmography can be used to distinguish SCC from healthy skin.

Proof of concept for the sapphire scalpel combining tissue dissection and optical diagnosis

OBJECTIVES

The development of compact diagnostic probes and instruments with an ability to direct access to organs and tissues and integration of these instruments into surgical workflows is an important task of modern physics and medicine. The need for such tools is essential for surgical oncology, where intraoperative visualization and demarcation of tumor margins define further prognosis and survival of patients. In this paper, the possible solution for this intraoperative imaging problem is proposed and its feasibility to detect tumorous tissue is studied experimentally.

METHODS

For this aim, the sapphire scalpel was developed and fabricated using the edge-defined film-fed growth technique aided by mechanical grinding, polishing, and chemical sharpening of the cutting edge. It possesses optical transparency, mechanical strength, chemical inertness, and thermal resistance alongside the presence of the as-grown hollow capillary channels in its volume for accommodating optical fibers. The rounding of the cutting edge exceeds the same for metal scalpels and can be as small as 110 nm. Thanks to these features, sapphire scalpel combines tissue dissection with light delivering and optical diagnosis. The feasibility for the tumor margin detection was studied, including both gelatin-based tissue phantoms and ex vivo freshly excised specimens of the basal cell carcinoma from humans and the glioma model 101.8 from rats. These tumors are commonly diagnosed either non-invasively or intraoperatively using different modalities of fluorescence spectroscopy and imaging, which makes them ideal candidates for our feasibility test. For this purpose, fiber-based spectroscopic measurements of the backscattered laser radiation and the fluorescence signals were carried out in the visible range.

RESULTS

Experimental studies show the feasibility of the proposed sapphire scalpel to provide a 2-mm-resolution of the tumor margins' detection, along with an ability to distinguish the tumor invasion region, which results from analysis of the backscattered optical fields and the endogenous or exogenous fluorescence data.

CONCLUSIONS

Our findings justified a strong potential of the sapphire scalpel for surgical oncology. However, further research and engineering efforts are required to optimize the sapphire scalpel geometry and the optical diagnosis protocols to meet the requirements of oncosurgery, including diagnosis and resection of neoplasms with different localizations and nosologies.

Single-Fiber Diffuse Reflectance Spectroscopy and Spatial Frequency Domain Imaging in Surgery Guidance: A Study on Optical Phantoms

Diffuse reflectance spectroscopy (DRS) and imaging are increasingly being used in surgical guidance for tumor margin detection during endoscopic operations. However, the accuracy of the boundary detection with optical techniques may depend on the acquisition parameters, and its evaluation is in high demand. In this work, using optical phantoms with homogeneous and heterogeneous distribution of chromophores mimicking normal and pathological bladder tissues, the accuracy of tumor margin detection using single-fiber diffuse reflectance spectroscopy and spatial frequency domain imaging was evaluated. We also showed how the diffuse reflectance response obtained at different spatial frequencies with the spatial frequency domain imaging technique could be used not only to quantitatively map absorption and scattering coefficients of normal tissues and tumor-like heterogeneities but also to estimate the tumor depth localization. The demonstrated results could be helpful for proper analysis of the DRS data measured in vivo and for translation of optical techniques for tumor margin detection to clinics.

Feasibility of dual-contrast fluorescence imaging of pathological breast tissues

The combination of intravital dye, methylene blue (MB), with molecular cancer marker, pH low insertion peptide (pHLIP) conjugated with fluorescent Alexa532 (Alexa532-pHLIP), was evaluated for enhancing contrast of pathological breast tissue ex vivo. Fresh, thick breast specimens were stained sequentially with Alexa532-pHLIP and aqueous MB and imaged using dual-channel fluorescence microscopy. MB and Alexa532-pHLIP accumulated in the nuclei and cytoplasm of cancer cells, respectively. MB also stained nuclei of normal cells. Some Alexa532-pHLIP fluorescence emission was detected from connective tissue and benign cell membranes. Overall, Alexa532-pHLIP showed high affinity to cancer, while MB highlighted tissue morphology. The results indicate that MB and Alexa532-pHLIP provide complementary information and show promise for the detection of breast cancer.

Mesoscopic fluorescence lifetime imaging: Fundamental principles, clinical applications and future directions

Fluorescence lifetime imaging (FLIm) is an optical spectroscopic imaging technique capable of real-time assessments of tissue properties in clinical settings. Label-free FLIm is sensitive to changes in tissue structure and biochemistry resulting from pathological conditions, thus providing optical contrast to identify and monitor the progression of disease. Technical and methodological advances over the last two decades have enabled the development of FLIm instrumentation for real-time, in situ, mesoscopic imaging compatible with standard clinical workflows. Herein, we review the fundamental working principles of mesoscopic FLIm, discuss the technical characteristics of current clinical FLIm instrumentation, highlight the most commonly used analytical methods to interpret fluorescence lifetime data and discuss the recent applications of FLIm in surgical oncology and cardiovascular diagnostics. Finally, we conclude with an outlook on the future directions of clinical FLIm.

A review of the application of machine learning in molecular imaging

Molecular imaging (MI) is a science that uses imaging methods to reflect the changes of molecular level in living state and conduct qualitative and quantitative studies on its biological behaviors in imaging. Optical molecular imaging (OMI) and nuclear medical imaging are two key research fields of MI. OMI technology refers to the optical information generated by the imaging target (such as tumors) due to drug intervention and other reasons. By collecting the optical information, researchers can track the motion trajectory of the imaging target at the molecular level. Owing to its high specificity and sensitivity, OMI has been widely used in preclinical research and clinical surgery. Nuclear medical imaging mainly detects ionizing radiation emitted by radioactive substances. It can provide molecular information for early diagnosis, effective treatment and basic research of diseases, which has become one of the frontiers and hot topics in the field of medicine in the world today. Both OMI and nuclear medical imaging technology require a lot of data processing and analysis. In recent years, artificial intelligence technology, especially neural network-based machine learning (ML) technology, has been widely used in MI because of its powerful data processing capability. It provides a feasible strategy to deal with large and complex data for the requirement of MI. In this review, we will focus on the applications of ML methods in OMI and nuclear medical imaging.

Intraoperative Mapping of Parathyroid Glands Using Fluorescence Lifetime Imaging

BACKGROUND

Hypoparathyroidism is a common complication following thyroidectomy. There is a need for technology to aid surgeons in identifying the parathyroid glands. In contrast to near infrared technologies, fluorescence lifetime imaging (FLIm) is not affected by ambient light and may be valuable in identifying parathyroid tissue, but has never been evaluated in this capacity.

METHODS

We used FLIm to measure the UV induced (355 nm) time-resolved autofluorescence signatures (average lifetimes in 3 spectral emission channels) of thyroid, parathyroid, lymphoid and adipose tissue in 21 patients undergoing thyroid and parathyroid surgery. The Mann-Whitney U test was used to assess the ability of FLIm to discriminate normocellular parathyroid from each of the other tissues. Various machine learning classifiers (random forests, neural network, support vector machine) were then evaluated to recognize parathyroid through a leave-one-out cross-validation.

RESULTS

Statistically significant differences in average lifetime were observed between parathyroid and each of the other tissue types in spectral channels 2 and 3 respectively. The largest change was observed between adipose tissue and parathyroid (P < 0.001), while less pronounced but still significant changes were observed when comparing parathyroid with lymphoid tissue (P < 0.05) and thyroid (P < 0.01). A random forest classifier trained on average lifetimes was found to detect parathyroid tissue with 100% sensitivity and 93% specificity at the acquisition run level.

CONCLUSION

We found that FLIm derived parameters can distinguish the parathyroid glands and other adjacent tissue types and has promise in scanning the surgical field to identify parathyroid tissue in real-time.

Intraoperative Margin Assessment in Oral and Oropharyngeal Cancer Using Label-Free Fluorescence Lifetime Imaging and Machine Learning

OBJECTIVE

To demonstrate the diagnostic ability of label-free, point-scanning, fiber-based Fluorescence Lifetime Imaging (FLIm) as a means of intraoperative guidance during oral and oropharyngeal cancer removal surgery.

METHODS

FLIm point-measurements acquired from 53 patients (n = 67893 pre-resection in vivo, n = 89695 post-resection ex vivo) undergoing oral or oropharyngeal cancer removal surgery were used for analysis. Discrimination of healthy tissue and cancer was investigated using various FLIm-derived parameter sets and classifiers (Support Vector Machine, Random Forests, CNN). Classifier output for the acquired set of point-measurements was visualized through an interpolation-based approach to generate a probabilistic heatmap of cancer within the surgical field. Classifier output for dysplasia at the resection margins was also investigated.

RESULTS

Statistically significant change (P 0.01) between healthy and cancer was observed in vivo for the acquired FLIm signal parameters (e.g., average lifetime) linked with metabolic activity. Superior classification was achieved at the tissue region level using the Random Forests method (ROC-AUC: 0.88). Classifier output for dysplasia (% probability of cancer) was observed to lie between that of cancer and healthy tissue, highlighting FLIm's ability to distinguish various conditions.

CONCLUSION

The developed approach demonstrates the potential of FLIm for fast, reliable intraoperative margin assessment without the need for contrast agents.

SIGNIFICANCE

Fiber-based FLIm has the potential to be used as a diagnostic tool during cancer resection surgery, including Transoral Robotic Surgery (TORS), helping ensure complete resections and improve the survival rate of oral and oropharyngeal cancer patients.

[New Trends of Development in Precision Pathological Diagnosis Promoted by Artificial Intelligence]

Precision pathological diagnosis plays a vital role in precision medicine. Both the limited resources available to pathologists and the incessant demands for further refinement and quantification of clinical diagnosis are posing new challenges for pathologists to meet the needs for precision pathological diagnosis. It is expected that artificial intelligence (AI) will be the powerful tool that will help find solutions to this problem from different angles. The author of this article elaborated on a number of ways in which AI can help promote precision pathological diagnosis, including AI-assisted precision extraction of tissue samples, AI-assisted precision histopathologic diagnosis, AI-assisted histological grading and quantitative scoring, AI-assisted precision assessment of tumor biomarkers, AI-assisted prediction of molecular features and precision interpretation of biological information based on hematoxylin-eosin (HE) stained images, the realization of in-depth precision diagnosis based on AI-assisted information integration, and AI-assisted accurate prediction of patient survival and prognosis based on HE-stained images. The paper presents to the readers the future of smart pathology that AI will help usher in.

Hyperspectral microscopy combined with DAPI staining for the identification of hepatic carcinoma cells

In this study, the DAPI staining is firstly reported for use in the identification of hepatic carcinoma cells based on hyperspectral microscopy. Nuclei in cancer cells usually contain more aneuploidies than that in normal cells, leading to the change of DNA content. Here, we stain hepatic carcinoma tissues and normal hepatic tissues with 4',6-diamidino-2-phenylindole (DAPI) which is sensitive to the DNA content as a fluorochrome binds to DNA. Consequently, the difference in DNA content between hepatic carcinoma cells and normal hepatic cells can be identified by the fluorescent spectral characteristics. Harnessing the hyperspectral microscopy, we find that the fluorescent properties of these two kinds of cells are different not only in the intensity but also in the spectral shape. These properties are exploited to train a support vector machine (SVM) model for classifying cells. The results show that the sensitivity and specificity for the identification of 1000 hepatic carcinoma samples are 99.3% and 99.1%, respectively.

Off-Peak Near-Infrared-II (NIR-II) Bioimaging of an Immunoconjugate Having Peak Fluorescence Emission in the NIR-I Spectral Region for Improving Tumor Margin Delineation

The primary treatment for malignant tumors remains to be resection. The strongest predictor of recurrence and postoperative prognosis is whether diseased tissue/cells remain(s) at the surgical margin. Cancer surgery entails surgeons having the capability to visually distinguish between subtle shades of color in attempts of differentiating between diseased tissue and healthy tissue under standard white-light illumination, as such tissue states appear identical at the meso-/macroscopic level. Accordingly, enhancing the capability of surgeons to do so such that they can accurately delineate the tumor margin is of paramount importance. Fluorescence-guided surgery facilitates in enhancing such capability by color-coding the surgical field with overlaid contrasting pseudo-colors from real-time intraoperative fluorescence emission via utilizing fluorescent constructs in tandem. Constructs undergoing clinical trials or that are FDA-approved provide peak fluorescence emission in the visible (405 - 700 nm) or near-infrared-I (NIR-I) spectral region (700-900 nm), whereby differentiation between tissue states progressively improves in sync with using constructs that emit longer wavelengths of light. Here, we repurpose the usage of such fluorescent constructs by establishing feasibility of a tumor-targeting immunoconjugate (cetuximab-IRDye800) having peak fluorescence emission at the NIR-I spectral region to provide improved tumor margin delineation by affording higher tumor-to-background ratios (TBRs) when measuring its off-peak fluorescence emission at the near-infrared-II (NIR-II) spectral region (1000-1700 nm) in in vivo applications. We prepared murine tumor models, administered such immunoconjugate, and imaged such models pre-/post-administration via utilizing imaging systems that separately afforded acquisition of fluorescence emission in the NIR-I or NIR-II spectral region. On doing so, we determined in vivo TBRs, ex vivo TBRs with/-out skin, and ex vivo biodistribution, all via measuring the fluorescence emission of the immunoconjugate at tumor site(s) at both spectral regions. Collectively, we established feasibility of using the immunoconjugate to afford improved tumor margin delineation by providing 2-fold higher TBRs via utilizing the NIR-II spectral region to capture off-peak fluorescence emission from a fluorescent construct having NIR-I peak fluorescence emission.

Attitudes Toward Artificial Intelligence Within Dermatopathology: An International Online Survey

Background: Artificial intelligence (AI) has recently surfaced as a research topic in dermatology and dermatopathology. In a recent survey, dermatologists were overall positive toward a development with an increased use of AI, but little is known about the corresponding attitudes among pathologists working with dermatopathology. The objective of this investigation was to make an inventory of these attitudes.

Participants and Methods: An anonymous and voluntary online survey was prepared and distributed to pathologists who regularly analyzed dermatopathology slides/images. The survey consisted of 39 question divided in five sections; (1) AI as a topic in pathology; (2) previous exposure to AI as a topic in general; (3) applications for AI in dermatopathology; (4) feelings and attitudes toward AI and (5) self-reported tech-savviness and demographics. The survey opened on March 13, 2020 and closed on May 5, 2020.

Results: Overall, 718 responders (64.1% females) representing 91 countries were analyzed. While 81.5% of responders were aware of AI as an emerging topic in pathology, only 18.8% had either good or excellent knowledge about AI. In terms of diagnosis classification, 42.6% saw strong or very strong potential for automated suggestion of skin tumor diagnoses. The corresponding figure for inflammatory skin diseases was 23.0% (P_adj < 0.0001). For specific applications, the highest potential was considered for automated detection of mitosis (79.2%), automated suggestion of tumor margins (62.1%) and immunostaining evaluation (62.7%). The potential for automated suggestion of immunostaining (37.6%) and genetic panels (48.3%) were lower. Age did not impact the overall attitudes toward AI. Only 6.0% of the responders agreed or strongly agreed that the human pathologist will be replaced by AI in the foreseeable future. For the entire group, 72.3% agreed or strongly agreed that AI will improve dermatopathology and 84.1% thought that AI should be a part of medical training.

Conclusions: Pathologists are generally optimistic about the impact and potential benefit of AI in dermatopathology. The highest potential is expected for narrow specified tasks rather than a global automated suggestion of diagnoses. There is a strong need for education about AI and its use within dermatopathology.

Machine Learning Methods for Fluorescence Lifetime Imaging (FLIM) Based Label-Free Detection of Microglia

Automated computational analysis techniques utilizing machine learning have been demonstrated to be able to extract more data from different imaging modalities compared to traditional analysis techniques. One new approach is to use machine learning techniques to existing multiphoton imaging modalities to better interpret intrinsically fluorescent cellular signals to characterize different cell types. Fluorescence Lifetime Imaging Microscopy (FLIM) is a high-resolution quantitative imaging tool that can detect metabolic cellular signatures based on the lifetime variations of intrinsically fluorescent metabolic co-factors such as nicotinamide adenine dinucleotide [NAD(P)H]. NAD(P)H lifetime-based discrimination techniques have previously been used to develop metabolic cell signatures for diverse cell types including immune cells such as macrophages. However, FLIM could be even more effective in characterizing cell types if machine learning was used to classify cells by utilizing FLIM parameters for classification. Here, we demonstrate the potential for FLIM-based, label-free NAD(P)H imaging to distinguish different cell types using Artificial Neural Network (ANN)-based machine learning. For our biological use case, we used the challenge of differentiating microglia from other glia cell types in the brain. Microglia are the resident macrophages of the brain and spinal cord and play a critical role in maintaining the neural environment and responding to injury. Microglia are challenging to identify as most fluorescent labeling approaches cross-react with other immune cell types, are often insensitive to activation state, and require the use of multiple specialized antibody labels. Furthermore, the use of these extrinsic antibody labels prevents application in in vivo animal models and possible future clinical adaptations such as neurodegenerative pathologies. With the ANN-based NAD(P)H FLIM analysis approach, we found that microglia in cell culture mixed with other glial cells can be identified with more than 0.9 True Positive Rate (TPR). We also extended our approach to identify microglia in fixed brain tissue with a TPR of 0.79. In both cases the False Discovery Rate was around 30%. This method can be further extended to potentially study and better understand microglia’s role in neurodegenerative disease with improved detection accuracy.

A Review on Advances in Intra-operative Imaging for Surgery and Therapy: Imagining the Operating Room of the Future

With the advent of Minimally Invasive Surgery (MIS), intra-operative imaging has become crucial for surgery and therapy guidance, allowing to partially compensate for the lack of information typical of MIS. This paper reviews the advancements in both classical (i.e. ultrasounds, X-ray, optical coherence tomography and magnetic resonance imaging) and more recent (i.e. multispectral, photoacoustic and Raman imaging) intra-operative imaging modalities. Each imaging modality was analyzed, focusing on benefits and disadvantages in terms of compatibility with the operating room, costs, acquisition time and image characteristics. Tables are included to summarize this information. New generation of hybrid surgical room and algorithms for real time/in room image processing were also investigated. Each imaging modality has its own (site- and procedure-specific) peculiarities in terms of spatial and temporal resolution, field of view and contrasted tissues. Besides the benefits that each technique offers for guidance, considerations about operators and patient risk, costs, and extra time required for surgical procedures have to be considered. The current trend is to equip surgical rooms with multimodal imaging systems, so as to integrate multiple information for real-time data extraction and computer-assisted processing. The future of surgery is to enhance surgeons eye to minimize intra- and after-surgery adverse events and provide surgeons with all possible support to objectify and optimize the care-delivery process.