1
|
Lboukili I, Stamatas G, Descombes X. Automating reflectance confocal microscopy image analysis for dermatological research: a review. JOURNAL OF BIOMEDICAL OPTICS 2022; 27:JBO-220021VRR. [PMID: 35879817 PMCID: PMC9309100 DOI: 10.1117/1.jbo.27.7.070902] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/28/2022] [Accepted: 07/08/2022] [Indexed: 05/31/2023]
Abstract
SIGNIFICANCE Reflectance confocal microscopy (RCM) is a noninvasive, in vivo technology that offers near histopathological resolution at the cellular level. It is useful in the study of phenomena for which obtaining a biopsy is impractical or would cause unnecessary tissue damage and trauma to the patient. AIM This review covers the use of RCM in the study of skin and the use of machine learning to automate information extraction. It has two goals: (1) an overview of information provided by RCM on skin structure and how it changes over time in response to stimuli and in disease and (2) an overview of machine learning approaches developed to automate the extraction of key morphological features from RCM images. APPROACH A PubMed search was conducted with additional literature obtained from references lists. RESULTS The application of RCM as an in vivo tool in dermatological research and the biologically relevant information derived from it are presented. Algorithms for image classification to epidermal layers, delineation of the dermal-epidermal junction, classification of skin lesions, and demarcation of individual cells within an image, all important factors in the makeup of the skin barrier, were reviewed. Application of image analysis methods in RCM is hindered by low image quality due to noise and/or poor contrast. Use of supervised machine learning is limited by time-consuming manual labeling of RCM images. CONCLUSIONS RCM has great potential in the study of skin structures. The use of artificial intelligence could enable an easier, more reproducible, precise, and rigorous study of RCM images for the understanding of skin structures, skin barrier, and skin inflammation and lesions. Although several attempts have been made, further work is still needed to provide a definite gold standard and overcome issues related to image quality, limited labeled datasets, and lack of phenotype variability in available databases.
Collapse
|
2
|
Bozkurt A, Kose K, Coll-Font J, Alessi-Fox C, Brooks DH, Dy JG, Rajadhyaksha M. Skin strata delineation in reflectance confocal microscopy images using recurrent convolutional networks with attention. Sci Rep 2021; 11:12576. [PMID: 34131165 PMCID: PMC8206415 DOI: 10.1038/s41598-021-90328-x] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2020] [Accepted: 04/12/2021] [Indexed: 11/29/2022] Open
Abstract
Reflectance confocal microscopy (RCM) is an effective non-invasive tool for cancer diagnosis. However, acquiring and reading RCM images requires extensive training and experience, and novice clinicians exhibit high discordance in diagnostic accuracy. Quantitative tools to standardize image acquisition could reduce both required training and diagnostic variability. To perform diagnostic analysis, clinicians collect a set of RCM mosaics (RCM images concatenated in a raster fashion to extend the field view) at 4-5 specific layers in skin, all localized in the junction between the epidermal and dermal layers (dermal-epidermal junction, DEJ), necessitating locating that junction before mosaic acquisition. In this study, we automate DEJ localization using deep recurrent convolutional neural networks to delineate skin strata in stacks of RCM images collected at consecutive depths. Success will guide to automated and quantitative mosaic acquisition thus reducing inter operator variability and bring standardization in imaging. Testing our model against an expert labeled dataset of 504 RCM stacks, we achieved [Formula: see text] classification accuracy and nine-fold reduction in the number of anatomically impossible errors compared to the previous state-of-the-art.
Collapse
Affiliation(s)
- Alican Bozkurt
- Northeastern University, Boston, MA, 02115, USA.
- Paige AI, New York, NY, USA.
| | - Kivanc Kose
- Memorial Sloan Kettering Cancer Center, New York, NY, 10022, USA
| | - Jaume Coll-Font
- Northeastern University, Boston, MA, 02115, USA
- Massachusetts General Hospital, Boston, MA, USA
| | | | | | | | | |
Collapse
|
3
|
Mehrabi JN, Baugh EG, Fast A, Lentsch G, Balu M, Lee BA, Kelly KM. A Clinical Perspective on the Automated Analysis of Reflectance Confocal Microscopy in Dermatology. Lasers Surg Med 2021; 53:1011-1019. [PMID: 33476062 DOI: 10.1002/lsm.23376] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2020] [Revised: 10/30/2020] [Accepted: 12/25/2020] [Indexed: 01/03/2023]
Abstract
BACKGROUND AND OBJECTIVES Non-invasive optical imaging has the potential to provide a diagnosis without the need for biopsy. One such technology is reflectance confocal microscopy (RCM), which uses low power, near-infrared laser light to enable real-time in vivo visualization of superficial human skin from the epidermis down to the papillary dermis. Although RCM has great potential as a diagnostic tool, there is a need for the development of reliable image analysis programs, as acquired grayscale images can be difficult and time-consuming to visually assess. The purpose of this review is to provide a clinical perspective on the current state of artificial intelligence (AI) for the analysis and diagnostic utility of RCM imaging. STUDY DESIGN/MATERIALS AND METHODS A systematic PubMed search was conducted with additional relevant literature obtained from reference lists. RESULTS Algorithms used for skin stratification, classification of pigmented lesions, and the quantification of photoaging were reviewed. Image segmentation, statistical methods, and machine learning techniques are among the most common methods used to analyze RCM image stacks. The poor visual contrast within RCM images and difficulty navigating image stacks were mediated by machine learning algorithms, which allowed the identification of specific skin layers. CONCLUSIONS AI analysis of RCM images has the potential to increase the clinical utility of this emerging technology. A number of different techniques have been utilized but further refinements are necessary to allow consistent accurate assessments for diagnosis. The automated detection of skin cancers requires more development, but future applications are truly boundless, and it is compelling to envision the role that AI will have in the practice of dermatology. Lasers Surg. Med. © 2020 Wiley Periodicals LLC.
Collapse
Affiliation(s)
- Joseph N Mehrabi
- Department of Dermatology, University of California, Irvine, California, 92697
| | - Erica G Baugh
- Department of Dermatology, University of California, Irvine, California, 92697
| | - Alexander Fast
- Beckman Laser Institute, University of California Irvine, Irvine, California, 92612
| | - Griffin Lentsch
- Beckman Laser Institute, University of California Irvine, Irvine, California, 92612
| | - Mihaela Balu
- Beckman Laser Institute, University of California Irvine, Irvine, California, 92612
| | - Bonnie A Lee
- Department of Dermatology, University of California, Irvine, California, 92697
| | - Kristen M Kelly
- Department of Dermatology, University of California, Irvine, California, 92697.,Beckman Laser Institute, University of California Irvine, Irvine, California, 92612
| |
Collapse
|
4
|
Kose K, Bozkurt A, Alessi-Fox C, Brooks DH, Dy JG, Rajadhyaksha M, Gill M. Utilizing Machine Learning for Image Quality Assessment for Reflectance Confocal Microscopy. J Invest Dermatol 2020; 140:1214-1222. [PMID: 31838127 PMCID: PMC7967900 DOI: 10.1016/j.jid.2019.10.018] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2019] [Revised: 10/09/2019] [Accepted: 10/16/2019] [Indexed: 10/25/2022]
Abstract
In vivo reflectance confocal microscopy (RCM) enables clinicians to examine lesions' morphological and cytological information in epidermal and dermal layers while reducing the need for biopsies. As RCM is being adopted more widely, the workflow is expanding from real-time diagnosis at the bedside to include a capture, store, and forward model with image interpretation and diagnosis occurring offsite, similar to radiology. As the patient may no longer be present at the time of image interpretation, quality assurance is key during image acquisition. Herein, we introduce a quality assurance process by means of automatically quantifying diagnostically uninformative areas within the lesional area by using RCM and coregistered dermoscopy images together. We trained and validated a pixel-level segmentation model on 117 RCM mosaics collected by international collaborators. The model delineates diagnostically uninformative areas with 82% sensitivity and 93% specificity. We further tested the model on a separate set of 372 coregistered RCM-dermoscopic image pairs and illustrate how the results of the RCM-only model can be improved via a multimodal (RCM + dermoscopy) approach, which can help quantify the uninformative regions within the lesional area. Our data suggest that machine learning-based automatic quantification offers a feasible objective quality control measure for RCM imaging.
Collapse
Affiliation(s)
- Kivanc Kose
- Dermatology Service, Memorial Sloan Kettering Cancer Center, New York, New York, USA.
| | - Alican Bozkurt
- Electrical and Computer Engineering, Northeastern University, Boston, Massachusetts, USA
| | | | - Dana H Brooks
- Electrical and Computer Engineering, Northeastern University, Boston, Massachusetts, USA
| | - Jennifer G Dy
- Electrical and Computer Engineering, Northeastern University, Boston, Massachusetts, USA
| | - Milind Rajadhyaksha
- Dermatology Service, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Melissa Gill
- Department of Pathology, SUNY Downstate Medical Center, Brooklyn, New York, USA; SkinMedical Research and Diagnostics, PLLC, Dobbs Ferry, New York, USA
| |
Collapse
|
5
|
REN FEIFEI, WANG ZHAOJUN, QIAN JIA, LIANG YANSHENG, DANG SHIPEI, CAI YANAN, BIANCO PIEROR, YAO BAOLI, LEI MING. Multi-view object topography measurement with optical sectioning structured illumination microscopy. APPLIED OPTICS 2019; 58:6288-6294. [PMID: 31503772 PMCID: PMC9575593 DOI: 10.1364/ao.58.006288] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
Various optical instruments have been developed for three-dimensional (3D) surface topography, including the white light interference, reflectance confocal microscopes, and digital holographic microscopes, etc. However, the steep local slope of objects may cause the light to be reflected in a way that it will not be captured by the objective lens because of the finite collection angle of the objective. To solve this "shadow problem," we report a method to enlarge the collection angle range of optical sectioning structured illumination microscopy by capturing sectioned images of the objects from multiple angle of views. We develop a multi-view image fusion algorithm to reconstruct a single 3D image. Using this method, we detect previously invisible details whose slopes are beyond the collection angle of the objective. The proposed approach is useful for height map measurement and quantitative analyses in a variety of fields, such as biology, materials science, microelectronics, etc.
Collapse
Affiliation(s)
- FEIFEI REN
- China State Key Laboratory of Transient Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - ZHAOJUN WANG
- China State Key Laboratory of Transient Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - JIA QIAN
- China State Key Laboratory of Transient Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - YANSHENG LIANG
- China State Key Laboratory of Transient Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
| | - SHIPEI DANG
- China State Key Laboratory of Transient Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - YANAN CAI
- China State Key Laboratory of Transient Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
- University of Chinese Academy of Sciences, Beijing 100049, China
| | - PIERO R. BIANCO
- Department of Microbiology and Immunology, Department of Biochemistry, Center for Single Molecule Biophysics, 321 Carry Hall, University at Buffalo, Buffalo, New York 14214, USA
| | - BAOLI YAO
- China State Key Laboratory of Transient Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
| | - MING LEI
- China State Key Laboratory of Transient Optics and Photonics, Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi’an 710119, China
- Corresponding author:
| |
Collapse
|
6
|
Robic J, Perret B, Nkengne A, Couprie M, Talbot H. Three-dimensional conditional random field for the dermal-epidermal junction segmentation. J Med Imaging (Bellingham) 2019; 6:024003. [PMID: 31065567 PMCID: PMC6487290 DOI: 10.1117/1.jmi.6.2.024003] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2018] [Accepted: 04/04/2019] [Indexed: 01/18/2023] Open
Abstract
The segmentation of the dermal-epidermal junction (DEJ) in in vivo confocal images represents a challenging task due to uncertainty in visual labeling and complex dependencies between skin layers. We propose a method to segment the DEJ surface, which combines random forest classification with spatial regularization based on a three-dimensional conditional random field (CRF) to improve the classification robustness. The CRF regularization introduces spatial constraints consistent with skin anatomy and its biological behavior. We propose to specify the interaction potentials between pixels according to their depth and their relative position to each other to model skin biological properties. The proposed approach adds regularity to the classification by prohibiting inconsistent transitions between skin layers. As a result, it improves the sensitivity and specificity of the classification results.
Collapse
Affiliation(s)
- Julie Robic
- Clarins Laboratories, Pontoise, France
- Université Paris-Est, LIGM UMR 8049, ESIEE Paris, France
| | | | | | - Michel Couprie
- Université Paris-Est, LIGM UMR 8049, ESIEE Paris, France
| | - Hugues Talbot
- CentraleSupelec, Centre de Vision Numérique, INRIA, France
| |
Collapse
|
7
|
Halimi A, Batatia H, Le Digabel J, Josse G, Tourneret JY. Wavelet-based statistical classification of skin images acquired with reflectance confocal microscopy. BIOMEDICAL OPTICS EXPRESS 2017; 8:5450-5467. [PMID: 29296480 PMCID: PMC5745095 DOI: 10.1364/boe.8.005450] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/29/2017] [Revised: 10/18/2017] [Accepted: 10/19/2017] [Indexed: 06/07/2023]
Abstract
Detecting skin lentigo in reflectance confocal microscopy images is an important and challenging problem. This imaging modality has not yet been widely investigated for this problem and there are a few automatic processing techniques. They are mostly based on machine learning approaches and rely on numerous classical image features that lead to high computational costs given the very large resolution of these images. This paper presents a detection method with very low computational complexity that is able to identify the skin depth at which the lentigo can be detected. The proposed method performs multiresolution decomposition of the image obtained at each skin depth. The distribution of image pixels at a given depth can be approximated accurately by a generalized Gaussian distribution whose parameters depend on the decomposition scale, resulting in a very-low-dimension parameter space. SVM classifiers are then investigated to classify the scale parameter of this distribution allowing real-time detection of lentigo. The method is applied to 45 healthy and lentigo patients from a clinical study, where sensitivity of 81.4% and specificity of 83.3% are achieved. Our results show that lentigo is identifiable at depths between 50μm and 60μm, corresponding to the average location of the the dermoepidermal junction. This result is in agreement with the clinical practices that characterize the lentigo by assessing the disorganization of the dermoepidermal junction.
Collapse
Affiliation(s)
- Abdelghafour Halimi
- University of Toulouse, IRIT-INPT, 2 rue Camichel, BP 7122, 31071 Toulouse cedex 7,
France
| | - Hadj Batatia
- University of Toulouse, IRIT-INPT, 2 rue Camichel, BP 7122, 31071 Toulouse cedex 7,
France
| | - Jimmy Le Digabel
- Centre de Recherche sur la Peau, Pierre Fabre Dermo-Cosmétique, 2 rue Viguerie, 31025 Toulouse Cedex 3, France
| | - Gwendal Josse
- Centre de Recherche sur la Peau, Pierre Fabre Dermo-Cosmétique, 2 rue Viguerie, 31025 Toulouse Cedex 3, France
| | - Jean Yves Tourneret
- University of Toulouse, IRIT-INPT, 2 rue Camichel, BP 7122, 31071 Toulouse cedex 7,
France
| |
Collapse
|