From colour to tissue histology: Physics-based interpretation of images of pigmented skin lesions

Adversarial multimodal fusion with attention mechanism for skin lesion classification using clinical and dermoscopic images

Accurate skin lesion diagnosis requires a great effort from experts to identify the characteristics from clinical and dermoscopic images. Deep multimodal learning-based methods can reduce intra- and inter-reader variability and improve diagnostic accuracy compared to the single modality-based methods. This study develops a novel method, named adversarial multimodal fusion with attention mechanism (AMFAM), to perform multimodal skin lesion classification. Specifically, we adopt a discriminator that uses adversarial learning to enforce the feature extractor to learn the correlated information explicitly. Moreover, we design an attention-based reconstruction strategy to encourage the feature extractor to concentrate on learning the features of the lesion area, thus, enhancing the feature vector from each modality with more discriminative information. Unlike existing multimodal-based approaches, which only focus on learning complementary features from dermoscopic and clinical images, our method considers both correlated and complementary information of the two modalities for multimodal fusion. To verify the effectiveness of our method, we conduct comprehensive experiments on a publicly available multimodal and multi-task skin lesion classification dataset: 7-point criteria evaluation database. The experimental results demonstrate that our proposed method outperforms the current state-of-the-art methods and improves the average AUC score by above 2% on the test set.

Melanoma Detection by Means of Multiple Instance Learning

We present an application to melanoma detection of a multiple instance learning (MIL) approach, whose objective, in the binary case, is to discriminate between positive and negative sets of items. In the MIL terminology these sets are called bags and the items inside the bags are called instances. Under the hypothesis that a bag is positive if at least one of its instances is positive and it is negative if all its instances are negative, the MIL paradigm fits very well with images classification, since an image (bag) is in general classified on the basis of some its subregions (instances). In this work we have applied a MIL algorithm on some clinical data constituted by color dermoscopic images, with the aim to discriminate between melanomas (positive images) and common nevi (negative images). In comparison with standard classification approaches, such as the well known support vector machine, our method performs very well in terms both of accuracy and sensitivity. In particular, using a leave-one-out validation on a data set constituted by 80 melanomas and 80 common nevi, we have obtained the following results: accuracy = 92.50%, sensitivity = 97.50% and specificity = 87.50%. Since the results appear promising, we conclude that a MIL technique could be at the basis of more sophisticated tools useful to physicians in melanoma detection.

A micro-perfusion bioreactor for on line investigation of ECM remodeling under hydrodynamic and biochemical stimulation

Tissue-on-chip (TOC) systems aim at replicating complex biological dynamics in vitro with the potential either to improve the understanding of human biology or to develop more accurate therapeutic strategies. To replicate faithfully the intricate interrelationships between cells and their surrounding microenvironment, the three-dimensional (3D) tissue model must possess a responsive extracellular matrix (ECM). ECM remodeling plays a pivotal role in guiding cells and tissues functions and such aspect is somewhat denied during in vitro studies. For this purpose, we fabricated a micro-perfusion bioreactor capable to sustain the viability of 3D engineered tissue models recapitulating the process of the native ECM deposition and assembly. Engineered human dermis micro-tissue precursors (HD-μTP) were used as building blocks to generate a final tissue. HD-μTP were loaded in the perfusion space of the micro-perfusion bioreactor and, under the superimposition of different fluid dynamic regimes and biochemical stimulation, they synthesized new collagen proteins that were, then, assembled in the perfusion space forming a continuum of cells embedded in their own ECM. The micro-perfusion bioreactor was fabricated to allow the on-line monitoring of the oxygen consumption and the assembly of the newly formed collagen network via real time acquisition of the second harmonic generation (SHG) signal. The possibility to detect the collagen reorganization due to both fluid dynamic and biochemical stimulation, let us to define the optimal perfusion configuration in order to obtain a TOC system based on an endogenous and responsive ECM.

Practical application of new technologies for melanoma diagnosis: Part I. Noninvasive approaches

Confirming a diagnosis of cutaneous melanoma requires obtaining a skin biopsy specimen. However, obtaining numerous biopsy specimens-which often happens in patients with increased melanoma risk-is associated with significant cost and morbidity. While some melanomas are easily recognized by the naked eye, many can be difficult to distinguish from nevi, and therefore there is a need and opportunity to develop new technologies that can facilitate clinical examination and melanoma diagnosis. In part I of this 2-part continuing medical education article, we will review the practical applications of emerging technologies for noninvasive melanoma diagnosis, including mobile (smartphone) applications, multispectral imaging (ie, MoleMate and MelaFind), and electrical impedance spectroscopy (Nevisense).

Physiological characterization of skin lesion using non-linear random forest regression model

The current diagnostic technique for melanoma solely relies on the surface level of skin and under-skin information is neglected. Since physiological features of skin such as melanin are closely related to development of melanoma, the non-linear physiological feature extraction model based on random forest regression is proposed. The proposed model characterizes the concentration of eumelanin and pheomelanin from standard camera images or dermoscopic images, which are conventionally used for diagnosis of melanoma. For the validation, the phantom study and the separability test using clinical images were conducted and compared against the state-of-the art non-linear and linear feature extraction models. The results showed that the proposed model outperformed other comparing models in phantom and clinical experiments. Promising results show that the quantitative characterization of skin features, which is provided by the proposed method, can allow dermatologists and clinicians to make a more accurate and improved diagnosis of melanoma.

Evaluation of a novel skin tone meter and the correlation between Fitzpatrick skin type and skin color

Background and objective:

To evaluate a novel skin tone meter (STM) to categorize skin tones into one of the six categories of the Fitzpatrick skin type (FST) classification system, thus optimizing safety in light-based dermatological procedures. This numerical classification method measures several components; principally the reaction of human skin to ultraviolet (UV) light exposure, which is used to help predict skin response in laser and intense pulsed light (IPL) treatments.

Materials and methods:

Two-hundred twenty volunteers of varying ethnic origin, age and gender were enrolled in a preliminary study. The subjects’ Fitzpatrick skin type was ascertained by a standardized questionnaire that determined their reaction to significant sunlight exposure. A calibrated prototype STM device (consisting of an optical head at 460 nm, detector, microprocessor, and a liquid crystal display) was used to measure the subjects’ inner arm skin; which typically has little UV exposure and minimal hair, and compared the obtained value with measurements taken from a skin color chart and digital photographs. To evaluate device performance (within subject) across different skin states, a section of skin from the inner arm of a sub-group of eight volunteers was marked into test areas using a template. The skin in each area was then prepared (

Results:

There was a consistent trend between the STM prototype and the assessed skin tone derived from a proprietary skin color chart against the measurement on skin across a range of skin conditions.

Conclusion:

The presented preliminary study demonstrated the subjective nature of the FST classification system and the weakness of skin tone self-assessment by an individual, as judged by expert assessors. The FST classification requires an objective measurement to replace the textual description for each skin tone. It may significantly decrease the risk of potential side effects through overtreatment, and extend treatment to a wider patient population with light-based dermatological procedures.

Vein visualization using a smart phone with multispectral Wiener estimation for point-of-care applications

Effective vein visualization is clinically important for various point-of-care applications, such as needle insertion. It can be achieved by utilizing ultrasound imaging or by applying infrared laser excitation and monitoring its absorption. However, while these approaches can be used for vein visualization, they are not suitable for point-of-care applications because of their cost, time, and accessibility. In this paper, a new vein visualization method based on multispectral Wiener estimation is proposed and its real-time implementation on a smart phone is presented. In the proposed method, a conventional RGB camera on a commercial smart phone (i.e., Galaxy Note 2, Samsung Electronics Inc., Suwon, Korea) is used to acquire reflectance information from veins. Wiener estimation is then applied to extract the multispectral information from the veins. To evaluate the performance of the proposed method, an experiment was conducted using a color calibration chart (ColorChecker Classic, X-rite, Grand Rapids, MI, USA) and an average root-mean-square error of 12.0% was obtained. In addition, an in vivo subcutaneous vein imaging experiment was performed to explore the clinical performance of the smart phone-based Wiener estimation. From the in vivo experiment, the veins at various sites were successfully localized using the reconstructed multispectral images and these results were confirmed by ultrasound B-mode and color Doppler images. These results indicate that the presented multispectral Wiener estimation method can be used for visualizing veins using a commercial smart phone for point-of-care applications (e.g., vein puncture guidance).

Assessment of SIAscopy in the triage of suspicious skin tumours

BACKGROUND/PURPOSE

Spectrophotometric Intracutaneous Analysis (SIAscopy) is a non-invasive, computerized technique for the diagnosis of pigmented skin tumours. The analysis is based on the evaluation of skin chromophores, i.e. melanin, haemoglobin and collagen within the epidermis and papillary dermis. Our aim was to assess the diagnostic validity of SIAscopy in the detection of melanoma and non-melanoma skin cancers compared to the clinical-dermoscopic diagnosis and the histopathologic results of the excised lesions.

METHODS

In total, 188 lesions of 180 patients were examined by dermoscopy and SIAscopy. A SIAscopy scoring system was first compared with the clinical-dermoscopic diagnosis and then with the histopathologic diagnosis of the excised lesions.

RESULTS

With respect to the clinical-dermoscopic evaluation, SIAscopy had sensitivity and specificity values of 85.7% and 65.4% respectively. Of the 188 evaluated lesions, 44 were excised with histopathologic examination revealing 31 malignant tumours, including 18 melanomas. With respect to histopathology SIAscopy had a sensitivity of 83.9%. Seven of the 13 benign excised lesions were scored as malignant by SIAscopy resulting in a specificity of 46.1%.

CONCLUSION

SIAscopy cannot replace the standard of care in skin cancer diagnosis, which includes clinical and dermoscopic examination. However, considering that the technique does not require specific training and expertise, it might represent an additional, relatively cost-effective tool to select lesions for referral.

3-D volume reconstruction of skin lesions for melanin and blood volume estimation and lesion severity analysis

Subsurface information about skin lesions, such as the blood volume beneath the lesion, is important for the analysis of lesion severity towards early detection of skin cancer such as malignant melanoma. Depth information can be obtained from diffuse reflectance based multispectral transillumination images of the skin. An inverse volume reconstruction method is presented which uses a genetic algorithm optimization procedure with a novel population initialization routine and nudge operator based on the multispectral images to reconstruct the melanin and blood layer volume components. Forward model evaluation for fitness calculation is performed using a parallel processing voxel-based Monte Carlo simulation of light in skin. Reconstruction results for simulated lesions show excellent volume accuracy. Preliminary validation is also done using a set of 14 clinical lesions, categorized into lesion severity by an expert dermatologist. Using two features, the average blood layer thickness and the ratio of blood volume to total lesion volume, the lesions can be classified into mild and moderate/severe classes with 100% accuracy. The method therefore has excellent potential for detection and analysis of pre-malignant lesions.

Histology image analysis for carcinoma detection and grading

This paper presents an overview of the image analysis techniques in the domain of histopathology, specifically, for the objective of automated carcinoma detection and classification. As in other biomedical imaging areas such as radiology, many computer assisted diagnosis (CAD) systems have been implemented to aid histopathologists and clinicians in cancer diagnosis and research, which have been attempted to significantly reduce the labor and subjectivity of traditional manual intervention with histology images. The task of automated histology image analysis is usually not simple due to the unique characteristics of histology imaging, including the variability in image preparation techniques, clinical interpretation protocols, and the complex structures and very large size of the images themselves. In this paper we discuss those characteristics, provide relevant background information about slide preparation and interpretation, and review the application of digital image processing techniques to the field of histology image analysis. In particular, emphasis is given to state-of-the-art image segmentation methods for feature extraction and disease classification. Four major carcinomas of cervix, prostate, breast, and lung are selected to illustrate the functions and capabilities of existing CAD systems.

Intrinsic melanin and hemoglobin colour components for skin lesion malignancy detection

In this paper we propose a new log-chromaticity 2-D colour space, an extension of previous approaches, which succeeds in removing confounding factors from dermoscopic images: (i) the effects of the particular camera characteristics for the camera system used in forming RGB images; (ii) the colour of the light used in the dermoscope; (iii) shading induced by imaging non-flat skin surfaces; (iv) and light intensity, removing the effect of light-intensity falloff toward the edges of the dermoscopic image. In the context of a blind source separation of the underlying colour, we arrive at intrinsic melanin and hemoglobin images, whose properties are then used in supervised learning to achieve excellent malignant vs. benign skin lesion classification. In addition, we propose using the geometric-mean of colour for skin lesion segmentation based on simple grey-level thresholding, with results outperforming the state of the art.

Quantitative principal component model for skin chromophore mapping using multi-spectral images and spatial priors

We describe a novel reconstruction algorithm based on Principal Component Analysis (PCA) applied to multi-spectral imaging data. Using numerical phantoms, based on a two layered skin model developed previously, we found analytical expressions, which convert qualitative PCA results into quantitative blood volume and oxygenation values, assuming the epidermal thickness to be known. We also evaluate the limits of accuracy of this method when the value of the epidermal thickness is not known. We show that blood volume can reliably be extracted (less than 6% error) even if the assumed thickness deviates 0.04mm from the actual value, whereas the error in blood oxygenation can be as large as 25% for the same deviation in thickness. This PCA based reconstruction was found to extract blood volume and blood oxygenation with less than 8% error, if the underlying structure is known. We then apply the method to in vivo multi-spectral images from a healthy volunteer's lower forearm, complemented by images of the same area using Optical Coherence Tomography (OCT) for measuring the epidermal thickness. Reconstruction of the imaging results using a two layered analytical skin model was compared to PCA based reconstruction results. A point wise correlation was found, showing the proof of principle of using PCA based reconstruction for blood volume and oxygenation extraction.

In vivo characterization of human pigmented lesions by degree of linear polarization image maps using incident linearly polarized light

BACKGROUND AND OBJECTIVE

Melanoma is the most serious form of skin cancer and often appears as an evolving multicolored skin growth. It is well documented that pre-existing atypical or dysplastic nevi can evolve into a melanoma. The development of an in vivo imaging system to characterize benign and malignant nevi has been emphasized to aid in early detection of melanoma. The goal of this study is to utilize a novel Stokes polarimetry imaging (SPI) system for the characterization of pigmented lesions, and to evaluate the SPI system in comparison to dermoscopy and histology images.

STUDY DESIGN/MATERIALS AND METHODS

Linearly polarized light with varying incident polarization angles (IPA) illuminated various types of pigmented lesions. The melanocytic nesting patterns of pigmented lesions were characterized by constructing the degree-of-linear-polarization (DOLP) image map with comparison to dermoscopy and histology. The incident polarized light was filtered by visible filters for spectral imaging and incident deeply penetrating red light was used to correlate the SPI image with histopathological examination.

RESULTS

The DOLP images with varying IPA at different visible wavelengths were used to characterize various kinds of pigmented lesions by showing subsurface melanocytic nesting distribution as well as morphological information with better resolution and contrast. In correlation with dermoscopy and histology, various defining features such as compound, junctional, lentiginous, reticular, globular patterns of melanocytic nests were identified.

CONCLUSION

When imaging pigmented melanocytic lesions, the SPI system with varying IPA at the red light wavelength can better define the melanocytic nesting patterns in both the dermal epidermal junction and the dermis. The SPI system has the potential to be an effective in vivo method of detecting pre-malignant nevi and melanoma.

Spectrophotometric intracutaneous analysis versus dermoscopy for the diagnosis of pigmented skin lesions: prospective, double-blind study in a secondary reference centre

Dermoscopy is considered to be the golden standard for the clinical assessment of pigmented skin lesions. In expert hands, this instrument improves both sensitivity and specificity for the diagnosis of melanoma, however, the outcome is highly dependent on the skills and experience of the examiner. Spectrophotometric intracutaneous analysis (SIAscopy) is a new, commercially available method of analyzing pigmented skin lesions noninvasively. The diagnosis is based on objective features such as the presence of dermal pigment, vascularity of the lesion, and the integrity of collagen. The objective of this study was to examine the usefulness of SIAscopy for the clinical diagnosis of malignant melanoma in a prospective, unbiased manner. We enrolled 65 patients with 83 lesions, where the diagnosis of melanoma could not be ruled out on the basis of the clinical evaluation by a nondermatologist. All lesions were investigated by dermoscopy and SIAscopy and subsequently excised. Histopathologically, 12 lesions were diagnosed as malignant melanoma. Both dermoscopy and SIAscopy overestimated the proportion of possible malignant lesions (n=24 and 41, respectively) and had sensitivities of 92 and 100%, respectively. The specificity of dermoscopy in this study was 81% against 59% for SIAscopy. Our result shows that dermoscopy remains the best diagnostic tool for the preoperative diagnosis of pigmented skin lesions. However, as the SIAscope in addition to the SIAgraph images produces dermoscopic images, it holds the advantages in training and archiving.

Selection of optimal filters for multispectral imaging

Preece and Claridge [IEEE Trans. Pattern Anal. Mach. Intell. 26, 913 (2004)] have proposed a technique for selecting filters for the maximally accurate recovery of object parameters such as chromophore concentrations from a multispectral image of an object. Their selection criteria are derived from an analysis of a model of light propagation in the object and take into account both errors in the modeling process and errors in the image acquisition process, as well as the inherent behavior and structure of the model. We investigate their method on simulated image data and show that filters selected according to their criteria are demonstrably superior to other choices.

Enhanced visualization of histological samples with an adjustable RGB contrast system with application for tissue used in photodynamic therapy

The analysis of histological sections has long been a valuable tool in the pathological studies. The interpretation of tissue conditions, however, relies directly on visual evaluation of tissue slides, which may be difficult to interpret because of poor contrast or poor color differentiation. The Chromatic Contrast Visualization System (CCV) combines an optical microscope with electronically controlled light-emitting diodes (LEDs) in order to generate adjustable intensities of RGB channels for sample illumination. While most image enhancement techniques rely on software post-processing of an image acquired under standard illumination conditions, CCV produces real-time variations in the color composition of the light source itself. The possibility of covering the entire RGB chromatic range, combined with the optical properties of the different tissues, allows for a substantial enhancement in image details. Traditional image acquisition methods do not exploit these visual enhancements which results in poorer visual distinction among tissue structures. Photodynamic therapy (PDT) procedures are of increasing interest in the treatment of several forms of cancer. This study uses histological slides of rat liver samples that were induced to necrosis after being exposed to PDT. Results show that visualization of tissue structures could be improved by changing colors and intensities of the microscope light source. PDT-necrosed tissue samples are better differentiated when illuminated with different color wavelengths, leading to an improved differentiation of cells in the necrosis area. Due to the potential benefits it can bring to interpretation and diagnosis, further research in this field could make CCV an attractive technique for medical applications.

Strategies for efficient virtual microscopy in pathological samples using JPEG2000

This paper describes the design, implementation and validation of a new strategy for efficiently browsing large microscopical images (mega-images). A mega-image is constructed by registering a sequential set of microscopic fields of view, compressed and stored in hard disk using the JPEG2000 standard (J2K). Navigation is accelerated by fully exploiting J2K properties through the introduction of a cache strategy and an optimal delivering of quality information. Cache is introduced at the level of the spatial and resolution dimensions while optimal delivering is implemented on the organisation of minimal information units. Navigation with the conventional use of J2K results in extraction times of about 500 ms. We show that these strategies can improve navigation velocities up to a 30%, while we can efficiently represent high-quality and high-resolution colour images of microscopic specimens.

Modelling and validation of spectral reflectance for the colon

The spectral reflectance of the colon is known to be affected by malignant and pre-malignant changes in the tissue. As part of long-term research on the derivation of diagnostically important parameters characterizing colon histology, we have investigated the effects of the normal histological variability on the remitted spectra. This paper presents a detailed optical model of the normal colon comprising mucosa, submucosa and the smooth muscle layer. Each layer is characterized by five variable histological parameters: the volume fraction of blood, the haemoglobin saturation, the size of the scattering particles, including collagen, the volume fraction of the scattering particles and the layer thickness, and three optical parameters: the anisotropy factor, the refractive index of the medium and the refractive index of the scattering particles. The paper specifies the parameter ranges corresponding to normal colon tissue, including some previously unpublished ones. Diffuse reflectance spectra were modelled using the Monte Carlo method. Validation of the model-generated spectra against measured spectra demonstrated that good correspondence was achieved between the two. The analysis of the effect of the individual histological parameters on the behaviour of the spectra has shown that the spectral variability originates mainly from changes in the mucosa. However, the submucosa and the muscle layer must be included in the model as they have a significant constant effect on the spectral reflectance above 600 nm. The nature of variations in the spectra also suggests that it may be possible to carry out model inversion and to recover parameters characterizing the colon from multi-spectral images. A preliminary study, in which the mucosal blood and collagen parameters were modified to reflect histopathological changes associated with colon cancer, has shown that the spectra predicted by our model resemble measured spectral reflectance of adenocarcinomas. This suggests that an extended model, which incorporates parameters corresponding to an abnormal colon, may be effective for differentiation between normal and cancerous tissues.