Classification of morphologically similar algae and cyanobacteria using Mueller matrix imaging and convolutional neural networks

Computer Vision Based Deep Learning Approach for the Detection and Classification of Algae Species Using Microscopic Images

The natural phenomenon of harmful algae bloom (HAB) has a bad impact on the quality of pure and freshwater. It increases the risk to human health, water bodies and overall aquatic ecosystem. It is necessary to continuously monitor and perform proper action against HAB. The inspection of algae blooms by using conventional methods, like algae detection under microscopes, is a difficult, expensive, and time-consuming task, however, computer vision-based deep learning models play a vital role in identifying and detecting harmful algae growth in aquatic ecosystems and water reservoirs. Many studies have been conducted to address harmful algae growth by using a CNN based model, however, the YOLO model is considered more accurate in identifying the algae. This advanced deep learning method is extensively used to detect algae and classify them according to their corresponding category. In this study, we used various versions of the convolution neural network (CNN) based on the You Only Look Once (YOLO) model. Recently YOLOv5 has been getting more attention due to its performance in real-time object detection. We performed a series of experiments on our custom microscopic images dataset by using YOLOv3, YOLOv4, and YOLOv5 to detect and classify the harmful algae bloom (HAB) of four classes. We used pre-processing techniques to enhance the quantity of data. The mean average precision (mAP) of YOLOv3, YOLOv4, and YOLO v5 is 75.3%, 83.0%, and 91.0% respectively. For the monitoring of algae bloom in freshwater, computer-aided based systems are very helpful and effective. To the best of our knowledge, this work is pioneering in the AI community for applying the YOLO models to detect algae and classify from microscopic images.

Combined Mueller matrix imaging and artificial intelligence classification framework for Hepatitis B detection

SIGNIFICANCE

The combination of polarized imaging with artificial intelligence (AI) technology has provided a powerful tool for performing an objective and precise diagnosis in medicine.

AIM

An approach is proposed for the detection of hepatitis B (HB) virus using a combined Mueller matrix imaging technique and deep learning method.

APPROACH

In the proposed approach, Mueller matrix imaging polarimetry is applied to obtain 4 × 4 Mueller matrix images of 138 HBsAg-containing (positive) serum samples and 136 HBsAg-free (negative) serum samples. The kernel estimation density results show that, of the 16 Mueller matrix elements, elements M 22 and M 33 provide the best discriminatory power between the positive and negative samples.

RESULTS

As a result, M 22 and M 33 are taken as the inputs to five different deep learning models: Xception, VGG16, VGG19, ResNet 50, and ResNet150. It is shown that the optimal classification accuracy (94.5%) is obtained using the VGG19 model with element M 22 as the input.

CONCLUSIONS

Overall, the results confirm that the proposed hybrid Mueller matrix imaging and AI framework provides a simple and effective approach for HB virus detection.

Detection of morphological changes caused by chemical stress in the cyanobacterium Planktothrix agardhii using convolutional neural networks

The presence of harmful algal bloom in many reservoirs around the world, alongside the lack of sanitation law/ordinance regarding cyanotoxin monitoring (particularly in developing countries), create a scenario in which the local population could potentially chronically consume cyanotoxin-contaminated waters. Therefore, it is crucial to develop low cost tools to detect possible systems failures and consequent toxin release inferred by morphological changes of cyanobacteria in the raw water. This paper aimed to look for the best combination of convolutional neural network (CNN), optimizer and image segmentation technique to differentiate P. agardhii trichomes before and after chemical stress caused by the addition of hydrogen peroxide. This method takes a step towards accurate monitoring of cyanobacteria in the field without the need for a mobile lab. After testing three different network architectures (AlexNet, 3ConvLayer and 2ConvLayer), four different optimizers (Adam, Adagrad, RMSProp and SDG) and five different image segmentations methods (Canny Edge Detection, Morphological Filter, HP filter, GrabCut and Watershed), the combination 2ConvLayer with Adam optimizer and GrabCut segmentation, provided the highest median accuracy (93.33%) for identifying H₂O₂-induced morphological changes in P. agardhii. Our results emphasize the fact that the trichome classification problem can be adequately tackled with a limited number of learned features due to the lack of complexity in micrographs from before and after chemical stress. To the authors' knowledge, this is the first time that CNNs were applied to detect morphological changes in cyanobacteria caused by chemical stress. Thus, it is a significant step forward in developing low cost tools based on image recognition, to shield water consumers, especially in the poorest regions, against cyanotoxin-contaminated water.

MuellerNet: a hybrid 3D-2D CNN for cell classification with Mueller matrix images

Different from conventional microimaging techniques, polarization imaging can generate multiple polarization images in a single perspective by changing the polarization angle. However, how to efficiently fuse the information in these multiple polarization images by a convolutional neural network (CNN) is still a challenging problem. In this paper, we propose a hybrid 3D-2D convolutional neural network called MuellerNet, to classify biological cells with Mueller matrix images (MMIs). The MuellerNet includes a normal stream and a polarimetric stream, in which the first Mueller matrix image is taken as the input of normal stream, and the rest MMIs are stacked to form the input of a polarimetric stream. The normal stream is mainly constructed with a backbone network and, in the polarimetric stream, the attention mechanism is used to adaptively assign weights to different convolutional maps. To improve the network's discrimination, a loss function is introduced to simultaneously optimize parameters of the two streams. Two Mueller matrix image datasets are built, which include four types of breast cancer cells and three types of algal cells, respectively. Experiments are conducted on these two datasets with many well-known and recent networks. Results show that the proposed network efficiently improves the classification accuracy and helps to find discriminative features in MMIs.

Drivers of harmful algal blooms in coastal areas of Eastern Mediterranean: a machine learning methodological approach

Harmful algal species are present in the Mediterranean Sea and are often associated with toxic events affecting the nearby coastal zones. The presence of 18 marine microalgae, at genus level, associated with potentially harmful characteristics was predicted using a number of machine learning techniques based exclusively on a small set of abiotic variables, already identified as drivers of blooms. Random Forest (RF) algorithm achieved the best predictive performance by correctly identifying the presence of most genera with a mean of 89.2% of total samples. Although, RF has shown lower predictive performance for genera present in a low number of samples, its predictive power remains at least "fair' in these cases. The main tree-based advantage of RF was thereafter used to assess the importance of the input variables in predicting the presence of the algal genera. Temperature had the most powerful effect on genera's presences, although this effect varies among genera. Finally, the genera were clustered based on their response to the considered abiotic variables and common trends in an ecological context were identified.

Polarization Measurements and Evaluation Based on Multidimensional Polarization Indices Applied in Analyzing Atmospheric Particulates

Online identification and characterization of suspended aerosols can provide a scientific basis for understanding aerosol transformations, quantitatively evaluating the impacts on air quality, public health, and the source apportionment of different atmospheric particulate matters. In this study, we confirm the validity of our developed high-throughput multi-angle polarized scattering vector detection of aerosols and multidimensional polarization scattering index systems. By observation of the mean values, variance, and Wilk’s Lambda of multidimensional polarization indices for different aerosol types, the polarization index shows unique characterization abilities for aerosol properties, and the optimal combination of polarization indices can always be found for a specific aerosol category with a high resolution and discrimination. Clearly, the multidimensional polarization indices of individual aerosols are more suitable for online and real-time aerosol identification and even help to explain the in situ microphysical characteristics of aerosols or evaluate the dynamic evolution of aerosols.

Auto-detection of cervical collagen and elastin in Mueller matrix polarimetry microscopic images using K-NN and semantic segmentation classification

We propose an approach for discriminating fibrillar collagen fibers from elastic fibers in the mouse cervix in Mueller matrix microscopy using convolutional neural networks (CNN) and K-nearest neighbor (K-NN) for classification. Second harmonic generation (SHG), two-photon excitation fluorescence (TPEF), and Mueller matrix polarimetry images of the mice cervix were collected with a self-validating Mueller matrix micro-mesoscope (SAMMM) system. The components and decompositions of each Mueller matrix were arranged as individual channels of information, forming one 3-D voxel per cervical slice. The classification algorithms analyzed each voxel and determined the amount of collagen and elastin, pixel by pixel, on each slice. SHG and TPEF were used as ground truths. To assess the accuracy of the results, mean-square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM) were used. Although the training and testing is limited to 11 and 5 cervical slices, respectively, MSE accuracy was above 85%, SNR was greater than 40 dB, and SSIM was larger than 90%.

Classification of marine microalgae using low-resolution Mueller matrix images and convolutional neural network

In this paper, we used a convolutional neural network to study the classification of marine microalgae by using low-resolution Mueller matrix images. Mueller matrix images of 12 species of algae from 5 families were measured by a Mueller matrix microscopy with an LED light source at 514 nm wavelength. The data sets of seven resolution levels were generated by the bicubic interpolation algorithm. We conducted two groups of classification experiments; one group classified the algae into 12 classes according to species category, and the other group classified the algae into 5 classes according to family category. In each group of classification experiments, we compared the classification results of the Mueller matrix images with those of the first element (M11) images. The classification accuracy of Mueller matrix images declines gently with the decrease of image resolution, while the accuracy of M11 images declines sharply. The classification accuracy of Mueller matrix images is higher than that of M11 images at each resolution level. At the lowest resolution level, the accuracy of 12-class classification and 5-class classification of full Mueller matrix images is 29.89% and 35.83% higher than those of M11 images, respectively. In addition, we also found that the polarization information of different species had different contributions to the classification. These results show that the polarization information can greatly improve the classification accuracy of low-resolution microalgal images.

Predicting Thioflavin Fluorescence of Retinal Amyloid Deposits Associated With Alzheimer's Disease from Their Polarimetric Properties

Purpose

To use machine learning in those with brain amyloid to predict thioflavin fluorescence (indicative of amyloid) of retinal deposits from their interactions with polarized light.

Methods

We imaged 933 retinal deposits in 28 subjects with post mortem evidence of brain amyloid using thioflavin fluorescence and polarization sensitive microscopy. Means and standard deviations of 14 polarimetric properties were input to machine learning algorithms. Two oversampling strategies were applied to overcome data imbalance. Three machine learning algorithms: linear discriminant analysis, supporting vector machine, and random forest (RF) were trained to predict thioflavin positive deposits. For each method; accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve were computed.

Results

For the polarimetric positive deposits, using 1 oversampling method, RF had the highest area under the receiver operating characteristic curve (0.986), which was not different from that with the second oversampling method. RF had 95% accuracy, 94% sensitivity, and 97% specificity. After including deposits with no polarimetric signals, polarimetry correctly predicted 93% of thioflavin positive deposits. Linear retardance and linear anisotropy were the dominant polarimetric properties in RF with 1 oversampling method, and no polarimetric properties were dominant in the second method.

Conclusions

Thioflavin positivity of retinal amyloid deposits can be predicted from their images in polarized light. Polarimetry is a promising dye-free method of detecting amyloid deposits in ex vivo retinal tissue. Further testing is required for translation to live eye imaging.

Translational Relevance

This dye-free method distinguishes retinal amyloid deposits, a promising biomarker of Alzheimer's disease, in human retinas imaged with polarimetry.

Probing the Cyanobacterial Microcystis Gas Vesicles after Static Pressure Treatment: A Potential In Situ Rapid Method

The vertical migration trend of cyanobacterial cells with gas vesicles in water ecosystems can reflect the changes in the natural environment, such as temperature, nutrients, light conditions, etc. The static pressure treatment is one of the most important approaches to study the properties of the cyanobacterial cell and its gas vesicles. In this paper, a polarized light scattering method is used to probe the collapse and regeneration of the cyanobacterial gas vesicles exposed to different static pressures. During the course, both the axenic and wild type strain of cyanobacterial Microcystis were first treated with different static pressures and then recovered on the normal light conditions. Combining the observation of transmission electron microscopy and floating-sinking photos, the results showed that the collapse and regeneration of the cyanobacterial gas vesicles exposed to different static pressures can be characterized by the polarization parameters. The turbidity as a traditional indicator of gas vesicles but subjected to the concentration of the sample was also measured and found to be correlated with the polarization parameters. More analysis indicated that the polarization parameters are more sensitive and characteristic. The polarized light scattering method can be used to probe the cyanobacterial gas vesicles exposed to different static pressures, which has the potential to provide an in situ rapid and damage-free monitoring tool for observing the vertical migration of cyanobacterial cells and forecasting cyanobacterial blooms.

Multi-Target Deep Learning for Algal Detection and Classification

Water quality has a direct impact on industry, agriculture, and public health. Algae species are common indicators of water quality. It is because algal communities are sensitive to changes in their habitats, giving valuable knowledge on variations in water quality. However, water quality analysis requires professional inspection of algal detection and classification under microscopes, which is very time-consuming and tedious. In this paper, we propose a novel multi-target deep learning framework for algal detection and classification. Extensive experiments were carried out on a large-scale colored microscopic algal dataset. Experimental results demonstrate that the proposed method leads to the promising performance on algal detection, class identification and genus identification.

Characterization of physiological states of the suspended marine microalgae using polarized light scattering

Physiological states of marine microalgal cells can influence photosynthesis efficiency, which affects approximately half of global carbon fixation. The detection of the algae physiological profiles is important for marine ecology and economy. In this paper, we propose a polarized light-scattering method to detect sensitive changes in the physiological states of the suspended marine microalgal cells. Our experimental setup is designed to measure the scattered polarization parameters of the cells suspended individually in the seawater. Two species of microalgal cells cultured in the laboratory were measured for several days. Experimental results showed that both species display distinctive changes in their polarized photon scattering features corresponding to changes in their physiological states. The changes are far more prominent than those displayed in unpolarized light scattering. Microscopy observations, simulations for microspheres of different diameters and refractive indices, or different shapes, indicated that the polarization features of the scattered photons are sensitive to the submicrometer microstructures of the cells. This study demonstrates the potential of the polarized light-scattering technique to characterize the physiological states of suspended marine microalgae.

Algal Morphological Identification in Watersheds for Drinking Water Supply Using Neural Architecture Search for Convolutional Neural Network

An excessive increase in algae often has various undesirable effects on drinking water supply systems, thus proper management is necessary. Algal monitoring and classification is one of the fundamental steps in the management of algal blooms. Conventional microscopic methods have been most widely used for algal classification, but such approaches are time-consuming and labor-intensive. Thus, the development of alternative methods for rapid, but reliable algal classification is essential where an advanced machine learning technique, known as deep learning, is considered to provide a possible approach for rapid algal classification. In recent years, one of the deep learning techniques, namely the convolutional neural network (CNN), has been increasingly used for image classification in various fields, including algal classification. However, previous studies on algal classification have used CNNs that were arbitrarily chosen, and did not explore possible CNNs fitting algal image data. In this paper, neural architecture search (NAS), an automatic approach for the design of artificial neural networks (ANN), is used to find a best CNN model for the classification of eight algal genera in watersheds experiencing algal blooms, including three cyanobacteria (Microcystis sp., Oscillatoria sp., and Anabaena sp.), three diatoms (Fragilaria sp., Synedra sp., and two green algae (Staurastrum sp. and Pediastrum sp.). The developed CNN model effectively classified the algal genus with an F1-score of 0.95 for the eight genera. The results indicate that the CNN models developed from NAS can outperform conventional CNN development approaches, and would be an effective tool for rapid operational responses to algal bloom events. In addition, we introduce a generic framework that provides a guideline for the development of the machine learning models for algal image analysis. Finally, we present the experimental results from the real-world environments using the framework and NAS.

A multiscale Mueller polarimetry module for a stereo zoom microscope

Mueller polarimetry is a quantitative polarized light imaging modality that is capable of label-free visualization of tissue pathology, does not require extensive sample preparation, and is suitable for wide-field tissue analysis. It holds promise for selected applications in biomedicine, but polarimetry systems are often constrained by limited end-user accessibility and/or long-imaging times. In order to address these needs, we designed a multiscale-polarimetry module that easily couples to a commercially available stereo zoom microscope. This paper describes the module design and provides initial polarimetry imaging results from a murine preclinical breast cancer model and human breast cancer samples. The resultant polarimetry module has variable resolution and field of view, is low-cost, and is simple to switch in or out of a commercial microscope. The module can reduce long imaging times by adopting the main imaging approach used in pathology: scanning at low resolution to identify regions of interest, then at high resolution to inspect the regions in detail. Preliminary results show how the system can aid in region of interest identification for pathology, but also highlight that more work is needed to understand how tissue structures of pathological interest appear in Mueller polarimetry images across varying spatial zoom scales.

Study on polarization scattering applied in aerosol recognition in the air

In this work, we present an in situ online aerosol recognition scheme by synchronized parallel polarization scattering analysis. By theoretical simulations, we select the feasible scattering angles and evaluate the potential of Stokes parameters to identify aerosols. Correspondingly, we develop a measurement system based on multi-angle optical scattering and multidimensional polarization analyzing technique. We construct two index groups based on non-normalized and normalized polarization parameters respectively, and employ their frequency distribution histograms instead of the simple average values to identify and classify different types of aerosols. The experimental verification confirms a future way of a multi-dimensional polarization parameter group applied in a fast and effective air pollutants monitoring.

Combining high-throughput imaging flow cytometry and deep learning for efficient species and life-cycle stage identification of phytoplankton

BACKGROUND

Phytoplankton species identification and counting is a crucial step of water quality assessment. Especially drinking water reservoirs, bathing and ballast water need to be regularly monitored for harmful species. In times of multiple environmental threats like eutrophication, climate warming and introduction of invasive species more intensive monitoring would be helpful to develop adequate measures. However, traditional methods such as microscopic counting by experts or high throughput flow cytometry based on scattering and fluorescence signals are either too time-consuming or inaccurate for species identification tasks. The combination of high qualitative microscopy with high throughput and latest development in machine learning techniques can overcome this hurdle.

RESULTS

In this study, image based cytometry was used to collect ~ 47,000 images for brightfield and Chl a fluorescence at 60× magnification for nine common freshwater species of nano- and micro-phytoplankton. A deep neuronal network trained on these images was applied to identify the species and the corresponding life cycle stage during the batch cultivation. The results show the high potential of this approach, where species identity and their respective life cycle stage could be predicted with a high accuracy of 97%.

CONCLUSIONS

These findings could pave the way for reliable and fast phytoplankton species determination of indicator species as a crucial step in water quality assessment.

Detection of non-small cell lung cancer cells based on microfluidic polarization microscopic image analysis

In early diagnosis of lung cancer, a polarization microscopy is a powerful tool to obtain the optical information of biological tissues. In this paper, a new microfluidic polarization imaging and analysis method was proposed for the detection and classification of cancer-associated fibroblasts and the two kinds of non-small cell lung cancer cells, A549 and H322. A polarizing microscopy system was constructed based on a commercial microscope to obtain 3*3 Mueller matrix of cells. Based on the Muller matrix decomposition algorithm and analysis in spatial domain and frequency domain, appropriate classification parameters were selected for the characterization of different polarization characteristics of cells. Finally, the logistic regression models based on machine learning were applied to determine optimal feature parameters and classify cells. This method integrated the morphological information of the cells, and the polarization characteristics of the cells in different polarization states. It is for the first time that the polarization microscopic image analysis method has been applied to the detection and classification of non-small cell lung cancer cells. The results show that the presented microfluidic polarization microscopic image analysis method could classify cells effectively. Compared with the Muller matrix measurement and calculation methods, the method proposed in this paper was greatly simplified in both the acquisition of polarized images and the analysis and processing of polarized images.

Differentiation of suspended particles by polarized light scattering at 120°

Probing suspended particles in seawater, such as microalgae, microplastics and silts, is very important for environmental monitoring and ecological research. We propose a method based on polarized light scattering to differentiate different suspended particles massively and rapidly. The optical path follows a similar design of a commonly used marine instrument, BB9, which records backscattering of non-polarized light at 120°. In addition, polarization elements are added to the incident and scattering path for taking polarization measurements. Experiments with polystyrene microspheres, porous polystyrene microspheres, silicon dioxide microspheres, and different marine microalgae show that by carefully choosing the incident polarization state and analyzing the polarization features of the scattered light at 120°, these particles can be effectively differentiated. Simulations based on the Mie scattering theory and discrete dipole approximation (DDA) have also been conducted for particles of different sizes, shapes and refractive indices, which help to understand the relationship between the polarization features and the physical properties of the particles. The laboratory system may serve as a prove-of-concept prototype of new instrumentations for applications on board or even with submersibles.

Polarimetric learning: a Siamese approach to learning distance metrics of algal Mueller matrix images

Polarimetric measurements are becoming increasingly accurate and fast to perform in modern applications. However, analysis on the polarimetric data usually suffers from its high-dimensional nature spatially, temporally, or spectrally. This paper associates polarimetric techniques with metric learning algorithms, namely, polarimetric learning, by introducing a distance metric learning method called Siamese network that aims to learn good distance metrics of algal Mueller matrix images in low-dimensional feature spaces. As an experimental example, 12,162 Mueller matrix images of eight algal species are measured via a forward Mueller matrix microscope. Eight classical metric learning algorithms, including principle component analysis, multidimensional scaling, isometric feature mapping, t-distributed stochastic neighbor embedding, Laplacian eigenmaps, locally linear embedding, linear discriminant analysis, and metric learning to rank, are considered, by which the algal Mueller matrix images are mapped to two-dimensional (2D) feature spaces with different distance metrics. Support-vector-machine-based holdout sample classification accuracies of the 2D feature vectors are provided in a supervised manner for quantitative comparisons of the low-dimensional distance metrics, including the results of the eight metric learning algorithms and 16 Siamese architectures with varying convolution, inception, and full connection modules. This study shows that the Siamese approach is an effective metric learning algorithm that can adaptively extract features exhibiting empirical correlations with the fast-axis-orientation-dependent and spatially variant algal retardance induced by the algal microstructures.