Computer vision for high content screening

Real-time semantic segmentation and anomaly detection of functional images for cell therapy manufacturing

BACKGROUND AIMS

Cell therapy is a promising treatment method that uses living cells to address a variety of diseases and conditions, including cardiovascular diseases, neurologic disorders and certain cancers. As interest in cell therapy grows, there is a need to shift to a more efficient, scalable and automated manufacturing process that can produce high-quality products at a lower cost.

METHODS

One way to achieve this is using non-invasive imaging and real-time image analysis techniques to monitor and control the manufacturing process. This work presents a machine learning-based image analysis pipeline that includes semantic segmentation and anomaly detection capabilities.

RESULTS/CONCLUSIONS

This method can be easily implemented even when given a limited dataset of annotated images, is able to segment cells and debris and can identify anomalies such as contamination or hardware failure.

PIFiA: Self-supervised Approach for Protein Functional Annotation from Single-Cell Imaging Data

Fluorescence microscopy data describe protein localization patterns at single-cell resolution and have the potential to reveal whole-proteome functional information with remarkable precision. Yet, extracting biologically meaningful representations from cell micrographs remains a major challenge. Existing approaches often fail to learn robust and noise-invariant features or rely on supervised labels for accurate annotations. We developed PIFiA, (Protein Image-based Functional Annotation), a self-supervised approach for protein functional annotation from single-cell imaging data. We imaged the global yeast ORF-GFP collection and applied PIFiA to generate protein feature profiles from single-cell images of fluorescently tagged proteins. We show that PIFiA outperforms existing approaches for molecular representation learning and describe a range of downstream analysis tasks to explore the information content of the feature profiles. Specifically, we cluster extracted features into a hierarchy of functional organization, study cell population heterogeneity, and develop techniques to distinguish multi-localizing proteins and identify functional modules. Finally, we confirm new PIFiA predictions using a colocalization assay, suggesting previously unappreciated biological roles for several proteins. Paired with a fully interactive website (https://thecellvision.org/pifia/), PIFiA is a resource for the quantitative analysis of protein organization within the cell.

Live cell microscopy: From image to insight

Live-cell microscopy is a powerful tool that can reveal cellular behavior as well as the underlying molecular processes. A key advantage of microscopy is that by visualizing biological processes, it can provide direct insights. Nevertheless, live-cell imaging can be technically challenging and prone to artifacts. For a successful experiment, many careful decisions are required at all steps from hardware selection to downstream image analysis. Facing these questions can be particularly intimidating due to the requirement for expertise in multiple disciplines, ranging from optics, biophysics, and programming to cell biology. In this review, we aim to summarize the key points that need to be considered when setting up and analyzing a live-cell imaging experiment. While we put a particular focus on yeast, many of the concepts discussed are applicable also to other organisms. In addition, we discuss reporting and data sharing strategies that we think are critical to improve reproducibility in the field.

Research on Face Recognition Algorithm Based on Image Processing

While network technology is convenient for our daily life, the problems that are exposed are also endless. The most important thing for everyone is information security. In order to improve the security level of network information and identify and detect faces, the method used in this paper has improved compared with the traditional AdaBoost method and skin color method. AdaBoost detection is performed on the image, which reduces the probability of false detection. The experiment compares the experimental results of the AdaBoost method, the skin color method and the skin color + AdaBoost method. All operations in the KPCA and KFDA algorithms are performed by the inner product kernel function defined in the original space, and no specific non-linear mapping function is involved.The full name of KPCA is kernel principal component analysis. The full name of KFDA is kernel Fisher discriminant analysis. Combining the zero-space method kernel discriminant analysis method improves the ability of discriminant analysis to extract non-linear features. Through the secondary extraction of PCA features, a better recognition result than the PCA method is obtained. This paper also proposes a zero-space based Fisher discriminant analysis method. Experiments show that the zero-space-based method makes full use of the useful discriminant information in the zero space of the intraclass dispersion matrix, which improves the accuracy of face recognition to some extent.If you choose the polynomial kernel function, when d = 0.8, KPCA has a higher recognition ability. When d = 2, the recognition rate of KFDA and zero space-based KFDA is the largest. For polynomial functions, in general, d = 2.

Imaging-Based Machine Learning Analysis of Patient-Derived Tumor Organoid Drug Response

Three-quarters of compounds that enter clinical trials fail to make it to market due to safety or efficacy concerns. This statistic strongly suggests a need for better screening methods that result in improved translatability of compounds during the preclinical testing period. Patient-derived organoids have been touted as a promising 3D preclinical model system to impact the drug discovery pipeline, particularly in oncology. However, assessing drug efficacy in such models poses its own set of challenges, and traditional cell viability readouts fail to leverage some of the advantages that the organoid systems provide. Consequently, phenotypically evaluating complex 3D cell culture models remains difficult due to intra- and inter-patient organoid size differences, cellular heterogeneities, and temporal response dynamics. Here, we present an image-based high-content assay that provides object level information on 3D patient-derived tumor organoids without the need for vital dyes. Leveraging computer vision, we segment and define organoids as independent regions of interest and obtain morphometric and textural information per organoid. By acquiring brightfield images at different timepoints in a robust, non-destructive manner, we can track the dynamic response of individual organoids to various drugs. Furthermore, to simplify the analysis of the resulting large, complex data files, we developed a web-based data visualization tool, the Organoizer, that is available for public use. Our work demonstrates the feasibility and utility of using imaging, computer vision and machine learning to determine the vital status of individual patient-derived organoids without relying upon vital dyes, thus taking advantage of the characteristics offered by this preclinical model system.

Modelling in Synthesis and Optimization of Active Vaccinal Components

Cancer is the second leading cause of mortality worldwide, behind heart diseases, accounting for 10 million deaths each year. This study focusses on adenocarcinoma, which is a target of a number of anticancer therapies presently being tested in medical and pharmaceutical studies. The innovative study for a therapeutic vaccine comprises the investigation of gold nanoparticles and their influence on the immune response for the annihilation of cancer cells. The model is intended to be realized using Quantitative-Structure Activity Relationship (QSAR) methods, explicitly artificial neural networks combined with fuzzy rules, to enhance automated properties of neural nets with human perception characteristics. Image processing techniques such as morphological transformations and watershed segmentation are used to extract and calculate certain molecular characteristics from hyperspectral images. The quantification of single-cell properties is one of the key resolutions, representing the treatment efficiency in therapy of colon and rectum cancerous conditions. This was accomplished by using manually counted cells as a reference point for comparing segmentation results. The early findings acquired are conclusive for further study; thus, the extracted features will be used in the feature optimization process first, followed by neural network building of the required model.

Deep Phenotypic Cell Classification using Capsule Neural Network

Recent developments in ultra-high-throughput microscopy have created a new generation of cell classification methodologies focused solely on image-based cell phenotypes. These image-based analyses enable morphological profiling and screening of thousands or even millions of single cells at a fraction of the cost. They have been shown to demonstrate the statistical significance required for understanding the role of cell heterogeneity in diverse biologists. However, these single-cell analysis techniques are slow and require expensive genetic/epigenetic analysis. This treatise proposes an innovative DL system based on the newly created capsule networks (CapsNet) architecture. The proposed deep CapsNet model employs "Capsules" for high-level feature abstraction relevant to the cell category. Experiments demonstrate that our proposed system can accurately classify different types of cells based on phenotypic label-free bright-field images with over 98.06% accuracy and that deep CapsNet models outperform CNN models in the prior art.

Image-based profiling for drug discovery: due for a machine-learning upgrade?

Image-based profiling is a maturing strategy by which the rich information present in biological images is reduced to a multidimensional profile, a collection of extracted image-based features. These profiles can be mined for relevant patterns, revealing unexpected biological activity that is useful for many steps in the drug discovery process. Such applications include identifying disease-associated screenable phenotypes, understanding disease mechanisms and predicting a drug's activity, toxicity or mechanism of action. Several of these applications have been recently validated and have moved into production mode within academia and the pharmaceutical industry. Some of these have yielded disappointing results in practice but are now of renewed interest due to improved machine-learning strategies that better leverage image-based information. Although challenges remain, novel computational technologies such as deep learning and single-cell methods that better capture the biological information in images hold promise for accelerating drug discovery.

Guidelines for establishing a 3-D printing biofabrication laboratory

Advanced manufacturing and 3D printing are transformative technologies currently undergoing rapid adoption in healthcare, a traditionally non-manufacturing sector. Recent development in this field, largely enabled by merging different disciplines, has led to important clinical applications from anatomical models to regenerative bioscaffolding and devices. Although much research to-date has focussed on materials, designs, processes, and products, little attention has been given to the design and requirements of facilities for enabling clinically relevant biofabrication solutions. These facilities are critical to overcoming the major hurdles to clinical translation, including solving important issues such as reproducibility, quality control, regulations, and commercialization. To improve process uniformity and ensure consistent development and production, large-scale manufacturing of engineered tissues and organs will require standardized facilities, equipment, qualification processes, automation, and information systems. This review presents current and forward-thinking guidelines to help design biofabrication laboratories engaged in engineering model and tissue constructs for therapeutic and non-therapeutic applications.

Back and to the Future: From Neurotoxin-Induced to Human Parkinson's Disease Models

Parkinson's disease (PD) is an age-related neurodegenerative disorder characterized by motor symptoms such as tremor, slowness of movement, rigidity, and postural instability, as well as non-motor features like sleep disturbances, loss of ability to smell, depression, constipation, and pain. Motor symptoms are caused by depletion of dopamine in the striatum due to the progressive loss of dopamine neurons in the substantia nigra pars compacta. Approximately 10% of PD cases are familial arising from genetic mutations in α-synuclein, LRRK2, DJ-1, PINK1, parkin, and several other proteins. The majority of PD cases are, however, idiopathic, i.e., having no clear etiology. PD is characterized by progressive accumulation of insoluble inclusions, known as Lewy bodies, mostly composed of α-synuclein and membrane components. The cause of PD is currently attributed to cellular proteostasis deregulation and mitochondrial dysfunction, which are likely interdependent. In addition, neuroinflammation is present in brains of PD patients, but whether it is the cause or consequence of neurodegeneration remains to be studied. Rodents do not develop PD or PD-like motor symptoms spontaneously; however, neurotoxins, genetic mutations, viral vector-mediated transgene expression and, recently, injections of misfolded α-synuclein have been successfully utilized to model certain aspects of the disease. Here, we critically review the advantages and drawbacks of rodent PD models and discuss approaches to advance pre-clinical PD research towards successful disease-modifying therapy. © 2020 The Authors.

A bird's-eye view of deep learning in bioimage analysis

Deep learning of artificial neural networks has become the de facto standard approach to solving data analysis problems in virtually all fields of science and engineering. Also in biology and medicine, deep learning technologies are fundamentally transforming how we acquire, process, analyze, and interpret data, with potentially far-reaching consequences for healthcare. In this mini-review, we take a bird's-eye view at the past, present, and future developments of deep learning, starting from science at large, to biomedical imaging, and bioimage analysis in particular.

Deep Learning-Based Single-Cell Optical Image Studies

Optical imaging technology that has the advantages of high sensitivity and cost-effectiveness greatly promotes the progress of nondestructive single-cell studies. Complex cellular image analysis tasks such as three-dimensional reconstruction call for machine-learning technology in cell optical image research. With the rapid developments of high-throughput imaging flow cytometry, big data cell optical images are always obtained that may require machine learning for data analysis. In recent years, deep learning has been prevalent in the field of machine learning for large-scale image processing and analysis, which brings a new dawn for single-cell optical image studies with an explosive growth of data availability. Popular deep learning techniques offer new ideas for multimodal and multitask single-cell optical image research. This article provides an overview of the basic knowledge of deep learning and its applications in single-cell optical image studies. We explore the feasibility of applying deep learning techniques to single-cell optical image analysis, where popular techniques such as transfer learning, multimodal learning, multitask learning, and end-to-end learning have been reviewed. Image preprocessing and deep learning model training methods are then summarized. Applications based on deep learning techniques in the field of single-cell optical image studies are reviewed, which include image segmentation, super-resolution image reconstruction, cell tracking, cell counting, cross-modal image reconstruction, and design and control of cell imaging systems. In addition, deep learning in popular single-cell optical imaging techniques such as label-free cell optical imaging, high-content screening, and high-throughput optical imaging cytometry are also mentioned. Finally, the perspectives of deep learning technology for single-cell optical image analysis are discussed. © 2020 International Society for Advancement of Cytometry.

Review of high-content screening applications in toxicology

High-content screening (HCS) technology combining automated microscopy and quantitative image analysis can address biological questions in academia and the pharmaceutical industry. Various HCS experimental applications have been utilized in the research field of in vitro toxicology. In this review, we describe several HCS application approaches used for studying the mechanism of compound toxicity, highlight some challenges faced in the toxicological community, and discuss the future directions of HCS in regards to new models, new reagents, data management, and informatics. Many specialized areas of toxicology including developmental toxicity, genotoxicity, developmental neurotoxicity/neurotoxicity, hepatotoxicity, cardiotoxicity, and nephrotoxicity will be examined. In addition, several newly developed cellular assay models including induced pluripotent stem cells (iPSCs), three-dimensional (3D) cell models, and tissues-on-a-chip will be discussed. New genome-editing technologies (e.g., CRISPR/Cas9), data analyzing tools for imaging, and coupling with high-content assays will be reviewed. Finally, the applications of machine learning to image processing will be explored. These new HCS approaches offer a huge step forward in dissecting biological processes, developing drugs, and making toxicology studies easier.

Automated Neuron Detection in High-Content Fluorescence Microscopy Images Using Machine Learning

The study of neuronal morphology in relation to function, and the development of effective medicines to positively impact this relationship in patients suffering from neurodegenerative diseases, increasingly involves image-based high-content screening and analysis. The first critical step toward fully automated high-content image analyses in such studies is to detect all neuronal cells and distinguish them from possible non-neuronal cells or artifacts in the images. Here we investigate the performance of well-established machine learning techniques for this purpose. These include support vector machines, random forests, k-nearest neighbors, and generalized linear model classifiers, operating on an extensive set of image features extracted using the compound hierarchy of algorithms representing morphology, and the scale-invariant feature transform. We present experiments on a dataset of rat hippocampal neurons from our own studies to find the most suitable classifier(s) and subset(s) of features in the common practical setting where there is very limited annotated data for training. The results indicate that a random forests classifier using the right feature subset ranks best for the considered task, although its performance is not statistically significantly better than some support vector machine based classification models.

Transfer Learning with Deep Convolutional Neural Networks for Classifying Cellular Morphological Changes

The quantification and identification of cellular phenotypes from high-content microscopy images has proven to be very useful for understanding biological activity in response to different drug treatments. The traditional approach has been to use classical image analysis to quantify changes in cell morphology, which requires several nontrivial and independent analysis steps. Recently, convolutional neural networks have emerged as a compelling alternative, offering good predictive performance and the possibility to replace traditional workflows with a single network architecture. In this study, we applied the pretrained deep convolutional neural networks ResNet50, InceptionV3, and InceptionResnetV2 to predict cell mechanisms of action in response to chemical perturbations for two cell profiling datasets from the Broad Bioimage Benchmark Collection. These networks were pretrained on ImageNet, enabling much quicker model training. We obtain higher predictive accuracy than previously reported, between 95% and 97%. The ability to quickly and accurately distinguish between different cell morphologies from a scarce amount of labeled data illustrates the combined benefit of transfer learning and deep convolutional neural networks for interrogating cell-based images.

Arrayed functional genetic screenings in pluripotency reprogramming and differentiation

Thoroughly understanding the molecular mechanisms responsible for the biological properties of pluripotent stem cells, as well as for the processes involved in reprograming, differentiation, and transition between Naïve and Primed pluripotent states, is of great interest in basic and applied research. Although pluripotent cells have been extensively characterized in terms of their transcriptome and miRNome, a comprehensive understanding of how these gene products specifically impact their biology, depends on gain- or loss-of-function experimental approaches capable to systematically interrogate their function. We review all studies carried up to date that used arrayed screening approaches to explore the function of these genetic elements on those biological contexts, using focused or genome-wide genetic libraries. We further discuss the limitations and advantages of approaches based on assays with population-level primary readouts, derived from single-parameter plate readers, or cell-level primary readouts, obtained using multiparametric flow cytometry or quantitative fluorescence microscopy (i.e., high-content screening). Finally, we discuss technical limitation and future perspectives, highlighting how the integration of screening data may lead to major advances in the field of stem cell research and therapy.

From Light Microscopy to Analytical Scanning Electron Microscopy (SEM) and Focused Ion Beam (FIB)/SEM in Biology: Fixed Coordinates, Flat Embedding, Absolute References

Correlative light and electron microscopy (CLEM) has been in use for several years, however it has remained a costly method with difficult sample preparation. Here, we report a series of technical improvements developed for precise and cost-effective correlative light and scanning electron microscopy (SEM) and focused ion beam (FIB)/SEM microscopy of single cells, as well as large tissue sections. Customized coordinate systems for both slides and coverslips were established for thin and ultra-thin embedding of a wide range of biological specimens. Immobilization of biological samples was examined with a variety of adhesives. For histological sections, a filter system for flat embedding was developed. We validated ultra-thin embedding on laser marked slides for efficient, high-resolution CLEM. Target cells can be re-located within minutes in SEM without protracted searching and correlative investigations were reduced to a minimum of preparation steps, while still reaching highest resolution. The FIB/SEM milling procedure is facilitated and significantly accelerated as: (i) milling a ramp becomes needless, (ii) significant re-deposition of milled material does not occur; and (iii) charging effects are markedly reduced. By optimizing all technical parameters FIB/SEM stacks with 2 nm iso-voxels were achieved over thousands of sections, in a wide range of biological samples.

Opportunities and obstacles for deep learning in biology and medicine

Deep learning describes a class of machine learning algorithms that are capable of combining raw inputs into layers of intermediate features. These algorithms have recently shown impressive results across a variety of domains. Biology and medicine are data-rich disciplines, but the data are complex and often ill-understood. Hence, deep learning techniques may be particularly well suited to solve problems of these fields. We examine applications of deep learning to a variety of biomedical problems-patient classification, fundamental biological processes and treatment of patients-and discuss whether deep learning will be able to transform these tasks or if the biomedical sphere poses unique challenges. Following from an extensive literature review, we find that deep learning has yet to revolutionize biomedicine or definitively resolve any of the most pressing challenges in the field, but promising advances have been made on the prior state of the art. Even though improvements over previous baselines have been modest in general, the recent progress indicates that deep learning methods will provide valuable means for speeding up or aiding human investigation. Though progress has been made linking a specific neural network's prediction to input features, understanding how users should interpret these models to make testable hypotheses about the system under study remains an open challenge. Furthermore, the limited amount of labelled data for training presents problems in some domains, as do legal and privacy constraints on work with sensitive health records. Nonetheless, we foresee deep learning enabling changes at both bench and bedside with the potential to transform several areas of biology and medicine.

Deep learning in bioinformatics

In the era of big data, transformation of biomedical big data into valuable knowledge has been one of the most important challenges in bioinformatics. Deep learning has advanced rapidly since the early 2000s and now demonstrates state-of-the-art performance in various fields. Accordingly, application of deep learning in bioinformatics to gain insight from data has been emphasized in both academia and industry. Here, we review deep learning in bioinformatics, presenting examples of current research. To provide a useful and comprehensive perspective, we categorize research both by the bioinformatics domain (i.e. omics, biomedical imaging, biomedical signal processing) and deep learning architecture (i.e. deep neural networks, convolutional neural networks, recurrent neural networks, emergent architectures) and present brief descriptions of each study. Additionally, we discuss theoretical and practical issues of deep learning in bioinformatics and suggest future research directions. We believe that this review will provide valuable insights and serve as a starting point for researchers to apply deep learning approaches in their bioinformatics studies.

Data-analysis strategies for image-based cell profiling

Image-based cell profiling is a high-throughput strategy for the quantification of phenotypic differences among a variety of cell populations. It paves the way to studying biological systems on a large scale by using chemical and genetic perturbations. The general workflow for this technology involves image acquisition with high-throughput microscopy systems and subsequent image processing and analysis. Here, we introduce the steps required to create high-quality image-based (i.e., morphological) profiles from a collection of microscopy images. We recommend techniques that have proven useful in each stage of the data analysis process, on the basis of the experience of 20 laboratories worldwide that are refining their image-based cell-profiling methodologies in pursuit of biological discovery. The recommended techniques cover alternatives that may suit various biological goals, experimental designs, and laboratories' preferences.

Automated analysis of high-content microscopy data with deep learning

Existing computational pipelines for quantitative analysis of high‐content microscopy data rely on traditional machine learning approaches that fail to accurately classify more than a single dataset without substantial tuning and training, requiring extensive analysis. Here, we demonstrate that the application of deep learning to biological image data can overcome the pitfalls associated with conventional machine learning classifiers. Using a deep convolutional neural network (DeepLoc) to analyze yeast cell images, we show improved performance over traditional approaches in the automated classification of protein subcellular localization. We also demonstrate the ability of DeepLoc to classify highly divergent image sets, including images of pheromone‐arrested cells with abnormal cellular morphology, as well as images generated in different genetic backgrounds and in different laboratories. We offer an open‐source implementation that enables updating DeepLoc on new microscopy datasets. This study highlights deep learning as an important tool for the expedited analysis of high‐content microscopy data.

Machine learning applications in cell image analysis

Machine learning (ML) refers to a set of automatic pattern recognition methods that have been successfully applied across various problem domains, including biomedical image analysis. This review focuses on ML applications for image analysis in light microscopy experiments with typical tasks of segmenting and tracking individual cells, and modelling of reconstructed lineage trees. After describing a typical image analysis pipeline and highlighting challenges of automatic analysis (for example, variability in cell morphology, tracking in presence of clutters) this review gives a brief historical outlook of ML, followed by basic concepts and definitions required for understanding examples. This article then presents several example applications at various image processing stages, including the use of supervised learning methods for improving cell segmentation, and the application of active learning for tracking. The review concludes with remarks on parameter setting and future directions.

Machine learning and computer vision approaches for phenotypic profiling

Grys et al. review computer vision and machine-learning methods that have been applied to phenotypic profiling of image-based data. Descriptions are provided for segmentation, feature extraction, selection, and dimensionality reduction, as well as clustering, outlier detection, and classification of data.

With recent advances in high-throughput, automated microscopy, there has been an increased demand for effective computational strategies to analyze large-scale, image-based data. To this end, computer vision approaches have been applied to cell segmentation and feature extraction, whereas machine-learning approaches have been developed to aid in phenotypic classification and clustering of data acquired from biological images. Here, we provide an overview of the commonly used computer vision and machine-learning methods for generating and categorizing phenotypic profiles, highlighting the general biological utility of each approach.

Classifying and segmenting microscopy images with deep multiple instance learning

MOTIVATION

High-content screening (HCS) technologies have enabled large scale imaging experiments for studying cell biology and for drug screening. These systems produce hundreds of thousands of microscopy images per day and their utility depends on automated image analysis. Recently, deep learning approaches that learn feature representations directly from pixel intensity values have dominated object recognition challenges. These tasks typically have a single centered object per image and existing models are not directly applicable to microscopy datasets. Here we develop an approach that combines deep convolutional neural networks (CNNs) with multiple instance learning (MIL) in order to classify and segment microscopy images using only whole image level annotations.

RESULTS

We introduce a new neural network architecture that uses MIL to simultaneously classify and segment microscopy images with populations of cells. We base our approach on the similarity between the aggregation function used in MIL and pooling layers used in CNNs. To facilitate aggregating across large numbers of instances in CNN feature maps we present the Noisy-AND pooling function, a new MIL operator that is robust to outliers. Combining CNNs with MIL enables training CNNs using whole microscopy images with image level labels. We show that training end-to-end MIL CNNs outperforms several previous methods on both mammalian and yeast datasets without requiring any segmentation steps.

AVAILABILITY AND IMPLEMENTATION

Torch7 implementation available upon request.

CONTACT

oren.kraus@mail.utoronto.ca.

Applications in image-based profiling of perturbations

A dramatic shift has occurred in how biologists use microscopy images. Whether experiments are small-scale or high-throughput, automatically quantifying biological properties in images is now widespread. We see yet another revolution under way: a transition towards using automated image analysis to not only identify phenotypes a biologist specifically seeks to measure ('screening') but also as an unbiased and sensitive tool to capture a wide variety of subtle features of cell (or organism) state ('profiling'). Mapping similarities among samples using image-based (morphological) profiling has tremendous potential to transform drug discovery, functional genomics, and basic biological research. Applications include target identification, lead hopping, library enrichment, functionally annotating genes/alleles, and identifying small molecule modulators of gene activity and disease-specific phenotypes.