Scaling biological discovery at the interface of deep learning and cellular imaging

Machine learning in microscopy - insights, opportunities and challenges

Machine learning (ML) is transforming the field of image processing and analysis, from automation of laborious tasks to open-ended exploration of visual patterns. This has striking implications for image-driven life science research, particularly microscopy. In this Review, we focus on the opportunities and challenges associated with applying ML-based pipelines for microscopy datasets from a user point of view. We investigate the significance of different data characteristics - quantity, transferability and content - and how this determines which ML model(s) to use, as well as their output(s). Within the context of cell biological questions and applications, we further discuss ML utility range, namely data curation, exploration, prediction and explanation, and what they entail and translate to in the context of microscopy. Finally, we explore the challenges, common artefacts and risks associated with ML in microscopy. Building on insights from other fields, we propose how these pitfalls might be mitigated for in microscopy.

Pooled multicolour tagging for visualizing subcellular protein dynamics

Imaging-based methods are widely used for studying the subcellular localization of proteins in living cells. While routine for individual proteins, global monitoring of protein dynamics following perturbation typically relies on arrayed panels of fluorescently tagged cell lines, limiting throughput and scalability. Here, we describe a strategy that combines high-throughput microscopy, computer vision and machine learning to detect perturbation-induced changes in multicolour tagged visual proteomics cell (vpCell) pools. We use genome-wide and cancer-focused intron-targeting sgRNA libraries to generate vpCell pools and a large, arrayed collection of clones each expressing two different endogenously tagged fluorescent proteins. Individual clones can be identified in vpCell pools by image analysis using the localization patterns and expression level of the tagged proteins as visual barcodes, enabling simultaneous live-cell monitoring of large sets of proteins. To demonstrate broad applicability and scale, we test the effects of antiproliferative compounds on a pool with cancer-related proteins, on which we identify widespread protein localization changes and new inhibitors of the nuclear import/export machinery. The time-resolved characterization of changes in subcellular localization and abundance of proteins upon perturbation in a pooled format highlights the power of the vpCell approach for drug discovery and mechanism-of-action studies.

A Foundation Model for Cell Segmentation

Cells are a fundamental unit of biological organization, and identifying them in imaging data - cell segmentation - is a critical task for various cellular imaging experiments. While deep learning methods have led to substantial progress on this problem, most models in use are specialist models that work well for specific domains. Methods that have learned the general notion of "what is a cell" and can identify them across different domains of cellular imaging data have proven elusive. In this work, we present CellSAM, a foundation model for cell segmentation that generalizes across diverse cellular imaging data. CellSAM builds on top of the Segment Anything Model (SAM) by developing a prompt engineering approach for mask generation. We train an object detector, CellFinder, to automatically detect cells and prompt SAM to generate segmentations. We show that this approach allows a single model to achieve human-level performance for segmenting images of mammalian cells (in tissues and cell culture), yeast, and bacteria collected across various imaging modalities. We show that CellSAM has strong zero-shot performance and can be improved with a few examples via few-shot learning. We also show that CellSAM can unify bioimaging analysis workflows such as spatial transcriptomics and cell tracking. A deployed version of CellSAM is available at https://cellsam.deepcell.org/.