Evaluation of methods to assign cell type labels to cell clusters from single-cell RNA-sequencing data

Predicting cell types with supervised contrastive learning on cells and their types

Single-cell RNA-sequencing (scRNA-seq) is a powerful technique that provides high-resolution expression profiling of individual cells. It significantly advances our understanding of cellular diversity and function. Despite its potential, the analysis of scRNA-seq data poses considerable challenges related to multicollinearity, data imbalance, and batch effect. One of the pivotal tasks in single-cell data analysis is cell type annotation, which classifies cells into discrete types based on their gene expression profiles. In this work, we propose a novel modeling formalism for cell type annotation with a supervised contrastive learning method, named SCLSC (Supervised Contrastive Learning for Single Cell). Different from the previous usage of contrastive learning in single cell data analysis, we employed the contrastive learning for instance-type pairs instead of instance-instance pairs. More specifically, in the cell type annotation task, the contrastive learning is applied to learn cell and cell type representation that render cells of the same type to be clustered in the new embedding space. Through this approach, the knowledge derived from annotated cells is transferred to the feature representation for scRNA-seq data. The whole training process becomes more efficient when conducting contrastive learning for cell and their types. Our experiment results demonstrate that the proposed SCLSC method consistently achieves superior accuracy in predicting cell types compared to five state-of-the-art methods. SCLSC also performs well in identifying cell types in different batch groups. The simplicity of our method allows for scalability, making it suitable for analyzing datasets with a large number of cells. In a real-world application of SCLSC to monitor the dynamics of immune cell subpopulations over time, SCLSC demonstrates a capability to discriminate cell subtypes of CD19+ B cells that were not present in the training dataset.

Challenges and opportunities to computationally deconvolve heterogeneous tissue with varying cell sizes using single-cell RNA-sequencing datasets

Deconvolution of cell mixtures in "bulk" transcriptomic samples from homogenate human tissue is important for understanding disease pathologies. However, several experimental and computational challenges impede transcriptomics-based deconvolution approaches using single-cell/nucleus RNA-seq reference atlases. Cells from the brain and blood have substantially different sizes, total mRNA, and transcriptional activities, and existing approaches may quantify total mRNA instead of cell type proportions. Further, standards are lacking for the use of cell reference atlases and integrative analyses of single-cell and spatial transcriptomics data. We discuss how to approach these key challenges with orthogonal "gold standard" datasets for evaluating deconvolution methods.

Predictive network analysis identifies JMJD6 and other potential key drivers in Alzheimer's disease

Despite decades of genetic studies on late-onset Alzheimer's disease, the underlying molecular mechanisms remain unclear. To better comprehend its complex etiology, we use an integrative approach to build robust predictive (causal) network models using two large human multi-omics datasets. We delineate bulk-tissue gene expression into single cell-type gene expression and integrate clinical and pathologic traits, single nucleotide variation, and deconvoluted gene expression for the construction of cell type-specific predictive network models. Here, we focus on neuron-specific network models and prioritize 19 predicted key drivers modulating Alzheimer's pathology, which we then validate by knockdown in human induced pluripotent stem cell-derived neurons. We find that neuronal knockdown of 10 of the 19 targets significantly modulates levels of amyloid-beta and/or phosphorylated tau peptides, most notably JMJD6. We also confirm our network structure by RNA sequencing in the neurons following knockdown of each of the 10 targets, which additionally predicts that they are upstream regulators of REST and VGF. Our work thus identifies robust neuronal key drivers of the Alzheimer's-associated network state which may represent therapeutic targets with relevance to both amyloid and tau pathology in Alzheimer's disease.

CAMML with the Integration of Marker Proteins (ChIMP)

MOTIVATION

Cell typing is a critical task in the analysis of single-cell data, particularly when studying complex diseased tissues. Unfortunately, the sparsity and noise of single-cell data make accurate cell typing of individual cells difficult. To address these challenges, we previously developed the CAMML method for multi-label cell typing of single-cell RNA-sequencing (scRNA-seq) data. CAMML uses weighted gene sets to score each profiled cell for multiple potential cell types. While CAMML outperforms other scRNA-seq cell typing techniques, it only leverages transcriptomic data so cannot take advantage of newer multi-omic single-cell assays that jointly profile gene expression and protein abundance (e.g. joint scRNA-seq/CITE-seq).

RESULTS

We developed the CAMML with the Integration of Marker Proteins (ChIMP) method to support multi-label cell typing of individual cells jointly profiled via scRNA-seq and CITE-seq. ChIMP combines cell type scores computed on scRNA-seq data via the CAMML approach with discretized CITE-seq measurements for cell type marker proteins. The multi-omic cell type scores generated by ChIMP allow researchers to more precisely and conservatively cell type joint scRNA-seq/CITE-seq data.

AVAILABILITY AND IMPLEMENTATION

An implementation of this work is available on CRAN at https://cran.r-project.org/web/packages/CAMML/.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

SFRP4+ stromal cell subpopulation with IGF1 signaling in human endometrial regeneration

Our understanding of full-thickness endometrial regeneration after injury is limited by an incomplete molecular characterization of the cell populations responsible for the organ functions. To help fill this knowledge gap, we characterized 10,551 cells of full-thickness normal human uterine from two menstrual phases (proliferative and secretory phase) using unbiased single cell RNA-sequencing. We dissected cell heterogeneity of main cell types (epithelial, stromal, endothelial, and immune cells) of the full thickness uterine tissues, cell population architectures of human uterus cells across the menstrual cycle. We identified an SFRP4⁺ stromal cell subpopulation that was highly enriched in the regenerative stage of the human endometria during the menstrual cycle, and the SFRP4⁺ stromal cells could significantly enhance the proliferation of human endometrial epithelial organoid in vitro, and promote the regeneration of endometrial epithelial glands and full-thickness endometrial injury through IGF1 signaling pathway in vivo. Our cell atlas of full-thickness uterine tissues revealed the cellular heterogeneities, cell population architectures, and their cell-cell communications during the monthly regeneration of the human endometria, which provide insight into the biology of human endometrial regeneration and the development of regenerative medicine treatments against endometrial damage and intrauterine adhesion.

Executable models of immune signaling pathways in HIV-associated atherosclerosis

Atherosclerosis (AS)-associated cardiovascular disease is an important cause of mortality in an aging population of people living with HIV (PLWH). This elevated risk has been attributed to viral infection, anti-retroviral therapy, chronic inflammation, and lifestyle factors. However, the rates at which PLWH develop AS vary even after controlling for length of infection, treatment duration, and for lifestyle factors. To investigate the molecular signaling underlying this variation, we sequenced 9368 peripheral blood mononuclear cells (PBMCs) from eight PLWH, four of whom have atherosclerosis (AS+). Additionally, a publicly available dataset of PBMCs from persons before and after HIV infection was used to investigate the effect of acute HIV infection. To characterize dysregulation of pathways rather than just measuring enrichment, we developed the single-cell Boolean Omics Network Invariant Time Analysis (scBONITA) algorithm. scBONITA infers executable dynamic pathway models and performs a perturbation analysis to identify high impact genes. These dynamic models are used for pathway analysis and to map sequenced cells to characteristic signaling states (attractor analysis). scBONITA revealed that lipid signaling regulates cell migration into the vascular endothelium in AS+ PLWH. Pathways implicated included AGE-RAGE and PI3K-AKT signaling in CD8+ T cells, and glucagon and cAMP signaling pathways in monocytes. Attractor analysis with scBONITA facilitated the pathway-based characterization of cellular states in CD8+ T cells and monocytes. In this manner, we identify critical cell-type specific molecular mechanisms underlying HIV-associated atherosclerosis using a novel computational method.

Single-Cell Sequencing Technologies in Precision Oncology

Single-cell sequencing technologies are revolutionizing cancer research and are poised to become the standard for translational cancer studies. Rapidly decreasing costs and increasing throughput and resolution are paving the way for the adoption of single-cell technologies in clinical settings for personalized medicine applications. In this chapter, we review the state of the art of single-cell DNA and RNA sequencing technologies, the computational tools to analyze the data, and their potential application to precision oncology. We also discuss the advantages of single-cell over bulk sequencing for the dissection of intra-tumor heterogeneity and the characterization of subclonal cell populations, the implementation of targeted drug repurposing approaches, and describe advanced methodologies for multi-omics data integration and to assess cell signaling at single-cell resolution.

The induction of peripheral trained immunity in the pancreas incites anti-tumor activity to control pancreatic cancer progression

Despite the remarkable success of immunotherapy in many types of cancer, pancreatic ductal adenocarcinoma has yet to benefit. Innate immune cells are critical to anti-tumor immunosurveillance and recent studies have revealed that these populations possess a form of memory, termed trained innate immunity, which occurs through transcriptomic, epigenetic, and metabolic reprograming. Here we demonstrate that yeast-derived particulate β-glucan, an inducer of trained immunity, traffics to the pancreas, which causes a CCR2-dependent influx of monocytes/macrophages to the pancreas that display features of trained immunity. These cells can be activated upon exposure to tumor cells and tumor-derived factors, and show enhanced cytotoxicity against pancreatic tumor cells. In orthotopic models of pancreatic ductal adenocarcinoma, β-glucan treated mice show significantly reduced tumor burden and prolonged survival, which is further enhanced when combined with immunotherapy. These findings characterize the dynamic mechanisms and localization of peripheral trained immunity and identify an application of trained immunity to cancer.

Antitumor immune effects of preoperative sitravatinib and nivolumab in oral cavity cancer: SNOW window-of-opportunity study

BACKGROUND

Sitravatinib, a tyrosine kinase inhibitor that targets TYRO3, AXL, MERTK and the VEGF receptor family, is predicted to increase the M1 to M2-polarized tumor-associated macrophages ratio in the tumor microenvironment and have synergistic antitumor activity in combination with anti-programmed death-1/ligand-1 agents. SNOW is a window-of-opportunity study designed to evaluate the immune and molecular effects of preoperative sitravatinib and nivolumab in patients with oral cavity squamous cell carcinoma.

METHODS

Patients with newly-diagnosed untreated T2-4a, N0-2 or T1 >1 cm-N2 oral cavity carcinomas were eligible. All patients received sitravatinib 120 mg daily from day 1 up to 48 hours pre-surgery and one dose of nivolumab 240 mg on day 15. Surgery was planned between day 23 and 30. Standard of care adjuvant radiotherapy was given based on clinical stage. Tumor photographs, fresh tumor biopsies and blood samples were collected at baseline, at day 15 after sitravatinib alone, and at surgery after sitravatinib-nivolumab combination. Tumor flow cytometry, multiplex immunofluorescence staining and single-cell RNA sequencing (scRNAseq) were performed on tumor biopsies to study changes in immune-cell populations. Tumor whole-exome sequencing and circulating tumor DNA and cell-free DNA were evaluated at each time point.

RESULTS

Ten patients were included. Grade 3 toxicity occurred in one patient (hypertension); one patient required sitravatinib dose reduction, and one patient required discontinuation and surgery delay due to G2 thrombocytopenia. Nine patients had clinical-to-pathological downstaging, with one complete response. Independent pathological treatment response (PTR) assessment confirmed a complete PTR and two major PTRs. With a median follow-up of 21 months, all patients are alive with no recurrence. Circulating tumor DNA and cell-free DNA dynamics correlated with clinical and pathological response and distinguished two patient groups with different tumor biological behavior after sitravatinib alone (1A) versus sitravatinib-nivolumab (1B). Tumor immunophenotyping and scRNAseq analyses revealed differential changes in the expression of immune cell populations and sitravatinib-targeted and hypoxia-related genes in group 1A vs 1B patients.

CONCLUSIONS

The SNOW study shows sitravatinib plus nivolumab is safe and leads to deep clinical and pathological responses in oral cavity carcinomas. Multi-omic biomarker analyses dissect the differential molecular effects of sitravatinib versus the sitravatinib-nivolumab and revealed patients with distinct tumor biology behavior.

TRIAL REGISTRATION NUMBER

NCT03575598.

An in vitro stem cell model of human epiblast and yolk sac interaction

Human embryogenesis entails complex signalling interactions between embryonic and extra-embryonic cells. However, how extra-embryonic cells direct morphogenesis within the human embryo remains largely unknown due to a lack of relevant stem cell models. Here, we have established conditions to differentiate human pluripotent stem cells (hPSCs) into yolk sac-like cells (YSLCs) that resemble the post-implantation human hypoblast molecularly and functionally. YSLCs induce the expression of pluripotency and anterior ectoderm markers in human embryonic stem cells (hESCs) at the expense of mesoderm and endoderm markers. This activity is mediated by the release of BMP and WNT signalling pathway inhibitors, and, therefore, resembles the functioning of the anterior visceral endoderm signalling centre of the mouse embryo, which establishes the anterior-posterior axis. Our results implicate the yolk sac in epiblast cell fate specification in the human embryo and propose YSLCs as a tool for studying post-implantation human embryo development in vitro.

Gene expression-based immune infiltration analyses of renal cancer and their associations with survival outcome

BACKGROUND

Renal cancer is a common malignant tumor with an increasing incidence rate.

METHODS

In this study, based on the gene expression profiles, we analyzed the compositions of tumor-infiltrating immune cells (TIICs) in renal cancer and paracancerous samples using CIBERSORT. The proportions of 22 TIICs subsets in 122 paired renal carcinoma and paracancerous samples, and 224 Wilms tumor (WT) samples varied between intragroup and intergroup.

RESULTS

After analyzed the difference of TIICs composition between renal cancer and paired paracancerous samples, we found that M0 macrophages and CD8 T cells were significantly elevated, while naive B cells were significantly decreased in renal cancer samples compared with paracancerous samples. Survival analysis showed that high overall TIICs proportion, the low proportion of resting mast cells and the high proportion of activated memory CD4 T cells were associated with poor prognosis of renal cancer patients. In addition, 3 clusters were identified by hierarchical clustering analysis, and they presented a distinct prognosis. Cluster 1 had superior survival outcomes, while cluster 2 had an inferior survival outcome.

CONCLUSIONS

Our study indicated that overall TIICs proportion, certain TIICs subset proportion, including resting mast cells and activated memory CD4 T cells, and distinct cluster patterns were associated with the prognosis of renal cancer, which was significant for the clinical surveillance and treatment of renal cancer.

Tutorial: guidelines for annotating single-cell transcriptomic maps using automated and manual methods

Single-cell transcriptomics can profile thousands of cells in a single experiment and identify novel cell types, states and dynamics in a wide variety of tissues and organisms. Standard experimental protocols and analysis workflows have been developed to create single-cell transcriptomic maps from tissues. This tutorial focuses on how to interpret these data to identify cell types, states and other biologically relevant patterns with the objective of creating an annotated map of cells. We recommend a three-step workflow including automatic cell annotation (wherever possible), manual cell annotation and verification. Frequently encountered challenges are discussed, as well as strategies to address them. Guiding principles and specific recommendations for software tools and resources that can be used for each step are covered, and an R notebook is included to help run the recommended workflow. Basic familiarity with computer software is assumed, and basic knowledge of programming (e.g., in the R language) is recommended.

Methods and tools for spatial mapping of single-cell RNAseq clusters in Drosophila

Single-cell RNA sequencing (scRNAseq) experiments provide a powerful means to identify clusters of cells that share common gene expression signatures. A major challenge in scRNAseq studies is to map the clusters to specific anatomical regions along the body and within tissues. Existing data, such as information obtained from large-scale in situ RNA hybridization studies, cell type specific transcriptomics, gene expression reporters, antibody stainings, and fluorescent tagged proteins, can help to map clusters to anatomy. However, in many cases, additional validation is needed to precisely map the spatial location of cells in clusters. Several approaches are available for spatial resolution in Drosophila, including mining of existing datasets, and use of existing or new tools for direct or indirect detection of RNA, or direct detection of proteins. Here, we review available resources and emerging technologies that will facilitate spatial mapping of scRNAseq clusters at high resolution in Drosophila. Importantly, we discuss the need, available approaches, and reagents for multiplexing gene expression detection in situ, as in most cases scRNAseq clusters are defined by the unique coexpression of sets of genes.

Single-cell mapper (scMappR): using scRNA-seq to infer the cell-type specificities of differentially expressed genes

RNA sequencing (RNA-seq) is widely used to identify differentially expressed genes (DEGs) and reveal biological mechanisms underlying complex biological processes. RNA-seq is often performed on heterogeneous samples and the resulting DEGs do not necessarily indicate the cell-types where the differential expression occurred. While single-cell RNA-seq (scRNA-seq) methods solve this problem, technical and cost constraints currently limit its widespread use. Here we present single cell Mapper (scMappR), a method that assigns cell-type specificity scores to DEGs obtained from bulk RNA-seq by leveraging cell-type expression data generated by scRNA-seq and existing deconvolution methods. After evaluating scMappR with simulated RNA-seq data and benchmarking scMappR using RNA-seq data obtained from sorted blood cells, we asked if scMappR could reveal known cell-type specific changes that occur during kidney regeneration. scMappR appropriately assigned DEGs to cell-types involved in kidney regeneration, including a relatively small population of immune cells. While scMappR can work with user-supplied scRNA-seq data, we curated scRNA-seq expression matrices for ∼100 human and mouse tissues to facilitate its stand-alone use with bulk RNA-seq data from these species. Overall, scMappR is a user-friendly R package that complements traditional differential gene expression analysis of bulk RNA-seq data.

Single-cell transcriptomics analysis of mild cognitive impairment in World Trade Center disaster responders

INTRODUCTION

Recent research has found that World Trade Center (WTC) responders in their mid-50s have an elevated prevalence of mild cognitive impairment (MCI) that is associated with neural degeneration and subcortical thinning. This article extends our understanding of the molecular complexity of MCI through gene expression profiling of blood.

METHODS

The transcriptomics of 40 male WTC responders were profiled across two cohorts (discovery: nine MCI and nine controls; replication: 11 MCI and 11 controls) using CITE-Seq at single-cell resolution in blood.

RESULTS

Comparing the transcriptomic signatures across seven major cell subpopulations, the largest differences were observed in monocytes in which 226 genes were differentially expressed. Pathway analysis on the genes unique to monocytes identified processes associated with cerebral immune response.

DISCUSSION

Our findings suggested monocytes may constitute a key cell type to target in blood-based biomarker studies for early detection of risk of MCI and development of new interventions.

Automated methods for cell type annotation on scRNA-seq data

The advent of single-cell sequencing started a new era of transcriptomic and genomic research, advancing our knowledge of the cellular heterogeneity and dynamics. Cell type annotation is a crucial step in analyzing single-cell RNA sequencing data, yet manual annotation is time-consuming and partially subjective. As an alternative, tools have been developed for automatic cell type identification. Different strategies have emerged to ultimately associate gene expression profiles of single cells with a cell type either by using curated marker gene databases, correlating reference expression data, or transferring labels by supervised classification. In this review, we present an overview of the available tools and the underlying approaches to perform automated cell type annotations on scRNA-seq data.

muscat detects subpopulation-specific state transitions from multi-sample multi-condition single-cell transcriptomics data

Single-cell RNA sequencing (scRNA-seq) has become an empowering technology to profile the transcriptomes of individual cells on a large scale. Early analyses of differential expression have aimed at identifying differences between subpopulations to identify subpopulation markers. More generally, such methods compare expression levels across sets of cells, thus leading to cross-condition analyses. Given the emergence of replicated multi-condition scRNA-seq datasets, an area of increasing focus is making sample-level inferences, termed here as differential state analysis; however, it is not clear which statistical framework best handles this situation. Here, we surveyed methods to perform cross-condition differential state analyses, including cell-level mixed models and methods based on aggregated pseudobulk data. To evaluate method performance, we developed a flexible simulation that mimics multi-sample scRNA-seq data. We analyzed scRNA-seq data from mouse cortex cells to uncover subpopulation-specific responses to lipopolysaccharide treatment, and provide robust tools for multi-condition analysis within the muscat R package.

CReSCENT: CanceR Single Cell ExpressioN Toolkit

CReSCENT: CanceR Single Cell ExpressioN Toolkit (https://crescent.cloud), is an intuitive and scalable web portal incorporating a containerized pipeline execution engine for standardized analysis of single-cell RNA sequencing (scRNA-seq) data. While scRNA-seq data for tumour specimens are readily generated, subsequent analysis requires high-performance computing infrastructure and user expertise to build analysis pipelines and tailor interpretation for cancer biology. CReSCENT uses public data sets and preconfigured pipelines that are accessible to computational biology non-experts and are user-editable to allow optimization, comparison, and reanalysis for specific experiments. Users can also upload their own scRNA-seq data for analysis and results can be kept private or shared with other users.

Tempora: Cell trajectory inference using time-series single-cell RNA sequencing data

Single-cell RNA sequencing (scRNA-seq) can map cell types, states and transitions during dynamic biological processes such as tissue development and regeneration. Many trajectory inference methods have been developed to order cells by their progression through a dynamic process. However, when time series data is available, most of these methods do not consider the available time information when ordering cells and are instead designed to work only on a single scRNA-seq data snapshot. We present Tempora, a novel cell trajectory inference method that orders cells using time information from time-series scRNA-seq data. In performance comparison tests, Tempora inferred known developmental lineages from three diverse tissue development time series data sets, beating state of the art methods in accuracy and speed. Tempora works at the level of cell clusters (types) and uses biological pathway information to help identify cell type relationships. This approach increases gene expression signal from single cells, processing speed, and interpretability of the inferred trajectory. Our results demonstrate the utility of a combination of time and pathway information to supervise trajectory inference for scRNA-seq based analysis.

Single-cell RNA sequencing (scRNA-seq) enables an unparalleled ability to map the heterogeneity of dynamic multicellular processes, such as tissue development, tumor growth, wound response and repair, and inflammation. Multiple methods have been developed to order cells along a pseudotime axis that represents a trajectory through such processes using the concept that cells that are closely related in a lineage will have similar transcriptomes. However, time series experiments provide another useful information source to order cells, from earlier to later time point. By introducing a novel use of biological pathway prior information, our Tempora algorithm improves the accuracy and speed of cell trajectory inference from time-series scRNA-seq data as measured by reconstructing known developmental trajectories from three diverse data sets. By analyzing scRNA-seq data at the cluster (cell type) level instead of at the single-cell level and by using known pathway information, Tempora amplifies gene expression signals from one cell using similar cells in a cluster and similar genes within a pathway. This approach also reduces computational time and resources needed to analyze large data sets because it works with a relatively small number of clusters instead of a potentially large number of cells. Finally, it eases interpretation, via operating on a relatively small number of clusters which usually represent known cell types, as well as by identifying time-dependent pathways. Tempora is useful for finding novel insights in dynamic processes.

scClassify: sample size estimation and multiscale classification of cells using single and multiple reference

Automated cell type identification is a key computational challenge in single-cell RNA-sequencing (scRNA-seq) data. To capitalise on the large collection of well-annotated scRNA-seq datasets, we developed scClassify, a multiscale classification framework based on ensemble learning and cell type hierarchies constructed from single or multiple annotated datasets as references. scClassify enables the estimation of sample size required for accurate classification of cell types in a cell type hierarchy and allows joint classification of cells when multiple references are available. We show that scClassify consistently performs better than other supervised cell type classification methods across 114 pairs of reference and testing data, representing a diverse combination of sizes, technologies and levels of complexity, and further demonstrate the unique components of scClassify through simulations and compendia of experimental datasets. Finally, we demonstrate the scalability of scClassify on large single-cell atlases and highlight a novel application of identifying subpopulations of cells from the Tabula Muris data that were unidentified in the original publication. Together, scClassify represents state-of-the-art methodology in automated cell type identification from scRNA-seq data.

Evaluation of methods to assign cell type labels to cell clusters from single-cell RNA-sequencing data

Background: Identification of cell type subpopulations from complex cell mixtures using single-cell RNA-sequencing (scRNA-seq) data includes automated steps from normalization to cell clustering. However, assigning cell type labels to cell clusters is often conducted manually, resulting in limited documentation, low reproducibility and uncontrolled vocabularies. This is partially due to the scarcity of reference cell type signatures and because some methods support limited cell type signatures. Methods: In this study, we benchmarked five methods representing first-generation enrichment analysis (ORA), second-generation approaches (GSEA and GSVA), machine learning tools (CIBERSORT) and network-based neighbor voting (METANEIGHBOR), for the task of assigning cell type labels to cell clusters from scRNA-seq data. We used five scRNA-seq datasets: human liver, 11 Tabula Muris mouse tissues, two human peripheral blood mononuclear cell datasets, and mouse retinal neurons, for which reference cell type signatures were available. The datasets span Drop-seq, 10X Chromium and Seq-Well technologies and range in size from ~3,700 to ~68,000 cells. Results: Our results show that, in general, all five methods perform well in the task as evaluated by receiver operating characteristic curve analysis (average area under the curve (AUC) = 0.91, sd = 0.06), whereas precision-recall analyses show a wide variation depending on the method and dataset (average AUC = 0.53, sd = 0.24). We observed an influence of the number of genes in cell type signatures on performance, with smaller signatures leading more frequently to incorrect results. Conclusions: GSVA was the overall top performer and was more robust in cell type signature subsampling simulations, although different methods performed well using different datasets. METANEIGHBOR and GSVA were the fastest methods. CIBERSORT and METANEIGHBOR were more influenced than the other methods by analyses including only expected cell types. We provide an extensible framework that can be used to evaluate other methods and datasets at https://github.com/jdime/scRNAseq_cell_cluster_labeling.

Evaluation of methods to assign cell type labels to cell clusters from single-cell RNA-sequencing data

Background: Identification of cell type subpopulations from complex cell mixtures using single-cell RNA-sequencing (scRNA-seq) data includes automated computational steps like data normalization, dimensionality reduction and cell clustering. However, assigning cell type labels to cell clusters is still conducted manually by most researchers, resulting in limited documentation, low reproducibility and uncontrolled vocabularies. Two bottlenecks to automating this task are the scarcity of reference cell type gene expression signatures and the fact that some dedicated methods are available only as web servers with limited cell type gene expression signatures. Methods: In this study, we benchmarked four methods (CIBERSORT, GSEA, GSVA, and ORA) for the task of assigning cell type labels to cell clusters from scRNA-seq data. We used scRNA-seq datasets from liver, peripheral blood mononuclear cells and retinal neurons for which reference cell type gene expression signatures were available. Results: Our results show that, in general, all four methods show a high performance in the task as evaluated by receiver operating characteristic curve analysis (average area under the curve (AUC) = 0.94, sd = 0.036), whereas precision-recall curve analyses show a wide variation depending on the method and dataset (average AUC = 0.53, sd = 0.24). Conclusions: CIBERSORT and GSVA were the top two performers. Additionally, GSVA was the fastest of the four methods and was more robust in cell type gene expression signature subsampling simulations. We provide an extensible framework to evaluate other methods and datasets at https://github.com/jdime/scRNAseq_cell_cluster_labeling.