New algorithms for unsupervised cell clustering from scRNA-seq data

The identification of cell types is a basic step of pipelines for Single-Cell RNA sequencing (scRNA-seq) data analysis. However, unsupervised clustering of cells from scRNA-seq data has multiple challenges: high dimensionality, sparseness of the expression matrix, and technical noise that generates false zero entries. In this study, we introduce new algorithms for clustering scRNA-seq data. The first algorithm builds a <i>k</i>-MST graph from distances obtained directly from the input data without dimensionality reduction. The computation follows an iterative procedure of k steps, calculating the edges of minimum spanning trees over different subgraphs obtained by removing edges selected in previous iterations. The Louvain algorithm is executed on the <i>k</i>-MST graph for cell clustering. We also explored an alternative based on neural networks, using an autoencoder to learn the parameters of a Gaussian mixture model. Benchmark experiments show that the algorithms have competitive accuracy, compared to previous solutions. Sequencing depth, number of cells and tissue types have important effects on the performance of the algorithms. Further experiments with scRNA-data taken from a patient with refractory epilepsy show that the autoencoder model achieved the best accuracy for this dataset, and the <i>k</i>-MST was competitive among graph-based approaches.