A point cloud segmentation framework for image-based spatial transcriptomics

Abstract Recent progress in image-based spatial RNA profiling enables to spatially resolve tens to hundreds of distinct RNA species with high spatial resolution. It presents new avenues for comprehending tissue organization. In this context, the ability to assign detected RNA transcripts to individual cells is crucial for downstream analyses, such as in-situ cell type calling. Yet, accurate cell segmentation can be challenging in tissue data, in particular in the absence of a high-quality membrane marker. To address this issue, we introduce ComSeg, a segmentation algorithm that operates directly on single RNA positions and that does not come with implicit or explicit priors on cell shape. ComSeg is applicable in complex tissues with arbitrary cell shapes. Through comprehensive evaluations on simulated and experimental datasets, we show that ComSeg outperforms existing state-of-the-art methods for in-situ single-cell RNA profiling and in-situ cell type calling. ComSeg is available as a documented and open source pip package at https://github.com/fish-quant/ComSeg .