Search

Abstract Background Metastatic prostate cancer is a highly heterogeneous and dynamic disease and practicable tools for patient stratification and resistance monitoring are urgently needed. Liquid biopsy analysis of circulating tumor cells (CTCs) and circulating tumor DNA are promising, however, comprehensive testing is essential due to diverse mechanisms of resistance. Previously, we demonstrated the utility of mRNA-based in situ padlock probe hybridization for characterizing CTCs. Methods We have developed a novel combinatorial dual-color (CoDuCo) assay for in situ mRNA detection, with enhanced multiplexing capacity, enabling the simultaneous analysis of up to 15 distinct markers. This approach was applied to CTCs, corresponding tumor tissue, cancer cell lines, and peripheral blood mononuclear cells for single-cell and spatial gene expression analysis. Using supervised machine learning, we trained a random forest classifier to identify CTCs. Image analysis and visualization of results was performed using open-source Python libraries, CellProfiler, and TissUUmaps. Results Our study presents data from multiple prostate cancer patients, demonstrating the CoDuCo assay’s ability to visualize diverse resistance mechanisms, such as neuroendocrine differentiation markers (SYP, CHGA, NCAM1) and AR-V7 expression. In addition, druggable targets and predictive markers (PSMA, DLL3, SLFN11) were detected in CTCs and formalin-fixed, paraffin-embedded tissue. The machine learning-based CTC classification achieved high performance, with a recall of 0.76 and a specificity of 0.99. Conclusions The combination of high multiplex capacity and microscopy-based single-cell analysis is a unique and powerful feature of the CoDuCo in situ assay. This synergy enables the simultaneous identification and characterization of CTCs with epithelial, epithelial-mesenchymal, and neuroendocrine phenotypes, the detection of CTC clusters, the visualization of CTC heterogeneity, as well as the spatial investigation of tumor tissue. This assay holds significant potential as a tool for monitoring dynamic molecular changes associated with drug response and resistance in prostate cancer.