1. PLUTO: Pathology-Universal Transformer
- Author
-
Juyal, Dinkar, Padigela, Harshith, Shah, Chintan, Shenker, Daniel, Harguindeguy, Natalia, Liu, Yi, Martin, Blake, Zhang, Yibo, Nercessian, Michael, Markey, Miles, Finberg, Isaac, Luu, Kelsey, Borders, Daniel, Javed, Syed Ashar, Krause, Emma, Biju, Raymond, Sood, Aashish, Ma, Allen, Nyman, Jackson, Shamshoian, John, Chhor, Guillaume, Sanghavi, Darpan, Thibault, Marc, Yu, Limin, Najdawi, Fedaa, Hipp, Jennifer A., Fahy, Darren, Glass, Benjamin, Walk, Eric, Abel, John, Pokkalla, Harsha, Beck, Andrew H., and Grullon, Sean
- Subjects
Electrical Engineering and Systems Science - Image and Video Processing ,Computer Science - Computer Vision and Pattern Recognition - Abstract
Pathology is the study of microscopic inspection of tissue, and a pathology diagnosis is often the medical gold standard to diagnose disease. Pathology images provide a unique challenge for computer-vision-based analysis: a single pathology Whole Slide Image (WSI) is gigapixel-sized and often contains hundreds of thousands to millions of objects of interest across multiple resolutions. In this work, we propose PathoLogy Universal TransfOrmer (PLUTO): a light-weight pathology FM that is pre-trained on a diverse dataset of 195 million image tiles collected from multiple sites and extracts meaningful representations across multiple WSI scales that enable a large variety of downstream pathology tasks. In particular, we design task-specific adaptation heads that utilize PLUTO's output embeddings for tasks which span pathology scales ranging from subcellular to slide-scale, including instance segmentation, tile classification, and slide-level prediction. We compare PLUTO's performance to other state-of-the-art methods on a diverse set of external and internal benchmarks covering multiple biologically relevant tasks, tissue types, resolutions, stains, and scanners. We find that PLUTO matches or outperforms existing task-specific baselines and pathology-specific foundation models, some of which use orders-of-magnitude larger datasets and model sizes when compared to PLUTO. Our findings present a path towards a universal embedding to power pathology image analysis, and motivate further exploration around pathology foundation models in terms of data diversity, architectural improvements, sample efficiency, and practical deployability in real-world applications.
- Published
- 2024