Search

The application of deep learning to spatial transcriptomics (ST) can reveal relationships between gene expression and tissue architecture. Prior work has demonstrated that inferring gene expression from tissue histomorphology can discern these spatial molecular markers to enable population scale studies, reducing the fiscal barriers associated with large-scale spatial profiling. However, while most improvements in algorithmic performance have focused on improving model architectures, little is known about how the quality of tissue preparation and imaging can affect deep learning model training for spatial inference from morphology and its potential for widespread clinical adoption. Prior studies for ST inference from histology typically utilize manually stained frozen sections with imaging on non-clinical grade scanners. Training such models on ST cohorts is also costly. We hypothesize that adopting tissue processing and imaging practices that mirror standards for clinical implementation (permanent sections, automated tissue staining, and clinical grade scanning) can significantly improve model performance. An enhanced specimen processing and imaging protocol was developed for deep learning-based ST inference from morphology. This protocol featured the Visium CytAssist assay to permit automated hematoxylin and eosin staining (e.g. Leica Bond), 40×-resolution imaging, and joining of multiple patients' tissue sections per capture area prior to ST profiling. Using a cohort of 13 pathologic T Stage-III stage colorectal cancer patients, we compared the performance of models trained on slide prepared using enhanced versus traditional (i.e. manual staining and low-resolution imaging) protocols. Leveraging Inceptionv3 neural networks, we predicted gene expression across serial, histologically-matched tissue sections using whole slide images (WSI) from both protocols. The data Shapley was used to quantify and compare marginal performance gains on a patient-by-patient basis attributed to using the enhanced protocol versus the actual costs of spatial profiling. Findings indicate that training and validating on WSI acquired through the enhanced protocol as opposed to the traditional method resulted in improved performance at lower fiscal cost. In the realm of ST, the enhancement of deep learning architectures frequently captures the spotlight; however, the significance of specimen processing and imaging is often understated. This research, informed through a game-theoretic lens, underscores the substantial impact that specimen preparation/imaging can have on spatial transcriptomic inference from morphology. It is essential to integrate such optimized processing protocols to facilitate the identification of prognostic markers at a larger scale., (© The Author(s) 2024. Published by Oxford University Press.)

Graph-based deep learning has shown great promise in cancer histopathology image analysis by contextualizing complex morphology and structure across whole slide images to make high quality downstream outcome predictions (ex: prognostication). These methods rely on informative representations (i.e., embeddings) of image patches comprising larger slides, which are used as node attributes in slide graphs. Spatial omics data, including spatial transcriptomics, is a novel paradigm offering a wealth of detailed information. Pairing this data with corresponding histological imaging localized at 50-micron resolution, may facilitate the development of algorithms which better appreciate the morphological and molecular underpinnings of carcinogenesis. Here, we explore the utility of leveraging spatial transcriptomics data with a contrastive crossmodal pretraining mechanism to generate deep learning models that can extract molecular and histological information for graph-based learning tasks. Performance on cancer staging, lymph node metastasis prediction, survival prediction, and tissue clustering analyses indicate that the proposed methods bring improvement to graph based deep learning models for histopathological slides compared to leveraging histological information from existing schemes, demonstrating the promise of mining spatial omics data to enhance deep learning for pathology workflows.

Intraoperative margin analysis is crucial for the successful removal of cutaneous squamous cell carcinomas (cSCC). Artificial intelligence technologies (AI) have previously demonstrated potential for facilitating rapid and complete tumour removal using intraoperative margin assessment for basal cell carcinoma. However, the varied morphologies of cSCC present challenges for AI margin assessment. The aim of this study was to develop and evaluate the accuracy of an AI algorithm for real-time histologic margin analysis of cSCC. To do this, a retrospective cohort study was conducted using frozen cSCC section slides. These slides were scanned and annotated, delineating benign tissue structures, inflammation and tumour to develop an AI algorithm for real-time margin analysis. A convolutional neural network workflow was used to extract histomorphological features predictive of cSCC. This algorithm demonstrated proof of concept for identifying cSCC with high accuracy, highlighting the potential for integration of AI into the surgical workflow. Incorporation of AI algorithms may improve efficiency and completeness of real-time margin assessment for cSCC removal, particularly in cases of moderately and poorly differentiated tumours/neoplasms. Further algorithmic improvement incorporating surrounding tissue context is necessary to remain sensitive to the unique epidermal landscape of well-differentiated tumours, and to map tumours to their original anatomical position/orientation., (© 2023 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd.)

Background: We developed a deep learning algorithm to evaluate defecatory patterns to identify dyssynergic defecation using 3-dimensional high definition anal manometry (3D-HDAM)., Aims: We developed a 3D-HDAM deep learning algorithm to evaluate for dyssynergia., Methods: Spatial-temporal data were extracted from consecutive 3D-HDAM studies performed between 2018 and 2020 at Dartmouth-Hitchcock Health. The technical procedure and gold standard definition of dyssynergia were based on the London consensus, adapted to the needs of 3D-HDAM technology. Three machine learning models were generated: (1) traditional machine learning informed by conventional anorectal function metrics, (2) deep learning, and (3) a hybrid approach. Diagnostic accuracy was evaluated using bootstrap sampling to calculate area-under-the-curve (AUC). To evaluate overfitting, models were validated by adding 502 simulated defecation maneuvers with diagnostic ambiguity., Results: 302 3D-HDAM studies representing 1208 simulated defecation maneuvers were included (average age 55.2 years; 80.5% women). The deep learning model had comparable diagnostic accuracy [AUC 0.91 (95% confidence interval 0.89-0.93)] to traditional [AUC 0.93(0.92-0.95)] and hybrid [AUC 0.96(0.94-0.97)] predictive models in training cohorts. However, the deep learning model handled ambiguous tests more cautiously than other models; the deep learning model was more likely to designate an ambiguous test as inconclusive [odds ratio 4.21(2.78-6.38)] versus traditional/hybrid approaches., Conclusions: Deep learning is capable of considering complex spatial-temporal information on 3D-HDAM technology. Future studies are needed to evaluate the clinical context of these preliminary findings., (© 2022. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.)

Background: Urine cytology is commonly used as a screening test for high-grade urothelial carcinoma for patients with risk factors or hematuria and is an essential step in longitudinal monitoring of patients with previous bladder cancer history. However, the semisubjective nature of current reporting systems for urine cytology (e.g., The Paris System) can hamper reproducibility. For instance, the incorporation of urothelial cell clusters into the classification schema is still an item of debate and perplexity among expert cytopathologists because several previous works have disputed their diagnostic relevance., Methods: In this work, an automated preprocessing tool for urothelial cell cluster assessment was developed that divides urothelial cell clusters into meaningful components for downstream assessment (ie, population-based studies, workflow automation)., Results: In this work, an automated preprocessing tool for urothelial cell cluster assessment was developed that divides urothelial cell clusters into meaningful components for downstream assessment (ie, population-based studies, workflow automation). Results indicate that cell cluster atypia (i.e., defined by whether the cell cluster harbored multiple atypical cells, thresholded by a minimum number of cells), cell border overlap and smoothness, and total number of clusters are important markers of specimen atypia when considering assessment of urothelial cell clusters., Conclusions: Markers established through techniques to separate cell clusters may have wider applicability for the design and implementation of machine learning approaches for urine cytology assessment., (© 2022 American Cancer Society.)

What is the relationship between mortality and satellite images as elucidated through the use of Convolutional Neural Networks? Background: Following a century of increase, life expectancy in the United States has stagnated and begun to decline in recent decades. Using satellite images and street view images, prior work has demonstrated associations of the built environment with income, education, access to care, and health factors such as obesity. However, assessment of learned image feature relationships with variation in crude mortality rate across the United States has been lacking. Objective: We sought to investigate if county-level mortality rates in the U.S. could be predicted from satellite images. Methods: Satellite images of neighborhoods surrounding schools were extracted with the Google Static Maps application programming interface for 430 counties representing ~68.9% of the US population. A convolutional neural network was trained using crude mortality rates for each county in 2015 to predict mortality. Learned image features were interpreted using Shapley Additive Feature Explanations, clustered, and compared to mortality and its associated covariate predictors. Results: Predicted mortality from satellite images in a held-out test set of counties was strongly correlated to the true crude mortality rate (Pearson r = 0.72). Direct prediction of mortality using a deep learning model across a cross-section of 430 U.S. counties identified key features in the environment (e.g., sidewalks, driveways, and hiking trails) associated with lower mortality. Learned image features were clustered, and we identified 10 clusters that were associated with education, income, geographical region, race, and age. Conclusions: The application of deep learning techniques to remotely-sensed features of the built environment can serve as a useful predictor of mortality in the United States. Although we identified features that were largely associated with demographic information, future modeling approaches that directly identify image features associated with health-related outcomes have the potential to inform targeted public health interventions., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2021 Levy, Lebeaux, Hoen, Christensen, Vaickus and MacKenzie.)

The diagnosis of disease often requires analysis of a biopsy. Many diagnoses depend not only on the presence of certain features but on their location within the tissue. Recently, a number of deep learning diagnostic aids have been developed to classify digitized biopsy slides. Clinical workflows often involve processing of more than 500 slides per day. But, clinical use of deep learning diagnostic aids would require a preprocessing workflow that is cost-effective, flexible, scalable, rapid, interpretable, and transparent. Here, we present such a workflow, optimized using Dask and mixed precision training via APEX, capable of handling any patch-level or slide level classification and prediction problem. The workflow uses a flexible and fast preprocessing and deep learning analytics pipeline, incorporates model interpretation and has a highly storage-efficient audit trail. We demonstrate the utility of this package on the analysis of a prototypical anatomic pathology specimen, liver biopsies for evaluation of hepatitis from a prospective cohort. The preliminary data indicate that PathFlowAI may become a cost-effective and time-efficient tool for clinical use of Artificial Intelligence (AI) algorithms.

Background: The Paris System for Urine Cytopathology (the Paris System) has succeeded in making the analysis of liquid-based urine preparations more reproducible. Any algorithm seeking to automate this system must accurately estimate the nuclear-to-cytoplasmic (N:C) ratio and produce a qualitative "atypia score." The authors propose a hybrid deep-learning and morphometric model that reliably automates the Paris System., Methods: Whole-slide images (WSI) of liquid-based urine cytology specimens were extracted from 51 negative, 60 atypical, 52 suspicious, and 54 positive cases. Morphometric algorithms were applied to decompose images to their component parts; and statistics, including the NC ratio, were tabulated using segmentation algorithms to create organized data structures, dubbed rich information matrices (RIMs). These RIM objects were enhanced using deep-learning algorithms to include qualitative measures. The augmented RIM objects were then used to reconstruct WSIs with filtering criteria and to generate pancellular statistical information., Results: The described system was used to calculate the N:C ratio for all cells, generate object classifications (atypical urothelial cell, squamous cell, crystal, etc), filter the original WSI to remove unwanted objects, rearrange the WSI to an efficient, condensed-grid format, and generate pancellular statistics containing quantitative/qualitative data for every cell in a WSI. In addition to developing novel techniques for managing WSIs, a system capable of automatically tabulating the Paris System criteria also was generated., Conclusions: A hybrid deep-learning and morphometric algorithm was developed for the analysis of urine cytology specimens that could reliably automate the Paris System and provide many avenues for increasing the efficiency of digital screening for urine WSIs and other cytology preparations., (© 2019 American Cancer Society.)

Successful treatment of solid cancers relies on complete surgical excision of the tumor either for definitive treatment or before adjuvant therapy. Intraoperative and postoperative radial sectioning, the most common form of margin assessment, can lead to incomplete excision and increase the risk of recurrence and repeat procedures. Mohs Micrographic Surgery is associated with complete removal of basal cell and squamous cell carcinoma through real-time margin assessment of 100% of the peripheral and deep margins. Real-time assessment in many tumor types is constrained by tissue size, complexity, and specimen processing / assessment time during general anesthesia. We developed an artificial intelligence platform to reduce the tissue preprocessing and histological assessment time through automated grossing recommendations, mapping and orientation of tumor to the surgical specimen. Using basal cell carcinoma as a model system, results demonstrate that this approach can address surgical laboratory efficiency bottlenecks for rapid and complete intraoperative margin assessment. [ABSTRACT FROM AUTHOR]

Background: Adopting a computational approach for the assessment of urine cytology specimens has the potential to improve the efficiency, accuracy, and reliability of bladder cancer screening, which has heretofore relied on semisubjective manual assessment methods. As rigorous, quantitative criteria and guidelines have been introduced for improving screening practices (e.g., The Paris System for Reporting Urinary Cytology), algorithms to emulate semiautonomous diagnostic decision‐making have lagged behind, in part because of the complex and nuanced nature of urine cytology reporting. Methods: In this study, the authors report on the development and large‐scale validation of a deep‐learning tool, AutoParis‐X, which can facilitate rapid, semiautonomous examination of urine cytology specimens. Results: The results of this large‐scale, retrospective validation study indicate that AutoParis‐X can accurately determine urothelial cell atypia and aggregate a wide variety of cell‐related and cluster‐related information across a slide to yield an atypia burden score, which correlates closely with overall specimen atypia and is predictive of Paris system diagnostic categories. Importantly, this approach accounts for challenges associated with the assessment of overlapping cell cluster borders, which improve the ability to predict specimen atypia and accurately estimate the nuclear‐to‐cytoplasm ratio for cells in these clusters. Conclusions: The authors developed a publicly available, open‐source, interactive web application that features a simple, easy‐to‐use display for examining urine cytology whole‐slide images and determining the level of atypia in specific cells, flagging the most abnormal cells for pathologist review. The accuracy of AutoParis‐X (and other semiautomated digital pathology systems) indicates that these technologies are approaching clinical readiness and necessitates full evaluation of these algorithms in head‐to‐head clinical trials. The authors developed a publicly available, open‐source, interactive web application featuring a simple, easy‐to‐use display for examining urine cytology whole‐slide images and determining the atypia level of specific cells, flagging the most abnormal cells for pathologist review. This proposed computational approach for the assessment of urine cytology specimens exhibited potential to improve the efficiency, accuracy, and reliability of bladder cancer screening. [ABSTRACT FROM AUTHOR]

Background: Urine cytology is generally considered the primary approach for screening for recurrence of bladder cancer. However, it is currently unclear how best to use cytological examinations for assessment and early detection of recurrence, beyond identifying a positive finding that requires more invasive methods to confirm recurrence and decide on therapeutic options. Because screening programs are frequent, and can be burdensome, finding quantitative means to reduce this burden for patients, cytopathologists, and urologists is an important endeavor and can improve both the efficiency and reliability of findings. Additionally, identifying ways to risk‐stratify patients is crucial for improving quality of life while reducing the risk of future recurrence or progression of the cancer. Methods: In this study, a computational machine learning tool, AutoParis‐X, was leveraged to extract imaging features from urine cytology examinations longitudinally to study the predictive potential of urine cytology for assessing recurrence risk. This study examined how the significance of imaging predictors changes over time before and after surgery to determine which predictors and time periods are most relevant for assessing recurrence risk. Results: Results indicate that imaging predictors extracted using AutoParis‐X can predict recurrence as well or better than traditional cytological/histological assessments alone and that the predictiveness of these features is variable across time, with key differences in overall specimen atypia identified immediately before tumor recurrence. Conclusions: Further research will clarify how computational methods can be effectively used in high‐volume screening programs to improve recurrence detection and complement traditional modes of assessment. This study used AutoParis‐X, a machine learning tool, to extract imaging features from urine cytology examinations to predict recurrence risk in patients with bladder cancer. The results demonstrate that quantitative features of urine specimen atypia can predict recurrence as well or better than traditional cytological/histological assessments alone and can potentially complement traditional methods of assessment in screening programs pending further development and validation of computational methods that leverage multiple longitudinal cytology examinations. [ABSTRACT FROM AUTHOR]

Background: Deep learning models can infer cancer patient prognosis from molecular and anatomic pathology information. Recent studies that leveraged information from complementary multimodal data improved prognostication, further illustrating the potential utility of such methods. However, current approaches: 1) do not comprehensively leverage biological and histomorphological relationships and 2) make use of emerging strategies to "pretrain" models (i.e., train models on a slightly orthogonal dataset/modeling objective) which may aid prognostication by reducing the amount of information required for achieving optimal performance. In addition, model interpretation is crucial for facilitating the clinical adoption of deep learning methods by fostering practitioner understanding and trust in the technology. Methods: Here, we develop an interpretable multimodal modeling framework that combines DNA methylation, gene expression, and histopathology (i.e., tissue slides) data, and we compare performance of crossmodal pretraining, contrastive learning, and transfer learning versus the standard procedure. Results: Our models outperform the existing state-of-the-art method (average 11.54% C-index increase), and baseline clinically driven models (average 11.7% C-index increase). Model interpretations elucidate consideration of biologically meaningful factors in making prognosis predictions. Discussion: Our results demonstrate that the selection of pretraining strategies is crucial for obtaining highly accurate prognostication models, even more so than devising an innovative model architecture, and further emphasize the all-important role of the tumor microenvironment on disease progression. [ABSTRACT FROM AUTHOR]

Over 150 000 Americans are diagnosed with colorectal cancer (CRC) every year, and annually over 50 000 individuals will die from CRC, necessitating improvements in screening, prognostication, disease management, and therapeutic options. Tumor metastasis is the primary factor related to the risk of recurrence and mortality. Yet, screening for nodal and distant metastasis is costly, and invasive and incomplete resection may hamper adequate assessment. Signatures of the tumor-immune microenvironment (TIME) at the primary site can provide valuable insights into the aggressiveness of the tumor and the effectiveness of various treatment options. Spatially resolved transcriptomics technologies offer an unprecedented characterization of TIME through high multiplexing, yet their scope is constrained by cost. Meanwhile, it has long been suspected that histological, cytological, and macroarchitectural tissue characteristics correlate well with molecular information (e.g., gene expression). Thus, a method for predicting transcriptomics data through inference of RNA patterns from whole slide images (WSI) is a key step in studying metastasis at scale. In this work, we collected tissue from 4 stage-III (pT3) matched colorectal cancer patients for spatial transcriptomics profiling. The Visium spatial transcriptomics (ST) assay was used to measure transcript abundance for 17 943 genes at up to 5000 55-micron (i.e., 1-10 cells) spots per patient sampled in a honeycomb pattern, coregistered with hematoxylin and eosin (H&E) stained WSI. The Visium ST assay can measure expression at these spots through tissue permeabilization of mRNAs, which are captured through spatially (i.e., x-y positional coordinates) barcoded, gene specific oligo probes. WSI subimages were extracted around each co-registered Visium spot and were used to predict the expression at these spots using machine learning models. We prototyped and compared several convolutional, transformer, and graph convolutional neural networks to predict spatial RNA patterns at the Visium spots under the hypothesis that the transformer- and graph-based approaches better capture relevant spatial tissue architecture. We further analyzed the model’s ability to recapitulate spatial autocorrelation statistics using SPARK and SpatialDE. Overall, the results indicate that the transformer- and graph-based approaches were unable to outperform the convolutional neural network architecture, though they exhibited optimal performance for relevant disease-associated genes. Initial findings suggest that different neural networks that operate on different scales are relevant for capturing distinct disease pathways (e.g., epithelial to mesenchymal transition). We add further evidence that deep learning models can accurately predict gene expression in whole slide images and comment on under studied factors which may increase its external applicability (e.g., tissue context). Our preliminary work will motivate further investigation of inference for molecular patterns from whole slide images as metastasis predictors and in other applications. [ABSTRACT FROM AUTHOR]

DNA methylation (DNAm) alterations have been heavily implicated in carcinogenesis and the pathophysiology of diseases through upstream regulation of gene expression. DNAm deep-learning approaches are able to capture features associated with aging, cell type, and disease progression, but lack incorporation of prior biological knowledge. Here, we present modular, user-friendly deep-learning methodology and software, MethylCapsNet and MethylSPWNet, that group CpGs into biologically relevant capsules—such as gene promoter context, CpG island relationship, or user-defined groupings—and relate them to diagnostic and prognostic outcomes. We demonstrate these models' utility on 3,897 individuals in the classification of central nervous system (CNS) tumors. MethylCapsNet and MethylSPWNet provide an opportunity to increase DNAm deep-learning analyses' interpretability by enabling a flexible organization of DNAm data into biologically relevant capsules. [ABSTRACT FROM AUTHOR]