Search

Clinicians and pathologists traditionally use patient data in addition to clinical examination to support their diagnoses. We investigated whether a combination of histologic whole slides image (WSI) analysis based on convolutional neural networks (CNNs) and commonly available patient data (age, sex and anatomical site of the lesion) in a binary melanoma/nevus classification task could increase the performance compared with CNNs alone. We used 431 WSIs from two different laboratories and analysed the performance of classifiers that used the image or patient data individually or three common fusion techniques. Furthermore, we tested a naive combination of patient data and an image classifier: for cases interpreted as 'uncertain' (CNN output score <0.7), the decision of the CNN was replaced by the decision of the patient data classifier. The CNN on its own achieved the best performance (mean ± standard deviation of five individual runs) with AUROC of 92.30% ± 0.23% and balanced accuracy of 83.17% ± 0.38%. While the classification performance was not significantly improved in general by any of the tested fusions, naive strategy of replacing the image classifier with the patient data classifier on slides with low output scores improved balanced accuracy to 86.72% ± 0.36%. In most cases, the CNN on its own was so accurate that patient data integration did not provide any benefit. However, incorporating patient data for lesions that were classified by the CNN with low 'confidence' improved balanced accuracy. • Pathologists incorporate patient data in addition to clinical examination. • They put more emphasis on patient data if they are uncertain. • We investigated fusing histologic image/patient data within CNN-based classifiers. • State-of-the-art fusing approaches in general did not yield a performance benefit. • Mimicking humans by fusing patient data only if CNN was uncertain raised accuracy. [ABSTRACT FROM AUTHOR]

Lymph node status is a prognostic marker and strongly influences therapeutic decisions in colorectal cancer (CRC). The objective of the study is to investigate whether image features extracted by a deep learning model from routine histological slides and/or clinical data can be used to predict CRC lymph node metastasis (LNM). Using histological whole slide images (WSIs) of primary tumours of 2431 patients in the DACHS cohort, we trained a convolutional neural network to predict LNM. In parallel, we used clinical data derived from the same cases in logistic regression analyses. Subsequently, the slide-based artificial intelligence predictor (SBAIP) score was included in the regression. WSIs and data from 582 patients of the TCGA cohort were used as the external test set. On the internal test set, the SBAIP achieved an area under receiver operating characteristic (AUROC) of 71.0%, the clinical classifier achieved an AUROC of 67.0% and a combination of the two classifiers yielded an improvement to 74.1%. Whereas the clinical classifier's performance remained stable on the TCGA set, performance of the SBAIP dropped to an AUROC of 61.2%. Performance of the clinical classifier depended strongly on the T stage. Deep learning–based image analysis may help predict LNM of patients with CRC using routine histological slides. Combination with clinical data such as T stage might be useful. Strategies to increase performance of the SBAIP on external images should be investigated. • We aimed to develop a new biomarker to predict lymph node status of patients with colorectal cancer (CRC). • Convolutional neural network–based image analysis and clinical data were used individually and in combination. • Models were trained and tested on haematoxylin and eosin slides of 3013 primary CRC tumours. • The combined model on the internal test set yielded the best results (area under receiver operating curve of 74.1%). • Studies should address generalisation of the image classifier to external data. [ABSTRACT FROM AUTHOR]

Gastrointestinal cancers account for approximately 20% of all cancer diagnoses and are responsible for 22.5% of cancer deaths worldwide. Artificial intelligence–based diagnostic support systems, in particular convolutional neural network (CNN)–based image analysis tools, have shown great potential in medical computer vision. In this systematic review, we summarise recent studies reporting CNN-based approaches for digital biomarkers for characterization and prognostication of gastrointestinal cancer pathology. Pubmed and Medline were screened for peer-reviewed papers dealing with CNN-based gastrointestinal cancer analyses from histological slides, published between 2015 and 2020.Seven hundred and ninety titles and abstracts were screened, and 58 full-text articles were assessed for eligibility. Sixteen publications fulfilled our inclusion criteria dealing with tumor or precursor lesion characterization or prognostic and predictive biomarkers: 14 studies on colorectal or rectal cancer, three studies on gastric cancer and none on esophageal cancer. These studies were categorised according to their end-points: polyp characterization, tumor characterization and patient outcome. Regarding the translation into clinical practice, we identified several studies demonstrating generalization of the classifier with external tests and comparisons with pathologists, but none presenting clinical implementation. Results of recent studies on CNN-based image analysis in gastrointestinal cancer pathology are promising, but studies were conducted in observational and retrospective settings. Large-scale trials are needed to assess performance and predict clinical usefulness. Furthermore, large-scale trials are required for approval of CNN-based prediction models as medical devices. • Computational medical image analysis is rising in gastrointestinal cancer pathology. • Several studies deal with CNN–based identification of biomarkers on HE slides. • Results are promising but studies are at early stages. • Further studies are needed to assess performance and potential clinical usefulness. [ABSTRACT FROM AUTHOR]