Search

A basic requirement for artificial intelligence (AI)–based image analysis systems, which are to be integrated into clinical practice, is a high robustness. Minor changes in how those images are acquired, for example, during routine skin cancer screening, should not change the diagnosis of such assistance systems. To quantify to what extent minor image perturbations affect the convolutional neural network (CNN)–mediated skin lesion classification and to evaluate three possible solutions for this problem (additional data augmentation, test-time augmentation, anti-aliasing). We trained three commonly used CNN architectures to differentiate between dermoscopic melanoma and nevus images. Subsequently, their performance and susceptibility to minor changes ('brittleness') was tested on two distinct test sets with multiple images per lesion. For the first set, image changes, such as rotations or zooms, were generated artificially. The second set contained natural changes that stemmed from multiple photographs taken of the same lesions. All architectures exhibited brittleness on the artificial and natural test set. The three reviewed methods were able to decrease brittleness to varying degrees while still maintaining performance. The observed improvement was greater for the artificial than for the natural test set, where enhancements were minor. Minor image changes, relatively inconspicuous for humans, can have an effect on the robustness of CNNs differentiating skin lesions. By the methods tested here, this effect can be reduced, but not fully eliminated. Thus, further research to sustain the performance of AI classifiers is needed to facilitate the translation of such systems into the clinic. • Convolutional neural networks (CNNs) show potential in skin cancer diagnoses. • Minor dermoscopic image changes suffice to influence the CNNs' diagnosis. • CNNs' susceptibility to such changes can be reduced but not eliminated. • Practitioners need to be aware and attempt to minimise this phenomenon. [ABSTRACT FROM AUTHOR]

Image-based cancer classifiers suffer from a variety of problems which negatively affect their performance. For example, variation in image brightness or different cameras can already suffice to diminish performance. Ensemble solutions, where multiple model predictions are combined into one, can improve these problems. However, ensembles are computationally intensive and less transparent to practitioners than single model solutions. Constructing model soups, by averaging the weights of multiple models into a single model, could circumvent these limitations while still improving performance. To investigate the performance of model soups for a dermoscopic melanoma-nevus skin cancer classification task with respect to (1) generalisation to images from other clinics, (2) robustness against small image changes and (3) calibration such that the confidences correspond closely to the actual predictive uncertainties. We construct model soups by fine-tuning pre-trained models on seven different image resolutions and subsequently averaging their weights. Performance is evaluated on a multi-source dataset including holdout and external components. We find that model soups improve generalisation and calibration on the external component while maintaining performance on the holdout component. For robustness, we observe performance improvements for pertubated test images, while the performance on corrupted test images remains on par. Overall, souping for skin cancer classifiers has a positive effect on generalisation, robustness and calibration. It is easy for practitioners to implement and by combining multiple models into a single model, complexity is reduced. This could be an important factor in achieving clinical applicability, as less complexity generally means more transparency. • Artificial intelligence-based skin cancer classifiers suffer from generalisation and robustness issues. • Ensemble solutions improve these issues but are expensive and complex. • Model soups combine multiple models into a single model of equal size and complexity. • Model soups improve classifier performance on images from other clinics. • They also have a positive effect on model robustness and calibration. [ABSTRACT FROM AUTHOR]

One prominent application for deep learning–based classifiers is skin cancer classification on dermoscopic images. However, classifier evaluation is often limited to holdout data which can mask common shortcomings such as susceptibility to confounding factors. To increase clinical applicability, it is necessary to thoroughly evaluate such classifiers on out-of-distribution (OOD) data. The objective of the study was to establish a dermoscopic skin cancer benchmark in which classifier robustness to OOD data can be measured. Using a proprietary dermoscopic image database and a set of image transformations, we create an OOD robustness benchmark and evaluate the robustness of four different convolutional neural network (CNN) architectures on it. The benchmark contains three data sets—Skin Archive Munich (SAM), SAM-corrupted (SAM-C) and SAM-perturbed (SAM-P)—and is publicly available for download. To maintain the benchmark's OOD status, ground truth labels are not provided and test results should be sent to us for assessment. The SAM data set contains 319 unmodified and biopsy-verified dermoscopic melanoma (n = 194) and nevus (n = 125) images. SAM-C and SAM-P contain images from SAM which were artificially modified to test a classifier against low-quality inputs and to measure its prediction stability over small image changes, respectively. All four CNNs showed susceptibility to corruptions and perturbations. This benchmark provides three data sets which allow for OOD testing of binary skin cancer classifiers. Our classifier performance confirms the shortcomings of CNNs and provides a frame of reference. Altogether, this benchmark should facilitate a more thorough evaluation process and thereby enable the development of more robust skin cancer classifiers. • AI-based skin cancer classifier testing is often limited to holdout data. • However, holdout testing can mask severe susceptibilities and shortcomings. • Therefore, testing on out-of-distribution (OOD) data should become standard practice. • We release a dermoscopic skin cancer image benchmark designed for OOD testing. • This benchmark incorporates various distribution shifts such as corrupted images. [ABSTRACT FROM AUTHOR]

Sentinel lymph node status is a central prognostic factor for melanomas. However, the surgical excision involves some risks for affected patients. In this study, we therefore aimed to develop a digital biomarker that can predict lymph node metastasis non-invasively from digitised H&E slides of primary melanoma tumours. A total of 415 H&E slides from primary melanoma tumours with known sentinel node (SN) status from three German university hospitals and one private pathological practice were digitised (150 SN positive/265 SN negative). Two hundred ninety-one slides were used to train artificial neural networks (ANNs). The remaining 124 slides were used to test the ability of the ANNs to predict sentinel status. ANNs were trained and/or tested on data sets that were matched or not matched between SN-positive and SN-negative cases for patient age, ulceration, and tumour thickness, factors that are known to correlate with lymph node status. The best accuracy was achieved by an ANN that was trained and tested on unmatched cases (61.8% ± 0.2%) area under the receiver operating characteristic (AUROC). In contrast, ANNs that were trained and/or tested on matched cases achieved (55.0% ± 3.5%) AUROC or less. Our results indicate that the image classifier can predict lymph node status to some, albeit so far not clinically relevant, extent. It may do so by mostly detecting equivalents of factors on histological slides that are already known to correlate with lymph node status. Our results provide a basis for future research with larger data cohorts. • Lymph node analysis conveys relevant prognostic information. • Surgical removal of lymph nodes can be associated with considerable morbidity. • Thus, biomarkers that allow prediction of lymph node status are needed. • We trained a CNN on H&E slides of node positive and -negative primary melanomas. • This classifier could predict sentinel node status to some extent. [ABSTRACT FROM AUTHOR]

Historically, cancer diagnoses have been made by pathologists using two-dimensional histological slides. However, with the advent of digital pathology and artificial intelligence, slides are being digitised, providing new opportunities to integrate their information. Since nature is 3-dimensional (3D), it seems intuitive to digitally reassemble the 3D structure for diagnosis. To develop the first human-3D-melanoma-histology-model with full data and code availability. Further, to evaluate the 3D-simulation together with experienced pathologists in the field and discuss the implications of digital 3D-models for the future of digital pathology. A malignant melanoma of the skin was digitised via 3 µm cuts by a slide scanner; an open-source software was then leveraged to construct the 3D model. A total of nine pathologists from four different countries with at least 10 years of experience in the histologic diagnosis of melanoma tested the model and discussed their experiences as well as implications for future pathology. We successfully constructed and tested the first 3D-model of human melanoma. Based on testing, 88.9% of pathologists believe that the technology is likely to enter routine pathology within the next 10 years; advantages include a better reflectance of anatomy, 3D assessment of symmetry and the opportunity to simultaneously evaluate different tissue levels at the same time; limitations include the high consumption of tissue and a yet inferior resolution due to computational limitations. 3D-histology-models are promising for digital pathology of cancer and melanoma specifically, however, there are yet limitations which need to be carefully addressed. • Presentation of the first human 3D-melanoma-histology-model. • Nine pathologists from four different countries evaluated the model. • Implications, limitations and the likelihood of a 2D to 3D transition are discussed. • Further development is ensured by complete data and code availability. [ABSTRACT FROM AUTHOR]

In recent studies, convolutional neural networks (CNNs) outperformed dermatologists in distinguishing dermoscopic images of melanoma and nevi. In these studies, dermatologists and artificial intelligence were considered as opponents. However, the combination of classifiers frequently yields superior results, both in machine learning and among humans. In this study, we investigated the potential benefit of combining human and artificial intelligence for skin cancer classification. Using 11,444 dermoscopic images, which were divided into five diagnostic categories, novel deep learning techniques were used to train a single CNN. Then, both 112 dermatologists of 13 German university hospitals and the trained CNN independently classified a set of 300 biopsy-verified skin lesions into those five classes. Taking into account the certainty of the decisions, the two independently determined diagnoses were combined to a new classifier with the help of a gradient boosting method. The primary end-point of the study was the correct classification of the images into five designated categories, whereas the secondary end-point was the correct classification of lesions as either benign or malignant (binary classification). Regarding the multiclass task, the combination of man and machine achieved an accuracy of 82.95%. This was 1.36% higher than the best of the two individual classifiers (81.59% achieved by the CNN). Owing to the class imbalance in the binary problem, sensitivity, but not accuracy, was examined and demonstrated to be superior (89%) to the best individual classifier (CNN with 86.1%). The specificity in the combined classifier decreased from 89.2% to 84%. However, at an equal sensitivity of 89%, the CNN achieved a specificity of only 81.5% Our findings indicate that the combination of human and artificial intelligence achieves superior results over the independent results of both of these systems. • This article describes the first experiment on combining human and artificial intelligence for the classification of images suspicious of skin cancer. • The combination achieved a superior accuracy of 82.95% (compared to 81.59%/42.94% achieved by artificial/human intelligence alone). • The combination of human and artificial intelligence indicates superiority over a separated approach. [ABSTRACT FROM AUTHOR]

Recent research revealed the superiority of artificial intelligence over dermatologists to diagnose melanoma from images. However, 30–50% of all melanomas and more than half of those in young patients evolve from initially benign lesions. Despite its high relevance for melanoma screening, neither clinicians nor computers are yet able to reliably predict a nevus' oncologic transformation. The cause of this lies in the static nature of lesion presentation in the current standard of care, both for clinicians and algorithms. The status quo makes it difficult to train algorithms (and clinicians) to precisely assess the likelihood of a benign skin lesion to transform into melanoma. In addition, it inhibits the precision of current algorithms since 'evolution' image features may not be part of their decision. The current literature reveals certain types of melanocytic nevi (i.e. 'spitzoid' or 'dysplastic' nevi) and criteria (i.e. visible vasculature) that, in general, appear to have a higher chance to transform into melanoma. However, owing to the cumulative nature of oncogenic mutations in melanoma, a more fine-grained early morphologic footprint is likely to be detectable by an algorithm. In this perspective article, the concept of melanoma prediction is further explored by the discussion of the evolution of melanoma, the concept for training of such a nevi classifier and the implications of early melanoma prediction for clinical practice. In conclusion, the authors believe that artificial intelligence trained on prospective image data could be transformative for skin cancer diagnostics by (a) predicting melanoma before it occurs (i.e. pre-in situ) and (b) further enhancing the accuracy of current melanoma classifiers. Necessary prospective images for this research are obtained via free mole-monitoring mobile apps. • Thirty percent to 50% of all melanomas and more than half of those in young patients evolve from initially benign lesions. • Neither clinicians nor artificial intelligence (AI) algorithms are yet able to reliably predict a nevus' oncologic transformation. • AI trained on prospective image data could predict melanoma before it occurs and enhance the accuracy of diagnoses. [ABSTRACT FROM AUTHOR]

Melanoma is the most dangerous type of skin cancer but is curable if detected early. Recent publications demonstrated that artificial intelligence is capable in classifying images of benign nevi and melanoma with dermatologist-level precision. However, a statistically significant improvement compared with dermatologist classification has not been reported to date. For this comparative study, 4204 biopsy-proven images of melanoma and nevi (1:1) were used for the training of a convolutional neural network (CNN). New techniques of deep learning were integrated. For the experiment, an additional 804 biopsy-proven dermoscopic images of melanoma and nevi (1:1) were randomly presented to dermatologists of nine German university hospitals, who evaluated the quality of each image and stated their recommended treatment (19,296 recommendations in total). Three McNemar's tests comparing the results of the CNN's test runs in terms of sensitivity, specificity and overall correctness were predefined as the main outcomes. The respective sensitivity and specificity of lesion classification by the dermatologists were 67.2% (95% confidence interval [CI]: 62.6%–71.7%) and 62.2% (95% CI: 57.6%–66.9%). In comparison, the trained CNN achieved a higher sensitivity of 82.3% (95% CI: 78.3%–85.7%) and a higher specificity of 77.9% (95% CI: 73.8%–81.8%). The three McNemar's tests in 2 × 2 tables all reached a significance level of p < 0.001. This significance level was sustained for both subgroups. For the first time, automated dermoscopic melanoma image classification was shown to be significantly superior to both junior and board-certified dermatologists (p < 0.001). • Recent publications demonstrated that deep learning is capable to classify images of benign nevi and melanoma with dermatologist-level precision. • A systematic outperformance of dermatologists was not demonstrated to date. • This study shows the first systematic (p < 0.001) outperformance of board-certified dermatologists in dermoscopic melanoma image classification. [ABSTRACT FROM AUTHOR]

The diagnosis of most cancers is made by a board-certified pathologist based on a tissue biopsy under the microscope. Recent research reveals a high discordance between individual pathologists. For melanoma, the literature reports on 25–26% of discordance for classifying a benign nevus versus malignant melanoma. A recent study indicated the potential of deep learning to lower these discordances. However, the performance of deep learning in classifying histopathologic melanoma images was never compared directly to human experts. The aim of this study is to perform such a first direct comparison. A total of 695 lesions were classified by an expert histopathologist in accordance with current guidelines (350 nevi/345 melanoma). Only the haematoxylin & eosin (H&E) slides of these lesions were digitalised via a slide scanner and then randomly cropped. A total of 595 of the resulting images were used to train a convolutional neural network (CNN). The additional 100 H&E image sections were used to test the results of the CNN in comparison to 11 histopathologists. Three combined McNemar tests comparing the results of the CNNs test runs in terms of sensitivity, specificity and accuracy were predefined to test for significance (p < 0.05). The CNN achieved a mean sensitivity/specificity/accuracy of 76%/60%/68% over 11 test runs. In comparison, the 11 pathologists achieved a mean sensitivity/specificity/accuracy of 51.8%/66.5%/59.2%. Thus, the CNN was significantly (p = 0.016) superior in classifying the cropped images. With limited image information available, a CNN was able to outperform 11 histopathologists in the classification of histopathological melanoma images and thus shows promise to assist human melanoma diagnoses. • A convolutional neural network (CNN) was trained with 595 histopathologic images of melanoma and nevi. • In a direct comparison, the CNN and 11 histopathologists classified a test set of 100 additional histopathologic images (1:1 melanoma/nevi). • The CNN systematically outperformed the 11 histopathologists in terms of overall accuracy, sensitivity and specificity (p = 0.016). [ABSTRACT FROM AUTHOR]

The diagnosis of most cancers is made by a board-certified pathologist based on a tissue biopsy under the microscope. Recent research reveals a high discordance between individual pathologists. For melanoma, the literature reports 25–26% of discordance for classifying a benign nevus versus malignant melanoma. Deep learning was successfully implemented to enhance the precision of lung and breast cancer diagnoses. The aim of this study is to illustrate the potential of deep learning to assist human assessment for a histopathologic melanoma diagnosis. Six hundred ninety-five lesions were classified by an expert histopathologist in accordance with current guidelines (350 nevi and 345 melanomas). Only the haematoxylin and eosin stained (H&E) slides of these lesions were digitalised using a slide scanner and then randomly cropped. Five hundred ninety-five of the resulting images were used for the training of a convolutional neural network (CNN). The additional 100 H&E image sections were used to test the results of the CNN in comparison with the original class labels. The total discordance with the histopathologist was 18% for melanoma (95% confidence interval [CI]: 7.4–28.6%), 20% for nevi (95% CI: 8.9–31.1%) and 19% for the full set of images (95% CI: 11.3–26.7%). Even in the worst case, the discordance of the CNN was about the same compared with the discordance between human pathologists as reported in the literature. Despite the vastly reduced amount of data, time necessary for diagnosis and cost compared with the pathologist, our CNN archived on-par performance. Conclusively, CNNs indicate to be a valuable tool to assist human melanoma diagnoses. • A convolutional neural network (CNN) was trained with 595 histopathologic images of melanomas and nevi that were classified by an expert dermatohistopathologist. • The CNN was then tested with 100 additional images (melanoma/nevi = 1:1) and revealed a discordance of only 19% to the histopathologist. • Thus, even in the worst case, the discordance of the CNN is about the same compared with the discordance between human pathologists as reported in the literature (25–26%). [ABSTRACT FROM AUTHOR]

Recent studies have successfully demonstrated the use of deep-learning algorithms for dermatologist-level classification of suspicious lesions by the use of excessive proprietary image databases and limited numbers of dermatologists. For the first time, the performance of a deep-learning algorithm trained by open-source images exclusively is compared to a large number of dermatologists covering all levels within the clinical hierarchy. We used methods from enhanced deep learning to train a convolutional neural network (CNN) with 12,378 open-source dermoscopic images. We used 100 images to compare the performance of the CNN to that of the 157 dermatologists from 12 university hospitals in Germany. Outperformance of dermatologists by the deep neural network was measured in terms of sensitivity, specificity and receiver operating characteristics. The mean sensitivity and specificity achieved by the dermatologists with dermoscopic images was 74.1% (range 40.0%–100%) and 60% (range 21.3%–91.3%), respectively. At a mean sensitivity of 74.1%, the CNN exhibited a mean specificity of 86.5% (range 70.8%–91.3%). At a mean specificity of 60%, a mean sensitivity of 87.5% (range 80%–95%) was achieved by our algorithm. Among the dermatologists, the chief physicians showed the highest mean specificity of 69.2% at a mean sensitivity of 73.3%. With the same high specificity of 69.2%, the CNN had a mean sensitivity of 84.5%. A CNN trained by open-source images exclusively outperformed 136 of the 157 dermatologists and all the different levels of experience (from junior to chief physicians) in terms of average specificity and sensitivity. • A convolutional neural network (CNN) received enhanced training with 12,378 open-source dermoscopic images. • In a head-to-head comparison, the CNN outperformed 136 of 157 participating dermatologists. • The CNN was capable to outperform dermatologists of all hierarchical subgroups (from junior to chief physicians) in dermoscopic melanoma image classification. [ABSTRACT FROM AUTHOR]

Abstract Background Several recent publications have demonstrated the use of convolutional neural networks to classify images of melanoma at par with board-certified dermatologists. However, the non-availability of a public human benchmark restricts the comparability of the performance of these algorithms and thereby the technical progress in this field. Methods An electronic questionnaire was sent to dermatologists at 12 German university hospitals. Each questionnaire comprised 100 dermoscopic and 100 clinical images (80 nevi images and 20 biopsy-verified melanoma images, each), all open-source. The questionnaire recorded factors such as the years of experience in dermatology, performed skin checks, age, sex and the rank within the university hospital or the status as resident physician. For each image, the dermatologists were asked to provide a management decision (treat/biopsy lesion or reassure the patient). Main outcome measures were sensitivity, specificity and the receiver operating characteristics (ROC). Results Total 157 dermatologists assessed all 100 dermoscopic images with an overall sensitivity of 74.1%, specificity of 60.0% and an ROC of 0.67 (range = 0.538–0.769); 145 dermatologists assessed all 100 clinical images with an overall sensitivity of 89.4%, specificity of 64.4% and an ROC of 0.769 (range = 0.613–0.9). Results between test-sets were significantly different (P < 0.05) confirming the need for a standardised benchmark. Conclusions We present the first public melanoma classification benchmark for both non-dermoscopic and dermoscopic images for comparing artificial intelligence algorithms with diagnostic performance of 145 or 157 dermatologists. Melanoma Classification Benchmark should be considered as a reference standard for white-skinned Western populations in the field of binary algorithmic melanoma classification. Highlights • This paper provides the first open access melanoma classification benchmark for both non-dermoscopic and dermoscopic images. • Algorithms can now be easily compared to the performance of dermatologists in terms of sensitivity, specificity and ROC. • The melanoma benchmark allows comparability between algorithms of different publications and provides a new reference standard. [ABSTRACT FROM AUTHOR]

Abstract Background Recent studies have demonstrated the use of convolutional neural networks (CNNs) to classify images of melanoma with accuracies comparable to those achieved by board-certified dermatologists. However, the performance of a CNN exclusively trained with dermoscopic images in a clinical image classification task in direct competition with a large number of dermatologists has not been measured to date. This study compares the performance of a convolutional neuronal network trained with dermoscopic images exclusively for identifying melanoma in clinical photographs with the manual grading of the same images by dermatologists. Methods We compared automatic digital melanoma classification with the performance of 145 dermatologists of 12 German university hospitals. We used methods from enhanced deep learning to train a CNN with 12,378 open-source dermoscopic images. We used 100 clinical images to compare the performance of the CNN to that of the dermatologists. Dermatologists were compared with the deep neural network in terms of sensitivity, specificity and receiver operating characteristics. Findings The mean sensitivity and specificity achieved by the dermatologists with clinical images was 89.4% (range: 55.0%–100%) and 64.4% (range: 22.5%–92.5%). At the same sensitivity, the CNN exhibited a mean specificity of 68.2% (range 47.5%–86.25%). Among the dermatologists, the attendings showed the highest mean sensitivity of 92.8% at a mean specificity of 57.7%. With the same high sensitivity of 92.8%, the CNN had a mean specificity of 61.1%. Interpretation For the first time, dermatologist-level image classification was achieved on a clinical image classification task without training on clinical images. The CNN had a smaller variance of results indicating a higher robustness of computer vision compared with human assessment for dermatologic image classification tasks. Highlights • A convolutional neural network (CNN) received enhanced training with 12,378 open-source dermoscopic images but was compared with dermatologists for classification of clinical images. • The mean sensitivity and specificity achieved by the dermatologists with clinical images was 89.4% and 64.4%. At the same sensitivity, the CNN exhibited a mean specificity of 68.2%. • For the first time, a CNN performed on par with dermatologists on a clinical image classification task without training on clinical images. [ABSTRACT FROM AUTHOR]

We hypothesized that the combination of oncolytic virotherapy with immune checkpoint modulators would reduce tumor burden by direct cell lysis and stimulate antitumor immunity. In this study, we have generated attenuated Measles virus (MV) vectors encoding antibodies against CTLA-4 and PD-L1 (MV-aCTLA-4 and MV-aPD-L1). We characterized the vectors in terms of growth kinetics, antibody expression, and cytotoxicity in vitro. Immunotherapeutic effects were assessed in a newly established, fully immunocompetent murine model of malignant melanoma, B16-CD20. Analyses of tumor-infiltrating lymphocytes and restimulation experiments indicated a favorable immune profile after MV-mediated checkpoint modulation. Therapeutic benefits in terms of delayed tumor progression and prolonged median overall survival were observed for animals treated with vectors encoding anti-CTLA-4 and anti-PD-L1, respectively. Combining systemic administration of antibodies with MV treatment also improved therapeutic outcome. In vivo oncolytic efficacy against human tumors was studied in melanoma xenografts. MV-aCTLA-4 and MV-aPD-L1 were equally efficient as parental MV in this model, with high rates of complete tumor remission (> 80%). Furthermore, we could demonstrate lysis of tumor cells and transgene expression in primary tissue from melanoma patients. The current results suggest rapid translation of combining immune checkpoint modulation with oncolytic viruses into clinical application. [ABSTRACT FROM AUTHOR]