Search

A basic requirement for artificial intelligence (AI)–based image analysis systems, which are to be integrated into clinical practice, is a high robustness. Minor changes in how those images are acquired, for example, during routine skin cancer screening, should not change the diagnosis of such assistance systems. To quantify to what extent minor image perturbations affect the convolutional neural network (CNN)–mediated skin lesion classification and to evaluate three possible solutions for this problem (additional data augmentation, test-time augmentation, anti-aliasing). We trained three commonly used CNN architectures to differentiate between dermoscopic melanoma and nevus images. Subsequently, their performance and susceptibility to minor changes ('brittleness') was tested on two distinct test sets with multiple images per lesion. For the first set, image changes, such as rotations or zooms, were generated artificially. The second set contained natural changes that stemmed from multiple photographs taken of the same lesions. All architectures exhibited brittleness on the artificial and natural test set. The three reviewed methods were able to decrease brittleness to varying degrees while still maintaining performance. The observed improvement was greater for the artificial than for the natural test set, where enhancements were minor. Minor image changes, relatively inconspicuous for humans, can have an effect on the robustness of CNNs differentiating skin lesions. By the methods tested here, this effect can be reduced, but not fully eliminated. Thus, further research to sustain the performance of AI classifiers is needed to facilitate the translation of such systems into the clinic. • Convolutional neural networks (CNNs) show potential in skin cancer diagnoses. • Minor dermoscopic image changes suffice to influence the CNNs' diagnosis. • CNNs' susceptibility to such changes can be reduced but not eliminated. • Practitioners need to be aware and attempt to minimise this phenomenon. [ABSTRACT FROM AUTHOR]

Image-based cancer classifiers suffer from a variety of problems which negatively affect their performance. For example, variation in image brightness or different cameras can already suffice to diminish performance. Ensemble solutions, where multiple model predictions are combined into one, can improve these problems. However, ensembles are computationally intensive and less transparent to practitioners than single model solutions. Constructing model soups, by averaging the weights of multiple models into a single model, could circumvent these limitations while still improving performance. To investigate the performance of model soups for a dermoscopic melanoma-nevus skin cancer classification task with respect to (1) generalisation to images from other clinics, (2) robustness against small image changes and (3) calibration such that the confidences correspond closely to the actual predictive uncertainties. We construct model soups by fine-tuning pre-trained models on seven different image resolutions and subsequently averaging their weights. Performance is evaluated on a multi-source dataset including holdout and external components. We find that model soups improve generalisation and calibration on the external component while maintaining performance on the holdout component. For robustness, we observe performance improvements for pertubated test images, while the performance on corrupted test images remains on par. Overall, souping for skin cancer classifiers has a positive effect on generalisation, robustness and calibration. It is easy for practitioners to implement and by combining multiple models into a single model, complexity is reduced. This could be an important factor in achieving clinical applicability, as less complexity generally means more transparency. • Artificial intelligence-based skin cancer classifiers suffer from generalisation and robustness issues. • Ensemble solutions improve these issues but are expensive and complex. • Model soups combine multiple models into a single model of equal size and complexity. • Model soups improve classifier performance on images from other clinics. • They also have a positive effect on model robustness and calibration. [ABSTRACT FROM AUTHOR]

One prominent application for deep learning–based classifiers is skin cancer classification on dermoscopic images. However, classifier evaluation is often limited to holdout data which can mask common shortcomings such as susceptibility to confounding factors. To increase clinical applicability, it is necessary to thoroughly evaluate such classifiers on out-of-distribution (OOD) data. The objective of the study was to establish a dermoscopic skin cancer benchmark in which classifier robustness to OOD data can be measured. Using a proprietary dermoscopic image database and a set of image transformations, we create an OOD robustness benchmark and evaluate the robustness of four different convolutional neural network (CNN) architectures on it. The benchmark contains three data sets—Skin Archive Munich (SAM), SAM-corrupted (SAM-C) and SAM-perturbed (SAM-P)—and is publicly available for download. To maintain the benchmark's OOD status, ground truth labels are not provided and test results should be sent to us for assessment. The SAM data set contains 319 unmodified and biopsy-verified dermoscopic melanoma (n = 194) and nevus (n = 125) images. SAM-C and SAM-P contain images from SAM which were artificially modified to test a classifier against low-quality inputs and to measure its prediction stability over small image changes, respectively. All four CNNs showed susceptibility to corruptions and perturbations. This benchmark provides three data sets which allow for OOD testing of binary skin cancer classifiers. Our classifier performance confirms the shortcomings of CNNs and provides a frame of reference. Altogether, this benchmark should facilitate a more thorough evaluation process and thereby enable the development of more robust skin cancer classifiers. • AI-based skin cancer classifier testing is often limited to holdout data. • However, holdout testing can mask severe susceptibilities and shortcomings. • Therefore, testing on out-of-distribution (OOD) data should become standard practice. • We release a dermoscopic skin cancer image benchmark designed for OOD testing. • This benchmark incorporates various distribution shifts such as corrupted images. [ABSTRACT FROM AUTHOR]

Historically, cancer diagnoses have been made by pathologists using two-dimensional histological slides. However, with the advent of digital pathology and artificial intelligence, slides are being digitised, providing new opportunities to integrate their information. Since nature is 3-dimensional (3D), it seems intuitive to digitally reassemble the 3D structure for diagnosis. To develop the first human-3D-melanoma-histology-model with full data and code availability. Further, to evaluate the 3D-simulation together with experienced pathologists in the field and discuss the implications of digital 3D-models for the future of digital pathology. A malignant melanoma of the skin was digitised via 3 µm cuts by a slide scanner; an open-source software was then leveraged to construct the 3D model. A total of nine pathologists from four different countries with at least 10 years of experience in the histologic diagnosis of melanoma tested the model and discussed their experiences as well as implications for future pathology. We successfully constructed and tested the first 3D-model of human melanoma. Based on testing, 88.9% of pathologists believe that the technology is likely to enter routine pathology within the next 10 years; advantages include a better reflectance of anatomy, 3D assessment of symmetry and the opportunity to simultaneously evaluate different tissue levels at the same time; limitations include the high consumption of tissue and a yet inferior resolution due to computational limitations. 3D-histology-models are promising for digital pathology of cancer and melanoma specifically, however, there are yet limitations which need to be carefully addressed. • Presentation of the first human 3D-melanoma-histology-model. • Nine pathologists from four different countries evaluated the model. • Implications, limitations and the likelihood of a 2D to 3D transition are discussed. • Further development is ensured by complete data and code availability. [ABSTRACT FROM AUTHOR]

In recent studies, convolutional neural networks (CNNs) outperformed dermatologists in distinguishing dermoscopic images of melanoma and nevi. In these studies, dermatologists and artificial intelligence were considered as opponents. However, the combination of classifiers frequently yields superior results, both in machine learning and among humans. In this study, we investigated the potential benefit of combining human and artificial intelligence for skin cancer classification. Using 11,444 dermoscopic images, which were divided into five diagnostic categories, novel deep learning techniques were used to train a single CNN. Then, both 112 dermatologists of 13 German university hospitals and the trained CNN independently classified a set of 300 biopsy-verified skin lesions into those five classes. Taking into account the certainty of the decisions, the two independently determined diagnoses were combined to a new classifier with the help of a gradient boosting method. The primary end-point of the study was the correct classification of the images into five designated categories, whereas the secondary end-point was the correct classification of lesions as either benign or malignant (binary classification). Regarding the multiclass task, the combination of man and machine achieved an accuracy of 82.95%. This was 1.36% higher than the best of the two individual classifiers (81.59% achieved by the CNN). Owing to the class imbalance in the binary problem, sensitivity, but not accuracy, was examined and demonstrated to be superior (89%) to the best individual classifier (CNN with 86.1%). The specificity in the combined classifier decreased from 89.2% to 84%. However, at an equal sensitivity of 89%, the CNN achieved a specificity of only 81.5% Our findings indicate that the combination of human and artificial intelligence achieves superior results over the independent results of both of these systems. • This article describes the first experiment on combining human and artificial intelligence for the classification of images suspicious of skin cancer. • The combination achieved a superior accuracy of 82.95% (compared to 81.59%/42.94% achieved by artificial/human intelligence alone). • The combination of human and artificial intelligence indicates superiority over a separated approach. [ABSTRACT FROM AUTHOR]

Recent research revealed the superiority of artificial intelligence over dermatologists to diagnose melanoma from images. However, 30–50% of all melanomas and more than half of those in young patients evolve from initially benign lesions. Despite its high relevance for melanoma screening, neither clinicians nor computers are yet able to reliably predict a nevus' oncologic transformation. The cause of this lies in the static nature of lesion presentation in the current standard of care, both for clinicians and algorithms. The status quo makes it difficult to train algorithms (and clinicians) to precisely assess the likelihood of a benign skin lesion to transform into melanoma. In addition, it inhibits the precision of current algorithms since 'evolution' image features may not be part of their decision. The current literature reveals certain types of melanocytic nevi (i.e. 'spitzoid' or 'dysplastic' nevi) and criteria (i.e. visible vasculature) that, in general, appear to have a higher chance to transform into melanoma. However, owing to the cumulative nature of oncogenic mutations in melanoma, a more fine-grained early morphologic footprint is likely to be detectable by an algorithm. In this perspective article, the concept of melanoma prediction is further explored by the discussion of the evolution of melanoma, the concept for training of such a nevi classifier and the implications of early melanoma prediction for clinical practice. In conclusion, the authors believe that artificial intelligence trained on prospective image data could be transformative for skin cancer diagnostics by (a) predicting melanoma before it occurs (i.e. pre-in situ) and (b) further enhancing the accuracy of current melanoma classifiers. Necessary prospective images for this research are obtained via free mole-monitoring mobile apps. • Thirty percent to 50% of all melanomas and more than half of those in young patients evolve from initially benign lesions. • Neither clinicians nor artificial intelligence (AI) algorithms are yet able to reliably predict a nevus' oncologic transformation. • AI trained on prospective image data could predict melanoma before it occurs and enhance the accuracy of diagnoses. [ABSTRACT FROM AUTHOR]

Recent studies have successfully demonstrated the use of deep-learning algorithms for dermatologist-level classification of suspicious lesions by the use of excessive proprietary image databases and limited numbers of dermatologists. For the first time, the performance of a deep-learning algorithm trained by open-source images exclusively is compared to a large number of dermatologists covering all levels within the clinical hierarchy. We used methods from enhanced deep learning to train a convolutional neural network (CNN) with 12,378 open-source dermoscopic images. We used 100 images to compare the performance of the CNN to that of the 157 dermatologists from 12 university hospitals in Germany. Outperformance of dermatologists by the deep neural network was measured in terms of sensitivity, specificity and receiver operating characteristics. The mean sensitivity and specificity achieved by the dermatologists with dermoscopic images was 74.1% (range 40.0%–100%) and 60% (range 21.3%–91.3%), respectively. At a mean sensitivity of 74.1%, the CNN exhibited a mean specificity of 86.5% (range 70.8%–91.3%). At a mean specificity of 60%, a mean sensitivity of 87.5% (range 80%–95%) was achieved by our algorithm. Among the dermatologists, the chief physicians showed the highest mean specificity of 69.2% at a mean sensitivity of 73.3%. With the same high specificity of 69.2%, the CNN had a mean sensitivity of 84.5%. A CNN trained by open-source images exclusively outperformed 136 of the 157 dermatologists and all the different levels of experience (from junior to chief physicians) in terms of average specificity and sensitivity. • A convolutional neural network (CNN) received enhanced training with 12,378 open-source dermoscopic images. • In a head-to-head comparison, the CNN outperformed 136 of 157 participating dermatologists. • The CNN was capable to outperform dermatologists of all hierarchical subgroups (from junior to chief physicians) in dermoscopic melanoma image classification. [ABSTRACT FROM AUTHOR]

Abstract Background Recent studies have demonstrated the use of convolutional neural networks (CNNs) to classify images of melanoma with accuracies comparable to those achieved by board-certified dermatologists. However, the performance of a CNN exclusively trained with dermoscopic images in a clinical image classification task in direct competition with a large number of dermatologists has not been measured to date. This study compares the performance of a convolutional neuronal network trained with dermoscopic images exclusively for identifying melanoma in clinical photographs with the manual grading of the same images by dermatologists. Methods We compared automatic digital melanoma classification with the performance of 145 dermatologists of 12 German university hospitals. We used methods from enhanced deep learning to train a CNN with 12,378 open-source dermoscopic images. We used 100 clinical images to compare the performance of the CNN to that of the dermatologists. Dermatologists were compared with the deep neural network in terms of sensitivity, specificity and receiver operating characteristics. Findings The mean sensitivity and specificity achieved by the dermatologists with clinical images was 89.4% (range: 55.0%–100%) and 64.4% (range: 22.5%–92.5%). At the same sensitivity, the CNN exhibited a mean specificity of 68.2% (range 47.5%–86.25%). Among the dermatologists, the attendings showed the highest mean sensitivity of 92.8% at a mean specificity of 57.7%. With the same high sensitivity of 92.8%, the CNN had a mean specificity of 61.1%. Interpretation For the first time, dermatologist-level image classification was achieved on a clinical image classification task without training on clinical images. The CNN had a smaller variance of results indicating a higher robustness of computer vision compared with human assessment for dermatologic image classification tasks. Highlights • A convolutional neural network (CNN) received enhanced training with 12,378 open-source dermoscopic images but was compared with dermatologists for classification of clinical images. • The mean sensitivity and specificity achieved by the dermatologists with clinical images was 89.4% and 64.4%. At the same sensitivity, the CNN exhibited a mean specificity of 68.2%. • For the first time, a CNN performed on par with dermatologists on a clinical image classification task without training on clinical images. [ABSTRACT FROM AUTHOR]