Search

Background A number of studies have shown promising performance of artificial intelligence (AI) algorithms for diagnosis of lesions in skin cancer. To date, none of these have assessed algorithm performance in the real-world setting. Objective The aim of this project is to evaluate practical issues of implementing a convolutional neural network developed by MoleMap Ltd and Monash University eResearch in the clinical setting. Methods Participants were recruited from the Alfred Hospital and Skin Health Institute, Melbourne, Australia, from November 1, 2019, to May 30, 2021. Any skin lesions of concern and at least two additional lesions were imaged using a proprietary dermoscopic camera. Images were uploaded directly to the study database by the research nurse via a custom interface installed on a clinic laptop. Doctors recorded their diagnosis and management plan for each lesion in real time. A pre-post study design was used. In the preintervention period, participating doctors were blinded to AI lesion assessment. An interim safety analysis for AI accuracy was then performed. In the postintervention period, the AI algorithm classified lesions as benign, malignant, or uncertain after the doctors’ initial assessment had been made. Doctors then had the opportunity to record an updated diagnosis and management plan. After discussing the AI diagnosis with the patient, a final management plan was agreed upon. Results Participants at both sites were high risk (for example, having a history of melanoma or being transplant recipients). 743 lesions were imaged in 214 participants. In total, 28 dermatology trainees and 17 consultant dermatologists provided diagnoses and management decisions, and 3 experienced teledermatologists provided remote assessments. A dedicated research nurse was essential to oversee study processes, maintain study documents, and assist with clinical workflow. In cases where AI algorithm and consultant dermatologist diagnoses were discordant, participant anxiety was an important factor in the final agreed management plan to biopsy or not. Conclusions Although AI algorithms are likely to be of most use in the primary care setting, higher event rates in specialist settings are important for the initial assessment of algorithm safety and accuracy. This study highlighted the importance of considering workflow issues and doctor-patient-AI interactions prior to larger-scale trials in community-based practices. Acknowledgments This research was supported by the Victorian Medical Research Acceleration Fund, with 1:1 contribution from MoleMap Ltd. VM is supported by the National Health and Medical Research Council Early Career Fellowship. CF is supported by the Monash University Research Training Program Scholarship. Conflicts of Interest SM is head of clinical research and regulatory affairs at Kahu.ai Ltd, a subsidiary of MoleMap Ltd. MH was the chief medical officer and a director of MoleMap Ltd, and holds shares in MoleMap Ltd. Trial Registration ClinicalTrials.gov NCT04040114; https://clinicaltrials.gov/ct2/show/NCT04040114

Background Convolutional neural networks (CNNs) are a type of artificial intelligence that show promise as a diagnostic aid for skin cancer. However, the majority are trained using retrospective image data sets of varying quality and image capture standardization. Objective The aim of our study is to use CNN models with the same architecture, but different training image sets, and test variability in performance when classifying skin cancer images in different populations, acquired with different devices. Additionally, we wanted to assess the performance of the models against Danish teledermatologists when tested on images acquired from Denmark. Methods Three CNNs with the same architecture were trained. CNN-NS was trained on 25,331 nonstandardized images taken from the International Skin Imaging Collaboration using different image capture devices. CNN-S was trained on 235,268 standardized images, and CNN-S2 was trained on 25,331 standardized images (matched for number and classes of training images to CNN-NS). Both standardized data sets (CNN-S and CNN-S2) were provided by Molemap using the same image capture device. A total of 495 Danish patients with 569 images of skin lesions predominantly involving Fitzpatrick skin types II and III were used to test the performance of the models. Four teledermatologists independently diagnosed and assessed the images taken of the lesions. Primary outcome measures were sensitivity, specificity, and area under the curve of the receiver operating characteristic (AUROC). Results A total of 569 images were taken from 495 patients (n=280, 57% women, n=215, 43% men; mean age 55, SD 17 years) for this study. On these images, CNN-S achieved an AUROC of 0.861 (95% CI 0.830-0.889; P Conclusions CNNs trained on standardized images had improved performance and therefore greater generalizability in skin cancer classification when applied to an unseen data set. This is an important consideration for future algorithm development, regulation, and approval. Further, when tested on these unseen test images, the teledermatologists clinically outperformed all the CNN models; however, the difference was deemed to be statistically insignificant when compared to CNN-S. Conflicts of Interest VM received speakers fees from Merck, Eli Lily, Novartis and Bristol Myers Squibb. VM is the principal investigator for a clinical trial funded by the Victorian Department of Health and Human Services with 1:1 contribution from MoleMap.