Search

Background: To address the numerous unmeet clinical needs, in recent years several Machine Learning models applied to medical images and clinical data have been introduced and developed. Even when they achieve encouraging results, they lack evolutionary progression, thus perpetuating their status as autonomous entities. We postulated that different algorithms which have been proposed in the literature to address the same diagnostic task, can be aggregated to enhance classification performance. We suggested a proof of concept to define an ensemble approach useful for integrating different algorithms proposed to solve the same clinical task. Methods: The proposed approach was developed starting from a public database consisting of radiomic features extracted from CT images relating to 535 patients suffering from lung cancer. Seven algorithms were trained independently by participants in the AI4MP working group on Artificial Intelligence of the Italian Association of Physics in Medicine to discriminate metastatic from non-metastatic patients. The classification scores generated by these algorithms are used to train SVM classifier. The Explainable Artificial Intelligence approach is applied to the final model. The ensemble model was validated following an 80–20 hold-out and leave-one-out scheme on the training set. Results: Compared to individual algorithms, a more accurate result was achieved. On the independent test the ensemble model achieved an accuracy of 0.78, a F1-score of 0.57 and a log-loss of 0.49. Shapley values representing the contribution of each algorithm to the final classification result of the ensemble model were calculated. This information represents an added value for the end user useful for evaluating the appropriateness of the classification result on a particular case. It also allows us to evaluate on a global level which methodological approaches of the individual algorithms are likely to have the most impact. Conclusion: Our proposal represents an innovative approach useful for integrating different algorithms that populate the literature and which lays the foundations for future evaluations in broader application scenarios. [ABSTRACT FROM AUTHOR]

[Display omitted] • AI4MP-Challenge is the first AIFM multicentric experience on machine learning. • The main objective is to improve knowledge and skills of medical physicists on machine learning. • Encountered pitfalls: violation of independence assumption, computation errors, data imbalance. • Providing both cross-validation and an independent test helps to detect implementation issue. • The exclusion of non-robust features does not allow to significantly increase model stability. A novel and unconventional approach to a machine learning challenge was designed to spread knowledge, identify robust methods and highlight potential pitfalls about machine learning within the Medical Physics community. A public dataset comprising 41 radiomic features and 535 patients was employed to assess the potential of radiomics in distinguishing between primary lung tumors and metastases. Each participant developed two classification models using: (i) all features (base model); (ii) only robust features (robust model). Both models were validated with cross-validation and on unseen data. The population stability index (PSI) was used as diagnostic metric for implementation issues. Performance was compared to reference. Base and robust models were compared in terms of performance and stability (coefficient of variation (CoV) of prediction probabilities). PSI detected potential implementation errors in 70 % of models. The dataset exhibited strong imbalance. The average Gmean (i.e. an appropriate metric for imbalance) among all participants was 0.67 ± 0.01, significantly higher than reference Gmean of 0.50 ± 0.04. Robust models performances were slightly worse than base models (p < 0.05). Regarding stability, robust models exhibited lower median CoV on training set only. AI4MP-Challenge models overperformed the reference, significantly improving the Gmean. Exclusion of less-robust features did not improve model robustness and it should be avoided when confounding effects are absent. Other methods, like harmonization or data augmentation, should be evaluated. This study demonstrated how the collaborative effort to foster knowledge on machine learning among medical physicists, through interactive sessions and exchange of information among participants, can result in improved models. [ABSTRACT FROM AUTHOR]