Search

International audience

Purpose The aim of this paper is to define the requirements and describe the design and implementation of a standard benchmark tool for evaluation and validation of PET-auto-segmentation (PET-AS) algorithms. This work follows the recommendations of Task Group 211 (TG211) appointed by the American Association of Physicists in Medicine (AAPM). Methods The recommendations published in the AAPM TG211 report were used to derive a set of required features and to guide the design and structure of a benchmarking software tool. These items included the selection of appropriate representative data and reference contours obtained from established approaches and the description of available metrics. The benchmark was designed in a way that it could be extendable and open to the implementation of bespoke segmentation methods, while maintaining its main purpose of being a standard testing platform for newly developed PET-AS methods. An example of implementation of the proposed framework, named PETASset, was built. In this work, a selection of PET-AS methods representing common approaches to PET image segmentation were evaluated within PETASset for the purpose of testing and demonstrating the capabilities of the software as a benchmark platform. Results A selection of clinical, physical and simulated phantom data, including “best estimates” reference contours from macroscopic specimens, simulation template, and CT scans was built into the PETASset application database. Specific metrics such as Dice Similarity Coefficient (DSC), Positive Predictive Value (PPV), and Sensitivity (S), were included to allow the user to compare the results of any given PET-AS algorithm to the reference contours. In addition, a tool to generate structured reports on the evaluation of the performance of PET-AS algorithms against the reference contours was built. The variation of the metric agreement values with the reference contours across the PET-AS methods evaluated for demonstration were between 0.51 and 0.83, 0.44 and 0.86, and 0.61 and 1.00 for DSC, PPV, and the S metric, respectively. Examples of agreement limits were provided to show how the software could be used to evaluate a new algorithm against the existing state-of-the art. Conclusions PETASset provides a platform that allows standardizing the evaluation and comparison of different PET-AS methods on a wide range of PET datasets. The developed platform will be available to users willing to evaluate their PET-AS methods and contribute more evaluation datasets.

Purpose: Radiomics is the high-throughput analysis of medical images for treatment individualisation. It conventionally relies on the quantification of different characteristics of a region of interest (ROI) delineated in the image, such as the mean intensity, volume and textural heterogeneity. The lack of standardisation of image features is one of the major limitations for reproducing and validating radiomic studies, and thus a major hurdle for further developments in the field and for clinical translation. To overcome this challenge, a large international collaboration of 19 teams from 8 countries was initiated to establish an image feature ontology, and to provide definitions of commonly used features, benchmarks for testing feature extraction and image processing software, and reporting guidelines. Methods: The initiative consisted of two phases. In phase 1, 351 commonly used features were specified and benchmarked against a simple digital phantom, without any requirement for image pre-processing steps. The feature set consisted of commonly used radiomic features and encompasses statistical, morphological and texture characteristics of the ROI, both slice-by-slice (2D) and as a volume (3D). In phase 2, image pre-processing steps were introduced, and features were benchmarked by evaluating five pre-processing configurations on a lung cancer patient CT image. The configurations differ in treatment of the image stack (2D: A-B; 3D: C-E), the interpolation method (none: A; bi/trilinear: B-D, tricubic: E) and the grey-level discretisation method (fixed bin size: A, C; fixed number of bins: B, D-E). Both phases were iterative, and participants had the opportunity to compare results and update their workflow implementation. We set the most frequently contributed value of each feature as its benchmark value, and subsequently determined its reliability based on the number of contributing groups and the consensus level. Results: 19 different software implementations were tested. In both phases, only a small number of features were found to be reliable initially. The number of reliable features increased over time as problems were identified and resolved, see Figure 1 and Table 1. Remaining features for which no agreement was reached were not commonly implemented (< 3 agreeing teams), and could therefore not be reliably assessed. Conclusion: We addressed the lack of standardised feature definitions, implementation and image pre-processing steps for radiomics by providing reliable benchmark values for commonly used features. During the initiative, the 19 teams demonstrated large initial differences, yet nevertheless managed to converge to common reference values by increasing adherence to standardised definitions. Therefore, the use of our standardised definitions and benchmarks to test and update radiomics software is imperative to increase reproducibility of future radiomics studies.