Search

This manuscript is the third in a five-part series related to statistical assessment methodology for technical performance of multi-parametric quantitative imaging biomarkers (mp-QIBs). We outline approaches and statistical methodologies for developing and evaluating a phenotype classification model from a set of multiparametric QIBs. We then describe validation studies of the classifier for precision, diagnostic accuracy, and interchangeability with a comparator classifier. We follow with an end-to-end real-world example of development and validation of a classifier for atherosclerotic plaque phenotypes. We consider diagnostic accuracy and interchangeability to be clinically meaningful claims for a phenotype classification model informed by mp-QIB inputs, aiming to provide tools to demonstrate agreement between imaging-derived characteristics and clinically established phenotypes. Understanding that we are working in an evolving field, we close our manuscript with an acknowledgement of existing challenges and a discussion of where additional work is needed. In particular, we discuss the challenges involved with technical performance and analytical validation of mp-QIBs. We intend for this manuscript to further advance the robust and promising science of multiparametric biomarker development.

This paper is the fifth in a five-part series on statistical methodology for performance assessment of multi-parametric quantitative imaging biomarkers (mpQIBs) for radiomic analysis. Radiomics is the process of extracting visually imperceptible features from radiographic medical images using data-driven algorithms. We refer to the radiomic features as data-driven imaging markers (DIMs), which are quantitative measures discovered under a data-driven framework from images beyond visual recognition but evident as patterns of disease processes irrespective of whether or not ground truth exists for the true value of the DIM. This paper aims to set guidelines on how to build machine learning models using DIMs in radiomics and to apply and report them appropriately. We provide a list of recommendations, named RANDAM (an abbreviation of "Radiomic ANalysis and DAta Modeling"), for analysis, modeling, and reporting in a radiomic study to make machine learning analyses in radiomics more reproducible. RANDAM contains five main components to use in reporting radiomics studies: design, data preparation, data analysis and modeling, reporting, and material availability. Real case studies in lung cancer research are presented along with simulation studies to compare different feature selection methods and several validation strategies.

Rationale and Objectives A critical performance metric for any quantitative imaging biomarker is its ability to reliably generate similar values on repeat testing. This is known as the repeatability of the biomarker, and it is used to determine the minimum detectable change needed in order to show that a change over time is real change and not just due to measurement error. Test-retest studies are the classic approach for estimating repeatability; however, these studies can be infeasible when the imaging is expensive, time-consuming, invasive, or requires contrast agents. The objective of this study was to develop and test a method for estimating repeatability without a test-retest study. Materials and Methods We present a statistical method for estimating repeatability and testing whether an imaging method meets a specified criterion for repeatability in the absence of a test-retest study. The new method is applicable for the particular situation where a reference standard is available. A Monte Carlo simulation study was conducted to evaluate the performance of the new method. Results The proposed estimator is unbiased, and hypothesis tests with the new estimator have nominal type I error rate and power similar to a test-retest study. We considered the situation where the reference standard provides the true value, as well as when the reference standard itself has various magnitudes of measurement error. An example from CT imaging biomarkers of atherosclerosis illustrates the new method. Conclusion Precision of a QIB can be measured without a test-retest study in the situation where a reference standard is available.

Multiparametric quantitative imaging biomarkers (QIBs) offer distinct advantages over single, univariate descriptors because they provide a more complete measure of complex, multidimensional biological systems. In disease, where structural and functional disturbances occur across a multitude of subsystems, multivariate QIBs are needed to measure the extent of system malfunction. This paper, the first Use Case in a series of articles on multiparameter imaging biomarkers, considers multiple QIBs as a multidimensional vector to represent all relevant disease constructs more completely. The approach proposed offers several advantages over QIBs as multiple endpoints and avoids combining them into a single composite that obscures the medical meaning of the individual measurements. We focus on establishing statistically rigorous methods to create a single, simultaneous measure from multiple QIBs that preserves the sensitivity of each univariate QIB while incorporating the correlation among QIBs. Details are provided for metrological methods to quantify the technical performance. Methods to reduce the set of QIBs, test the superiority of the mp-QIB model to any univariate QIB model, and design study strategies for generating precision and validity claims are also provided. QIBs of Alzheimer's Disease from the ADNI merge data set are used as a case study to illustrate the methods described.

RATIONALE AND OBJECTIVES To evaluate a new approach to establish compliance of segmentation tools with the computed tomography volumetry profile of the Quantitative Imaging Biomarker Alliance (QIBA); and determine the statistical exchangeability between real and simulated lesions through an international challenge. MATERIALS AND METHODS The study used an anthropomorphic phantom with 16 embedded physical lesions and 30 patient cases from the Reference Image Database to Evaluate Therapy Response with pathologically confirmed malignancies. Hybrid datasets were generated by virtually inserting simulated lesions corresponding to physical lesions into the phantom datasets using one projection-domain-based method (Method 1), two image-domain insertion methods (Methods 2 and 3), and simulated lesions corresponding to real lesions into the Reference Image Database to Evaluate Therapy Response dataset (using Method 2). The volumes of the real and simulated lesions were compared based on bias (measured mean volume differences between physical and virtually inserted lesions in phantoms as quantified by segmentation algorithms), repeatability, reproducibility, equivalence (phantom phase), and overall QIBA compliance (phantom and clinical phase). RESULTS For phantom phase, three of eight groups were fully QIBA compliant, and one was marginally compliant. For compliant groups, the estimated biases were -1.8 ± 1.4%, -2.5 ± 1.1%, -3 ± 1%, -1.8 ± 1.5% (±95% confidence interval). No virtual insertion method showed statistical equivalence to physical insertion in bias equivalence testing using Schuirmann's two one-sided test (±5% equivalence margin). Differences in repeatability and reproducibility across physical and simulated lesions were largely comparable (0.1%-16% and 7%-18% differences, respectively). For clinical phase, 7 of 16 groups were QIBA compliant. CONCLUSION Hybrid datasets yielded conclusions similar to real computed tomography datasets where phantom QIBA compliant was also compliant for hybrid datasets. Some groups deemed compliant for simulated methods, not for physical lesion measurements. The magnitude of this difference was small (

Rationale and Objectives This study aimed to review the current understanding and capabilities regarding use of imaging for noninvasive lesion characterization and its relationship to lung cancer screening and treatment. Materials and Methods Our review of the state of the art was broken down into questions about the different lung cancer image phenotypes being characterized, the role of imaging and requirements for increasing its value with respect to increasing diagnostic confidence and quantitative assessment, and a review of the current capabilities with respect to those needs. Results The preponderance of the literature has so far been focused on the measurement of lesion size, with increasing contributions being made to determine the formal performance of scanners, measurement tools, and human operators in terms of bias and variability. Concurrently, an increasing number of investigators are reporting utility and predictive value of measures other than size, and sensitivity and specificity is being reported. Relatively little has been documented on quantitative measurement of non-size features with corresponding estimation of measurement performance and reproducibility. Conclusions The weight of the evidence suggests characterization of pulmonary lesions built on quantitative measures adds value to the screening for, and treatment of, lung cancer. Advanced image analysis techniques may identify patterns or biomarkers not readily assessed by eye and may also facilitate management of multidimensional imaging data in such a way as to efficiently integrate it into the clinical workflow.

Rationale and Objectives New ways to understand biology as well as increasing interest in personalized treatments requires new capabilities for the assessment of therapy response. The lack of consensus methods and qualification evidence needed for large-scale multicenter trials, and in turn the standardization that allows them, are widely acknowledged to be the limiting factor in the deployment of qualified imaging biomarkers. Materials and Methods The Quantitative Imaging Biomarker Alliance is organized to establish a methodology whereby multiple stakeholders collaborate. It has charged the Volumetric Computed Tomography (CT) Technical Subcommittee with investigating the technical feasibility and clinical value of quantifying changes over time in either volume or other parameters as biomarkers. The group selected solid tumors of the chest in subjects with lung cancer as its first case in point. Success is defined as sufficiently rigorous improvements in CT-based outcome measures to allow individual patients in clinical settings to switch treatments sooner if they are no longer responding to their current regimens, and reduce the costs of evaluating investigational new drugs to treat lung cancer. Results The team has completed a systems engineering analysis, has begun a roadmap of experimental groundwork, documented profile claims and protocols, and documented a process for imaging biomarker qualification as a general paradigm for qualifying other imaging biomarkers as well. Conclusion This report addresses a procedural template for the qualification of quantitative imaging biomarkers. This mechanism is cost-effective for stakeholders while simultaneously advancing the public health by promoting the use of measures that prove effective.

This study aimed to review the current understanding and capabilities regarding use of imaging for noninvasive lesion characterization and its relationship to lung cancer screening and treatment.Our review of the state of the art was broken down into questions about the different lung cancer image phenotypes being characterized, the role of imaging and requirements for increasing its value with respect to increasing diagnostic confidence and quantitative assessment, and a review of the current capabilities with respect to those needs.The preponderance of the literature has so far been focused on the measurement of lesion size, with increasing contributions being made to determine the formal performance of scanners, measurement tools, and human operators in terms of bias and variability. Concurrently, an increasing number of investigators are reporting utility and predictive value of measures other than size, and sensitivity and specificity is being reported. Relatively little has been documented on quantitative measurement of non-size features with corresponding estimation of measurement performance and reproducibility.The weight of the evidence suggests characterization of pulmonary lesions built on quantitative measures adds value to the screening for, and treatment of, lung cancer. Advanced image analysis techniques may identify patterns or biomarkers not readily assessed by eye and may also facilitate management of multidimensional imaging data in such a way as to efficiently integrate it into the clinical workflow.

Rationale and Objectives To estimate and statistically compare the bias and variance of radiologists measuring the size of spherical and complex synthetic nodules. Materials and Methods This study did not require the institutional review board approval. Six radiologists estimated the size of 10 synthetic nodules embedded within an anthropomorphic thorax phantom from computed tomography scans at 0.8- and 5-mm slice thicknesses. The readers measured the nodule size using unidimensional (1D) longest in-slice dimension, bidimensional (2D) area from longest in-slice and longest perpendicular dimension, and three-dimensional (3D) semiautomated volume. Intercomparisons of bias (difference between average and true size) and variance among methods were performed after converting the 2D and 3D estimates to a compatible 1D scale. Results The relative biases of radiologists with the 3D tool were −1.8%, −0.4%, −0.7%, −0.4%, and −1.6% for 10-mm spherical, 20-mm spherical, 20-mm elliptical, 10-mm lobulated, and 10-mm spiculated nodules compared to 1.4%, −0.1%, −26.5%, −7.8%, and −39.8% for 1D. The three-dimensional measurements were significantly less biased than 1D for elliptical, lobulated, and spiculated nodules. The relative standard deviations for 3D were 7.5%, 3.9%, 3.6%, 9.7%, and 8.3% compared to 5.7%, 2.6%, 20.3%, 5.3%, and 16.4% for 1D. Unidimensional sizing was significantly less variable than 3D for the lobulated nodule and significantly more variable for the ellipsoid and spiculated nodules. Three-dimensional bias and variability were smaller for thin 0.8-mm slice data compared to thick 5.0-mm data. Conclusions The study shows that radiologist-controlled 3D volumetric lesion sizing can not only achieve smaller bias but also achieve similar or smaller variability compared to 1D sizing, especially for complex lesion shapes.

New ways to understand biology as well as increasing interest in personalized treatments requires new capabilities for the assessment of therapy response. The lack of consensus methods and qualification evidence needed for large-scale multicenter trials, and in turn the standardization that allows them, are widely acknowledged to be the limiting factor in the deployment of qualified imaging biomarkers.The Quantitative Imaging Biomarker Alliance is organized to establish a methodology whereby multiple stakeholders collaborate. It has charged the Volumetric Computed Tomography (CT) Technical Subcommittee with investigating the technical feasibility and clinical value of quantifying changes over time in either volume or other parameters as biomarkers. The group selected solid tumors of the chest in subjects with lung cancer as its first case in point. Success is defined as sufficiently rigorous improvements in CT-based outcome measures to allow individual patients in clinical settings to switch treatments sooner if they are no longer responding to their current regimens, and reduce the costs of evaluating investigational new drugs to treat lung cancer.The team has completed a systems engineering analysis, has begun a roadmap of experimental groundwork, documented profile claims and protocols, and documented a process for imaging biomarker qualification as a general paradigm for qualifying other imaging biomarkers as well.This report addresses a procedural template for the qualification of quantitative imaging biomarkers. This mechanism is cost-effective for stakeholders while simultaneously advancing the public health by promoting the use of measures that prove effective.