Search

Rationale and Objectives: Evidence is building in support of the clinical utility of atherosclerotic plaque imaging by computed tomography angiography (CTA). There is increasing organized activity to embrace non-calcified plaque (NCP) as a formally defined biomarker for clinical trials, and high-risk plaque (HRP) for clinical care, as the most relevant measures for the field to advance and worthy of community efforts to validate. Yet the ability to assess the quantitative performance of any given specific solution to make these measurements or classifications is not available. Vendors use differing definitions, assessment metrics, and validation data sets to describe their offerings without clinician users having the capability to make objective assessments of accuracy and precision and how this affects diagnostic confidence., Materials and Methods: The QIBA Profile for Atherosclerosis Biomarkers by CTA was created by the Quantitative Imaging Biomarkers Alliance (QIBA) to improve objectivity and decrease the variability of noninvasive plaque phenotyping. The Profile provides claims on the accuracy and precision of plaque measures individually and when combined., Results: Individual plaque morphology measurements are evaluated in terms of bias (accuracy), slope (consistency of the bias across the measurement range, needed for measurements of change), and variability. The multiparametric plaque stability phenotype is evaluated in terms of agreement with expert pathologists. The Profile is intended for a broad audience, including those engaged in discovery science, clinical trials, and patient care., Conclusion: This report provides a rationale and overview of the Profile claims and how to comply with the Profile in research and clinical practice., Summary Statement: This article summarizes objective means to validate the analytical performance of non-calcified plaque (NCP), other emerging plaque morphology measurements, and multiparametric histology-defined high-risk plaque (HRP), as outlined in the QIBA Profile for Atherosclerosis Biomarkers by CTA., Competing Interests: Declaration of Competing Interest The nature of the work is a standards-based activity undertaken by a cross-section of academic, practicing, and industry professionals. The RSNA maintains a comprehensive vetting and representation of conflict statements, but in short, no conflicts are noted that undermine the integrity and openness of the work. The first author Andrew J. Buckler is a founder and shareholder of Elucid Bioimaging Inc., (Copyright © 2024 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Multiparametric quantitative imaging biomarkers (QIBs) offer distinct advantages over single, univariate descriptors because they provide a more complete measure of complex, multidimensional biological systems. In disease, where structural and functional disturbances occur across a multitude of subsystems, multivariate QIBs are needed to measure the extent of system malfunction. This paper, the first Use Case in a series of articles on multiparameter imaging biomarkers, considers multiple QIBs as a multidimensional vector to represent all relevant disease constructs more completely. The approach proposed offers several advantages over QIBs as multiple endpoints and avoids combining them into a single composite that obscures the medical meaning of the individual measurements. We focus on establishing statistically rigorous methods to create a single, simultaneous measure from multiple QIBs that preserves the sensitivity of each univariate QIB while incorporating the correlation among QIBs. Details are provided for metrological methods to quantify the technical performance. Methods to reduce the set of QIBs, test the superiority of the mp-QIB model to any univariate QIB model, and design study strategies for generating precision and validity claims are also provided. QIBs of Alzheimer's Disease from the ADNI merge data set are used as a case study to illustrate the methods described., (Copyright © 2022 The Association of University Radiologists. All rights reserved.)

Combinations of multiple quantitative imaging biomarkers (QIBs) are often able to predict the likelihood of an event of interest such as death or disease recurrence more effectively than single imaging measurements can alone. The development of such multiparametric quantitative imaging and evaluation of its fitness of use differs from the analogous processes for individual QIBs in several key aspects. A computational procedure to combine the QIB values into a model output must be specified. The output must also be reproducible and be shown to have reasonably strong ability to predict the risk of an event of interest. Attention must be paid to statistical issues not often encountered in the single QIB scenario, including overfitting and bias in the estimates of model performance. This is the fourth in a five-part series on statistical methodology for assessing the technical performance of multiparametric quantitative imaging. Considerations for data acquisition are discussed and recommendations from the literature on methodology to construct and evaluate QIB-based models for risk prediction are summarized. The findings in the literature upon which these recommendations are based are demonstrated through simulation studies. The concepts in this manuscript are applied to a real-life example involving prediction of major adverse cardiac events using automated plaque analysis., (Published by Elsevier Inc.)

This manuscript is the third in a five-part series related to statistical assessment methodology for technical performance of multi-parametric quantitative imaging biomarkers (mp-QIBs). We outline approaches and statistical methodologies for developing and evaluating a phenotype classification model from a set of multiparametric QIBs. We then describe validation studies of the classifier for precision, diagnostic accuracy, and interchangeability with a comparator classifier. We follow with an end-to-end real-world example of development and validation of a classifier for atherosclerotic plaque phenotypes. We consider diagnostic accuracy and interchangeability to be clinically meaningful claims for a phenotype classification model informed by mp-QIB inputs, aiming to provide tools to demonstrate agreement between imaging-derived characteristics and clinically established phenotypes. Understanding that we are working in an evolving field, we close our manuscript with an acknowledgement of existing challenges and a discussion of where additional work is needed. In particular, we discuss the challenges involved with technical performance and analytical validation of mp-QIBs. We intend for this manuscript to further advance the robust and promising science of multiparametric biomarker development., (Published by Elsevier Inc.)

This paper is the fifth in a five-part series on statistical methodology for performance assessment of multi-parametric quantitative imaging biomarkers (mpQIBs) for radiomic analysis. Radiomics is the process of extracting visually imperceptible features from radiographic medical images using data-driven algorithms. We refer to the radiomic features as data-driven imaging markers (DIMs), which are quantitative measures discovered under a data-driven framework from images beyond visual recognition but evident as patterns of disease processes irrespective of whether or not ground truth exists for the true value of the DIM. This paper aims to set guidelines on how to build machine learning models using DIMs in radiomics and to apply and report them appropriately. We provide a list of recommendations, named RANDAM (an abbreviation of "Radiomic ANalysis and DAta Modeling"), for analysis, modeling, and reporting in a radiomic study to make machine learning analyses in radiomics more reproducible. RANDAM contains five main components to use in reporting radiomics studies: design, data preparation, data analysis and modeling, reporting, and material availability. Real case studies in lung cancer research are presented along with simulation studies to compare different feature selection methods and several validation strategies., (Copyright © 2022 The Association of University Radiologists. All rights reserved.)

Multiparameter quantitative imaging incorporates anatomical, functional, and/or behavioral biomarkers to characterize tissue, detect disease, identify phenotypes, define longitudinal change, or predict outcome. Multiple imaging parameters are sometimes considered separately but ideally are evaluated collectively. Often, they are transformed as Likert interpretations, ignoring the correlations of quantitative properties that may result in better reproducibility or outcome prediction. In this paper we present three use cases of multiparameter quantitative imaging: i) multidimensional descriptor, ii) phenotype classification, and iii) risk prediction. A fourth application based on data-driven markers from radiomics is also presented. We describe the technical performance characteristics and their metrics common to all use cases, and provide a structure for the development, estimation, and testing of multiparameter quantitative imaging. This paper serves as an overview for a series of individual articles on the four applications, providing the statistical framework for multiparameter imaging applications in medicine., (Copyright © 2022 The Association of University Radiologists. All rights reserved.)

Rationale and Objectives: The purpose of this study was to test for superiority of wide-angle digital breast tomosynthesis plus synthetic mammography (Insight 2D) in comparison to full-field digital mammography (FFDM)., Materials and Methods: In this study, twenty readers interpreted 350 screening and diagnostic cases of wide-angle digital breast tomosynthesis (DBT) plus Insight 2D and FFDM in two separate reading sessions separated by at least a 6-week washout period. Breast-level estimates of the area under the curve and sensitivity along with subject-level recall rate were measured and compared between wide-angle DBT plus Insight 2D and FFDM. The same measures were also assessed for dense breasts. A hierarchical analysis plan was used to control the study's type I error rate at 0.05., Results: The mean breast-level area under the curve for distinguishing breasts with cancer from non-cancer breasts was 0.893 with DBT plus Insight 2D versus 0.837 with FFDM, showing superiority of DBT plus Insight 2D (p < 0.001). Breast-level sensitivity was significantly superior for DBT plus Insight 2D in comparison to FFDM (0.852 vs. 0.805, p = 0.043). Subject-level recall rate for DBT plus Insight 2D was significantly lower in comparison to FFDM (0.344 vs. 0.473, p < 0.001). For dense breasts, the readers' accuracy with DBT plus Insight 2D was superior to their accuracy with FFDM (0.875 vs. 0.830, p = 0.026), and their recall rate was significantly lower for DBT plus Insight 2D in comparison to FFDM (0.338 vs. 0.441, p = 0.003)., Conclusion: Reader performance with wide-angle DBT plus Insight 2D is superior to that with FFDM, showing significantly higher breast-level accuracy and sensitivity and significantly lower recall rates., (Copyright © 2022 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: In diagnostic accuracy studies, cases in which a reader does not see the condition of interest are often given the same score for ROC analysis (e.g. confidence score of 0%). However, many of these cases can be further distinguished and doing so may result in more robust ROC results., Materials and Methods: We examined two recent, real-world studies which used different procedures to encourage readers to further distinguish subjects who appear to be without the condition of interest. For each study, we calculated the results under two conditions. In the "zeros distinguished" (ZD) condition, we incorporated the confidence scores collected to further distinguish the normal-looking subjects. In the "zeros not distinguished" (ZND) condition, we disregarded these scores and simply gave the unit of analysis a score of zero whenever the reader did not identify the condition of interest in that unit. We compared the two conditions on (1) coverage of the ROC space and (2) discrepancy between parametric and nonparametric results., Results: Compared to the ZND condition, coverage of the ROC space was improved in the ZD condition for all ROC curves in both studies. In the first study, there was a significant reduction in the discrepancy between parametric and nonparametric results (median discrepancy in ZND vs ZD condition: 0.033 vs 0.011, p = 0.012). A similar reduction was not seen in the second study, though the discrepancies were very low in both conditions (0.003 vs 0.006, p = 0.313)., Conclusion: Prompting readers to further distinguish cases in which they do not see the condition of interest may result in more robust ROC results, though further exploration of this topic is warranted., (Copyright © 2021 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: A critical performance metric for any quantitative imaging biomarker is its ability to reliably generate similar values on repeat testing. This is known as the repeatability of the biomarker, and it is used to determine the minimum detectable change needed in order to show that a change over time is real change and not just due to measurement error. Test-retest studies are the classic approach for estimating repeatability; however, these studies can be infeasible when the imaging is expensive, time-consuming, invasive, or requires contrast agents. The objective of this study was to develop and test a method for estimating repeatability without a test-retest study., Materials and Methods: We present a statistical method for estimating repeatability and testing whether an imaging method meets a specified criterion for repeatability in the absence of a test-retest study. The new method is applicable for the particular situation where a reference standard is available. A Monte Carlo simulation study was conducted to evaluate the performance of the new method., Results: The proposed estimator is unbiased, and hypothesis tests with the new estimator have nominal type I error rate and power similar to a test-retest study. We considered the situation where the reference standard provides the true value, as well as when the reference standard itself has various magnitudes of measurement error. An example from CT imaging biomarkers of atherosclerosis illustrates the new method., Conclusion: Precision of a QIB can be measured without a test-retest study in the situation where a reference standard is available., (Copyright © 2021 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: To evaluate a new approach to establish compliance of segmentation tools with the computed tomography volumetry profile of the Quantitative Imaging Biomarker Alliance (QIBA); and determine the statistical exchangeability between real and simulated lesions through an international challenge., Materials and Methods: The study used an anthropomorphic phantom with 16 embedded physical lesions and 30 patient cases from the Reference Image Database to Evaluate Therapy Response with pathologically confirmed malignancies. Hybrid datasets were generated by virtually inserting simulated lesions corresponding to physical lesions into the phantom datasets using one projection-domain-based method (Method 1), two image-domain insertion methods (Methods 2 and 3), and simulated lesions corresponding to real lesions into the Reference Image Database to Evaluate Therapy Response dataset (using Method 2). The volumes of the real and simulated lesions were compared based on bias (measured mean volume differences between physical and virtually inserted lesions in phantoms as quantified by segmentation algorithms), repeatability, reproducibility, equivalence (phantom phase), and overall QIBA compliance (phantom and clinical phase)., Results: For phantom phase, three of eight groups were fully QIBA compliant, and one was marginally compliant. For compliant groups, the estimated biases were -1.8 ± 1.4%, -2.5 ± 1.1%, -3 ± 1%, -1.8 ± 1.5% (±95% confidence interval). No virtual insertion method showed statistical equivalence to physical insertion in bias equivalence testing using Schuirmann's two one-sided test (±5% equivalence margin). Differences in repeatability and reproducibility across physical and simulated lesions were largely comparable (0.1%-16% and 7%-18% differences, respectively). For clinical phase, 7 of 16 groups were QIBA compliant., Conclusion: Hybrid datasets yielded conclusions similar to real computed tomography datasets where phantom QIBA compliant was also compliant for hybrid datasets. Some groups deemed compliant for simulated methods, not for physical lesion measurements. The magnitude of this difference was small (<5.4%). While technical performance is not equivalent, they correlate, such that, volumetrically simulated lesions could potentially serve as practical proxies., (Copyright © 2018 The Association of University Radiologists. All rights reserved.)

Rationale and Objectives: Quantitative imaging biomarkers (QIBs) are becoming increasingly adopted into clinical practice to monitor changes in patients' conditions. The repeatability coefficient (RC) is the clinical cut-point used to discern between changes in a biomarker's measurements due to measurement error and changes that exceed measurement error, thus indicating real change in the patient. Imaging biomarkers have characteristics that make them difficult for estimating the repeatability coefficient, including nonconstant error, non-Gaussian distributions, and measurement error that must be estimated from small studies., Methods: We conducted a Monte Carlo simulation study to investigate how well three statistical methods for estimating the repeatability coefficient perform under five settings common for QIBs., Results: When the measurement error is constant and replicates are normally distributed, all of the statistical methods perform well. When the measurement error is proportional to the true value, approaches that use the log transformation or coefficient of variation perform similarly. For other common settings, none of the methods for estimating the repeatability coefficient perform adequately., Conclusion: Many of the common approaches to estimating the repeatability coefficient perform well for only limited scenarios. The optimal approach depends strongly on the pattern of the within-subject variability; thus, a precision profile is critical in evaluating the technical performance of QIBs. Asymmetric bounds for detecting regression vs progression can be implemented and should be used when clinically appropriate., (Copyright © 2018 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: The purpose of this study was to characterize analytic performance of software-aided arterial vessel structure measurements across a range of scanner settings for computed tomography angiography where ground truth is known. We characterized performance for measurands that may be efficiently measured for clinical cases without use of software, as well as those that may be done manually but which is generally not done due to the effort level required unless software is employed., Materials and Methods: Four measurands (lumen area, stenosis, wall area, wall thickness) were evaluated using tissue-mimicking phantoms to estimate bias, heteroscedasticity, and limits of quantitation both pooled across scanner settings and individually for eight different settings. Reproducibility across scanner settings was also estimated., Results: Measurements of lumen area have a near constant bias of +1.3 mm for measurements ranging from 3 mm 2 to 40 mm 2 ; stenosis bias is +7% across a 30%-70% range; wall area bias is +14% across a 50-450 mm 2 range; and wall thickness bias is +1.2 mm across a 3-9 mm range. All measurements possess properties that make them suitable for measuring longitudinal change. Lumen area demonstrates the most sensitivity to scanner settings (bias from as low as +.1 mm to as high as +2.7 mm); wall thickness demonstrates negligible sensitivity., Conclusions: Variability across scanner settings for lumen measurands was generally higher than bias for a given setting. The converse was true for the wall measurands, where variability due to scanner settings was very low. Both bias and variability due to scanner settings of vessel structure were within clinically useful levels., (Copyright © 2017 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rational and Objectives: This study aims to evaluate the diagnostic utility of breast imaging using transmission ultrasound. We present readers' accuracy in determining whether a breast lesion is a cyst versus a solid using transmission ultrasound as an adjunct to mammography., Materials and Methods: This retrospective multi-reader, multi-case receiver operating characteristic study included 37 lesions seen on mammography and transmission ultrasound. Cyst cases were confirmed as cysts using their appearance on handheld ultrasound. Solid cases were confirmed as solids with pathology results. Fourteen readers performed blinded, randomized reads with mammography + quantitative transmission scan images, assigning both a confidence score (0-100) and a binary classification of cyst or solid. A 95% percentile bootstrap confidence interval (CI) was computed for the readers' mean receiver operating characteristic area, sensitivity, and specificity., Results: Using the readers' binary classification of cyst or solid lesions, the mean sensitivity and specificity were 0.933 [95% CI: 0.837, 0.995] and 0.858 [95% CI: 0.701, 0.985], respectively. When the readers' confidence scores were used to distinguish a cyst versus solid, the mean receiver operating characteristic area was 0.920 [95% CI: 0.827, 0.985]., Conclusions: Transmission ultrasound can provide an accurate assessment of a cyst versus a solid lesion in the breast. Prospective clinical trials will further delineate the role of transmission ultrasound as an adjunct to mammography to increase specificity in breast evaluation., (Copyright © 2017 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: Quantifying changes in lung tumor volume is important for diagnosis, therapy planning, and evaluation of response to therapy. The aim of this study was to assess the performance of multiple algorithms on a reference data set. The study was organized by the Quantitative Imaging Biomarker Alliance (QIBA)., Materials and Methods: The study was organized as a public challenge. Computed tomography scans of synthetic lung tumors in an anthropomorphic phantom were acquired by the Food and Drug Administration. Tumors varied in size, shape, and radiodensity. Participants applied their own semi-automated volume estimation algorithms that either did not allow or allowed post-segmentation correction (type 1 or 2, respectively). Statistical analysis of accuracy (percent bias) and precision (repeatability and reproducibility) was conducted across algorithms, as well as across nodule characteristics, slice thickness, and algorithm type., Results: Eighty-four percent of volume measurements of QIBA-compliant tumors were within 15% of the true volume, ranging from 66% to 93% across algorithms, compared to 61% of volume measurements for all tumors (ranging from 37% to 84%). Algorithm type did not affect bias substantially; however, it was an important factor in measurement precision. Algorithm precision was notably better as tumor size increased, worse for irregularly shaped tumors, and on the average better for type 1 algorithms. Over all nodules meeting the QIBA Profile, precision, as measured by the repeatability coefficient, was 9.0% compared to 18.4% overall., Conclusion: The results achieved in this study, using a heterogeneous set of measurement algorithms, support QIBA quantitative performance claims in terms of volume measurement repeatability for nodules meeting the QIBA Profile criteria., (Copyright © 2016 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

A major initiative of the Quantitative Imaging Biomarker Alliance is to develop standards-based documents called "Profiles," which describe one or more technical performance claims for a given imaging modality. The term "actor" denotes any entity (device, software, or person) whose performance must meet certain specifications for the claim to be met. The objective of this paper is to present the statistical issues in testing actors' conformance with the specifications. In particular, we present the general rationale and interpretation of the claims, the minimum requirements for testing whether an actor achieves the performance requirements, the study designs used for testing conformity, and the statistical analysis plan. We use three examples to illustrate the process: apparent diffusion coefficient in solid tumors measured by MRI, change in Perc 15 as a biomarker for the progression of emphysema, and percent change in solid tumor volume by computed tomography as a biomarker for lung cancer progression., (Copyright © 2016 The Association of University Radiologists. All rights reserved.)

Rationale and Objectives: Tumor volume change has potential as a biomarker for diagnosis, therapy planning, and treatment response. Precision was evaluated and compared among semiautomated lung tumor volume measurement algorithms from clinical thoracic computed tomography data sets. The results inform approaches and testing requirements for establishing conformance with the Quantitative Imaging Biomarker Alliance (QIBA) Computed Tomography Volumetry Profile., Materials and Methods: Industry and academic groups participated in a challenge study. Intra-algorithm repeatability and inter-algorithm reproducibility were estimated. Relative magnitudes of various sources of variability were estimated using a linear mixed effects model. Segmentation boundaries were compared to provide a basis on which to optimize algorithm performance for developers., Results: Intra-algorithm repeatability ranged from 13% (best performing) to 100% (least performing), with most algorithms demonstrating improved repeatability as the tumor size increased. Inter-algorithm reproducibility was determined in three partitions and was found to be 58% for the four best performing groups, 70% for the set of groups meeting repeatability requirements, and 84% when all groups but the least performer were included. The best performing partition performed markedly better on tumors with equivalent diameters greater than 40 mm. Larger tumors benefitted by human editing but smaller tumors did not. One-fifth to one-half of the total variability came from sources independent of the algorithms. Segmentation boundaries differed substantially, not ony in overall volume but also in detail., Conclusions: Nine of the 12 participating algorithms pass precision requirements similar to what is indicated in the QIBA Profile, with the caveat that the present study was not designed to explicitly evaluate algorithm profile conformance. Change in tumor volume can be measured with confidence to within ±14% using any of these nine algorithms on tumor sizes greater than 10 mm. No partition of the algorithms was able to meet the QIBA requirements for interchangeability down to 10 mm, although the partition comprising best performing algorithms did meet this requirement for a tumor size of greater than approximately 40 mm., (Copyright © 2015 AUR. All rights reserved.)

Rationale and Objectives: New tests are typically assessed by estimating their technical and diagnostic performance through comparisons with a reference standard. A valid reference standard, however, is not always available and is not required for assessing the interchangeability of a new test with an existing one., Materials and Methods: To show interchangeability of a new test with an existing test, one compares the differences in diagnoses between the new and existing tests to differences between diagnoses made with the existing test on several occasions. We illustrate the test for interchangeability with two studies. In a transcatheter aortic valve replacement study, we test whether semiautomated analysis can be used interchangeably with manual reconstructions from three-dimensional computed tomography (CT) images. In patients with femoroacetabular impingement, we test whether magnetic resonance imaging (MRI) can replace CT to measure acetabular version., Results: Although the semiautomated method agreed often with the manual measurement of aortic valve size (87.6%), interchanging the semiautomated method with manual measurements by an expert would lead to a 1.7%-12.2% increase in the frequency of disagreement. Interchanging MRI for CT to measure acetabular version would lead to differences in angle measurements of 2.0° to 3.1° in excess of the differences we would expect to see with CT alone., Conclusions: Testing for agreement or correlation between a new and an existing test is not sufficient evidence of the performance of a new test. A formal evaluation of interchangeability can be conducted in the absence of a reference standard., (Copyright © 2014 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: To evaluate whether the presence of facial photographs obtained at the point-of-care of portable radiography leads to increased detection of wrong-patient errors., Materials and Methods: In this institutional review board-approved study, 166 radiograph-photograph combinations were obtained from 30 patients. Consecutive radiographs from the same patients resulted in 83 unique pairs (ie, a new radiograph and prior, comparison radiograph) for interpretation. To simulate wrong-patient errors, mismatched pairs were generated by pairing radiographs from different patients chosen randomly from the sample. Ninety radiologists each interpreted a unique randomly chosen set of 10 radiographic pairs, containing up to 10% mismatches (ie, error pairs). Radiologists were randomly assigned to interpret radiographs with or without photographs. The number of mismatches was identified, and interpretation times were recorded., Results: Ninety radiologists with 21 ± 10 (mean ± standard deviation) years of experience were recruited to participate in this observer study. With the introduction of photographs, the proportion of errors detected increased from 31% (9 of 29) to 77% (23 of 30; P = .006). The odds ratio for detection of error with photographs to detection without photographs was 7.3 (95% confidence interval: 2.29-23.18). Observer qualifications, training, or practice in cardiothoracic radiology did not influence sensitivity for error detection. There is no significant difference in interpretation time for studies without photographs and those with photographs (60 ± 22 vs. 61 ± 25 seconds; P = .77)., Conclusions: In this observer study, facial photographs obtained simultaneously with portable chest radiographs increased the identification of any wrong-patient errors, without substantial increase in interpretation time. This technique offers a potential means to increase patient safety through correct patient identification., (Copyright © 2014 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: Delays between order and magnetic resonance (MR) exam often result when using the conventional paper-based MR safety screening process. The impact of an electronic MR safety screening process imbedded in a computerized physician order entry (CPOE) system was evaluated., Materials and Methods: Retrospective chart review of 4 months of inpatient MR exam orders and reports was performed before and after implementation of electronic MR safety documentation. Time from order to MR exam completion, time from MR exam completion to final radiology report, and time from first order to final report were analyzed by exam anatomy. Length of stay (LOS) and date of service within the admission were also analyzed., Results: We evaluated 1947 individual MR orders in 1549 patients under an institutional review board exemption and a waiver of informed consent. Implementation of the electronic safety screening process resulted in a significant decrease of 1.1 hours (95% confidence interval 1.0-1.3 hours) in the mean time between first order to final report and a nonsignificant decrease of 0.8 hour in the median time from first order to exam end. There was a 1-day reduction (P = .697) in the time from admission to the MR exam compared to the paper process. No significant change in LOS was found except in neurological intensive care patients imaged within the first 24 hours of their admission, where a mean 0.9-day decrease was found., Conclusion: Benefits of an electronic process for MR safety screening include enabling inpatients to have decreased time to MR exams, thus enabling earlier diagnosis and treatment and reduced LOS., (Copyright © 2013 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: Multireader imaging trials often use a factorial design, in which study patients undergo testing with all imaging modalities and readers interpret the results of all tests for all patients. A drawback of this design is the large number of interpretations required of each reader. Split-plot designs have been proposed as an alternative, in which one or a subset of readers interprets all images of a sample of patients, while other readers interpret the images of other samples of patients. In this paper, the authors compare three methods of analysis for the split-plot design., Materials and Methods: Three statistical methods are presented: the Obuchowski-Rockette method modified for the split-plot design, a newly proposed marginal-mean analysis-of-variance approach, and an extension of the three-sample U-statistic method. A simulation study using the Roe-Metz model was performed to compare the type I error rate, power, and confidence interval coverage of the three test statistics., Results: The type I error rates for all three methods are close to the nominal level but tend to be slightly conservative. The statistical power is nearly identical for the three methods. The coverage of 95% confidence intervals falls close to the nominal coverage for small and large sample sizes., Conclusions: The split-plot multireader, multicase study design can be statistically efficient compared to the factorial design, reducing the number of interpretations required per reader. Three methods of analysis, shown to have nominal type I error rates, similar power, and nominal confidence interval coverage, are available for this study design., (Copyright © 2012 AUR. All rights reserved.)

Rationale and Objectives: Before computer-aided detection (CAD) algorithms can be used in clinical practice, they must be shown to improve readers' diagnostic accuracy over their unaided performance. This is usually accomplished through a large multireader, multicase (MRMC) clinical trial. It is burdensome, however, for an MRMC study to be performed with each new release of a CAD algorithm. The aim of this report is to present an approach for building models to predict readers' accuracy with a new CAD algorithm., Materials and Methods: A modeling approach for predicting readers' results with a new CAD algorithm is described. Multiple-variable logistic regression was used to build models for readers' sensitivity and false-positive rate, given the results of an MRMC study with an older CAD algorithm and the stand-alone performance results of a new CAD algorithm. Data from a large lung MRMC CAD trial are used to illustrate the modeling approach and test the ability of the models to predict readers' accuracy with the new CAD algorithm., Results: The model overestimated the readers' actual sensitivity with the new CAD algorithm, but this did not reach statistical significance (0.621 vs 0.603, P = .147). The observed and predicted false-positive rates also did not differ significantly (0.275 vs 0.285, P = .250)., Conclusions: Using one clinical study as a test case, it is shown that the modeling approach is feasible. More testing of the approach is needed to determine if and under what circumstances it can be used as an alternative to a full-scale MRMC study. Meanwhile, the approach can be used to determine if a new CAD algorithm is likely to improve readers' accuracy before embarking on a full-scale MRMC study., (Copyright © 2011 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: We describe a step-by-step procedure for estimating power and sample size for planned multireader receiver operating characteristic (ROC) studies that will be analyzed using either the Dorfman-Berbaum-Metz (DBM) or Obuchowski-Rockette (OR) method. This procedure updates previous approaches by incorporating recent methodological developments and unifies the approaches by allowing inputs to be conjectured parameter values or outputs from either a DBM or OR pilot-study analysis., Materials and Methods: Power computations are described in a step-by-step procedure and the theoretical basis for the procedure is described. Updates include using the currently recommended denominator degrees of freedom, accounting for different pilot and planned study normal-to-abnormal case ratios, and a new method for computing the OR test-by-reader variance component., Results: Using a real dataset we illustrate how to compute the power for two planned studies, one having the same normal-to-abnormal case ratio as the pilot study and the other having a different ratio. In a simulation study, we show that the proposed procedure gives mean power estimates close to the true power., Conclusions: Application of the updated procedure is straightforward. It is important that pilot data be comparable to the planned study with respect to the modalities, reader expertise, and case selection. Variability of the power estimates warrants further investigation., (Published by Elsevier Inc.)

Rationale and Objectives: The goal of many multiple-observer computer-aided detection (CADe) studies is to estimate the change in observers' diagnostic performance with CADe from their unaided performance. A key issue in these studies is the method for estimating the observers' unaided performance. The crossover design is considered the most valid. The sequential design takes less time and is less expensive but may be biased. We conducted a study to investigate the differences between these two designs., Materials and Methods: Data from two large CADe studies using both types of unaided reads were analyzed. The first study involved three radiologists examining the chest x-rays of 200 patients for lung nodules. The second study involved 19 observers interpreting the computed tomography colonography images of 100 patients for polyps. Observers' sensitivity, specificity, and receiver operating characteristic areas were estimated while unaided in both designs and compared to their accuracy with CADe. Bias, inter-observer variability, and correlations between unaided and aided results were assessed., Results: Observers tend to perform better while unaided in the sequential design than while unaided in the crossover design, but the differences are small. The inter-observer variability is larger in the sequential design. The correlations between unaided and aided results are larger in the sequential design. 95% CIs for the change with CADe are narrower with the sequential design., Conclusion: The estimated effect of CADe on observer performance is similar regardless of the study design. Use of the sequential design may save investigators time and resources., (Copyright (c) 2010 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: Multireader, multicase (MRMC) receiver-operating characteristic studies often require large numbers of patients, readers, and reader interpretations. The objective of this work is to evaluate a new "mixed" MRMC study design that reduces the number of reader interpretations., Materials and Methods: As compared to the traditional MRMC design, the number of reader interpretations and the number of cases that must be truth-verified for the new mixed design was evaluated theoretically and empirically for various correlation values and sample sizes., Results: For large MRMC studies, the new mixed design offers a substantial savings in the number of reader interpretations if the magnitude of the difference in between-reader correlations is not zero. For example, compared to a traditional design with 20 readers, 200 total cases, and a difference in between-reader correlations of 0.05, the newly proposed mixed design requires each reader to interpret only 132 cases, but at a cost of truth-verifying an additional 64 cases., Conclusions: The mixed design can reduce the number of cases that readers need to interpret and the overall duration of a study, but at a cost in terms of the number of cases that must be truth-verified. The mixed design is particularly useful for studies where the condition being detected is not rare and patients routinely undergo the gold standard assessment.

Rationale and Objectives: Total body screening, despite its popularity, has not been evaluated in clinical trials. Even the appropriate target for screening has not been addressed. In this study, we determined the variables from a subject's demographic and medical and family history that are predictive of actionable findings on total body screening., Materials and Methods: Over a 3-year period, 982 self-referred subjects underwent total body screening with multislice computed tomography and completed a demographic and medical history questionnaire. The study sample was divided into training and testing samples. Univariate and multiple-variable statistical methods were used on the training sample to derive models that predict actionable lung findings, actionable heart findings, actionable abdomen/pelvis findings, and any actionable findings on total body screening. The training models were then applied and evaluated on the test sample., Results: A subject's age at the time of screening was the single most important predictor and often the only significant predictor of actionable findings. Among subjects younger than 40 years of age, 22.5% had actionable findings; this number nearly doubled, to 43.5%, for subjects between 40 and 49, and increased to 80% for subjects 80 years and older. Overall, every increase of 10 years in age brings an increase of 1.6 in the likelihood of an actionable finding., Conclusions: Total body screening targeted at older subjects has the highest yield of actionable findings. The efficacy and cost-effectiveness of total body screening for older subjects is unknown and needs further assessment.

Rationale and Objectives: Investigators often need to assess the accuracies of diagnostic tests when the gold standard is not binary-scale. The objective of this article is to describe nonparametric estimators of diagnostic test accuracy when the gold standard is continuous, ordinal, and nominal scale., Materials and Methods: A nonparametric method of estimating and comparing the area under receiver operating characteristic (ROC) curves, proposed by DeLong et al, is extended to situations in which the gold standard is not binary. Two examples illustrate the methods., Results: Measures of diagnostic test accuracy, their variance, and tests for comparing two diagnostic tests' accuracies in paired designs are presented for situations in which the gold standard is continuous, ordinal, and nominal scale. These summary measures of diagnostic test accuracy are analogous in form and interpretation to the area under the ROC curve., Conclusion: Dichotomizing the outcomes of a gold standard so that traditional ROC methods can be applied can lead to bias. The methods described here are useful for assessing and comparing summary test accuracy when the gold standard is not binary scale. They have limitations similar to other summary indices.

Rationale and Objectives: Several statistical methods have been developed for analyzing multireader, multicase (MRMC) receiver operating characteristic (ROC) studies. The objective of this article is to increase awareness of these methods and determine if their results are concordant for published datasets., Materials and Methods: Data from three previously published studies were reanalyzed using five MRMC methods. For each method the 95% confidence intervals (CIs) for the mean of the readers' ROC areas for each diagnostic test, the P value for the comparison of the diagnostic tests' mean accuracies, and the 95% CIs for the mean difference in ROC areas of the diagnostic tests were reported., Results: Important differences in P values and CIs were seen when using parametric versus nonparametric estimates of accuracy, and there were the expected differences for random-reader versus fixed-reader models. Controlling for these differences, the Dorfman-Berbaum-Metz (DBM), Obuchowski-Rockette, Beiden-Wagner-Campbell, and Song's multivariate Wilcoxon-Mann-Whitney (WMW) methods gave almost identical results for the fixed-reader model. For the random-reader model, the DBM, Obuchowski-Rockette, and Beiden-Wagner-Campbell methods yielded approximately the same inferences, but the CIs for the Beiden-Wagner-Campbell method tend to be broader. Ishwaran's hierarchical ROC sometimes yielded significance not found with other methods. Song's modification of DBM's jack-knifing algorithm sometimes led to different conclusions than the original DBM algorithm., Conclusion: In choosing and applying MRMC methods, it is important to recognize: (1) the distinction between random-reader and fixed-reader models, the uncertainties accounted for by each, and thus the level of generalizeability expected from each; (2) assumptions made by the various MRMC methods; and (3) limitations of a five- or six-reader study when the reader variability is great.

Rationale and Objectives: In studies with small samples, the authors often encounter data sets in which the estimated area under the receiver operating characteristic (ROC) curve is 1.0. In such cases, neither asymptotic nor resampling methods provide a means of estimating the standard error or constructing a lower confidence bound. The purpose of this study was to develop tables for determining the approximate 95% lower confidence bound when the estimated ROC area is 1.0., Materials and Methods: Using Monte Carlo simulation, the authors generated 10,000 data sets for each specification of sample sizes, ROC curve shape, and data format (continuous and ordinal scale). For each of these combinations the authors determined the 95% lower confidence bound., Results: When the estimated ROC area is 1.0, the 95% lower confidence bounds differ dramatically depending on the shape of the ROC curve and on whether the test results are ordinal or continuous. Four tables of 95% lower confidence bounds are provided, along with guidelines for their use., Conclusion: Given the different shapes of ROC curves and the different formats in which ROC data are collected, it is not feasible to offer one simple method of constructing confidence bounds that works for all ROC curves. The tables provided in this article are useful for interpreting studies with estimated ROC areas of 1.0.