Reproducibility of gene expression signature-based predictions in replicate experiments

Purpose: The goals of this analysis were to (a) determine concordance of gene expression results from replicate experiments, (b) examine prediction agreement of multigene predictors on replicate data, and (c) assess the robustness of prediction results in the face of noise. Patients and Methods: Affymetrix U133A gene chips were used for gene expression profiling of 97 fine-needle aspiration biopsies from breast cancer. Thirty-five cases were profiled in replicates: 17 within the same laboratory, 11 in two different laboratories, and 15 to assess manual and robotic labeling. We used data from 62 cases to develop 111 distinct pharmacogenomic predictors of response to therapy. These were tested on cases profiled in duplicates to determine prediction agreement and accuracy. To evaluate the robustness of the pharmacogenomic predictors, we also introduced random noise into the informative genes in one half of the replicates. Results: The average concordance correlation coefficient was 0.978 (range, 0.96-0.99) for intralaboratory replicates, 0.962 (range, 0.94-0.98) for between-laboratory replicates, and 0.971 (range, 0.93-0.99) for manual versus robotic labeling. The mean % prediction agreement on replicate data was 97% (95% CI, 0.96-0.98; SD, 0.006), 92% (95% CI, 0.90-0.93; SD, 0.009), and 94% (95% CI, 0.92-0.95; SD, 0.008) for support vector machines, diagonal linear discriminant analysis, and k-nearest neighbor prediction methods, respectively. Mean accuracy in the test set was 77% (95% CI, 0.74-0.79; SD, 0.014), 66% (95% CI, 0.63-0.73; SD, 0.015), and 64% (95% CI, 0.60-0.67; SD, 0.016), respectively. Conclusion: Gene expression results obtained with Affymetrix U133A chips are highly reproducible within and across two high-volume laboratories. Pharmacogenomic predictions yielded >90% agreement in replicate data.