Analysis of Proportional Data in Reproductive and Developmental Toxicity Studies: Comparison of Sensitivities of Logit Transformation, Arcsine Square Root Transformation, and Nonparametric Analysis

BACKGROUND: In developmental and reproductive toxicity studies, analysis of litter-based binary endpoints (e.g., incidence of malformed fetuses) is complex in that littermates often are not entirely independent of one another. It is well established that the litter, not the individual fetus, is the proper independent experimental unit in statistical analysis. Accordingly, analysis is often based on the proportion affected per litter and the litter proportions are analyzed as continuous data. Because these proportional data generally do not meet assumptions of symmetry or normality, data are typically analyzed by nonparametric methods, arcsine square root transformation, or logit transformation. METHODS: We conducted power calculations to compare different approaches (nonparametric, arcsine square root-transformed, logit-transformed, untransformed) for analyzing litter-based proportional data. A reproductive toxicity study with a control and one treated group provided data for two endpoints: prenatal loss, and fertility by in utero insemination (IUI). Type 1 error and power were estimated by 10,000 simulations based on two-sample one-tailed t-tests with varying numbers of litters per group. To further compare the different approaches, we conducted additional analyses with shifted mean proportions to produce illustrative scenarios. RESULTS: Analyses based on logit-transformed proportions had greater power than those based on untransformed or arcsine square root-transformed proportions, or nonparametric procedures. CONCLUSION: The logit transformation is preferred to the other approaches considered when making inferences concerning litter-based proportional endpoints, particularly with skewed distributions. The improved performance of the logit transformation becomes increasingly pronounced as the response proportions are increasingly close to the boundaries of the parameter space.