Measuring e-Commerce Metric Changes in Online Experiments

Digital technology organizations routinely use online experiments (e.g. A/B tests) to guide their product and business decisions. In e-commerce, we often measure changes to transaction- or item-based business metrics such as Average Basket Value (ABV), Average Basket Size (ABS), and Average Selling Price (ASP); yet it remains a common pitfall to ignore the dependency between the value/size of transactions/items during experiment design and analysis. We present empirical evidence on such dependency, its impact on measurement uncertainty, and practical implications on A/B test outcomes if left unmitigated. By making the evidence available, we hope to drive awareness of the pitfall among experimenters in e-commerce and hence encourage the adoption of established mitigation approaches. We also share lessons learned when incorporating selected mitigation approaches into our experimentation analysis platform currently in production.
Comment: To appear in WWW '23 Companion. 5 pages, 4 figures, 2 tables. The experiment code and results on the two publicly available datasets are available on GitHub/Zenodo: https://doi.org/10.5281/zenodo.7659092. This version supersedes a previous working paper with a different title