Latent Stratification for Incrementality Experiments.

A method is proposed to compute more precise standard errors for marketing A/B tests. Incrementality experiments compare customers exposed to a marketing action designed to increase sales with those randomly assigned to a control group. These experiments suffer from noisy responses, which make precise estimation of the average treatment effect (ATE) and marketing return difficult. We develop a model that improves the precision by estimating separate treatment effects for three latent strata defined by potential outcomes in the experiment—customers who would buy regardless of ad exposure, those who would buy only if exposed to ads, and those who would not buy regardless. The overall ATE is estimated by averaging the strata-level effects, and this produces a more precise estimator of the ATE over a wide range of conditions typical of marketing experiments. Analytical results and simulations show that the method decreases the sampling variance of the ATE most when (1) there are large differences in the treatment effect between latent strata and (2) the model used to estimate the strata-level effects is well identified. Applying the procedure to five catalog experiments shows a reduction of 30%–60% in the variance of the overall ATE. This leads to a substantial decrease in decision errors when the estimator is used to determine whether ads should be continued or discontinued. History: Olivier Toubia served as the senior editor. Funding: This work was supported by Adobe Systems [Adobe Digital Experience Research Award]. R. Berman was supported by the United States-Israel Binational Science Foundation [Grant 2020022] and the Wharton Dean's Research Fund, and E. M. Feit was supported by the Dean's Fellowship from LeBow College of Business. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mksc.2022.0297. [ABSTRACT FROM AUTHOR]