Search

This research studies the use of predetermined experimental plans in a live setting with a finite implementation horizon. In this context, we seek to determine the optimal experimental budget in different environments using a Bayesian framework. We derive theoretical results on the optimal allocation of resources to treatments with the objective of minimizing cumulative regret, a metric commonly used in online statistical learning. Our base case studies a setting with two treatments assuming Gaussian priors for the treatment means and noise distributions. We extend our study through analytical and semi-analytical techniques which explore worst-case bounds, the presence of unequal prior distributions, and the generalization to k treatments. We determine theoretical limits for the experimental budget across all possible scenarios. The optimal level of experimentation that is recommended by this study varies extensively and depends on the experimental environment as well as the number of available units. This highlights the importance of such an approach which incorporates these factors to determine the budget.

This paper presents a conceptually simple and resource efficient method for robust parameter design. The proposed method varies control factors according to an adaptive one-factor-at-a-time plan while varying noise factors using a two-level resolution III fractional factorial array. This method is compared with crossed arrays by analyzing a set of four case studies to which both approaches were applied. The proposed method improves system robustness effectively, attaining more than 80% of the potential improvement on average if experimental error is low. This figure improves to about 90% if prior knowledge of the system is used to define a promising starting point for the search. The results vary across the case studies, but, in general, both the average amount of improvement and the consistency of the results are better than those provided by crossed arrays if experimental error is low or if the system contains some large interactions involving two or more control factors. This is true despite the fact that the proposed method generally uses fewer experiments than crossed arrays. The case studies reveal that the proposed method provides these benefits by exploiting, with high probability, both control by noise interactions and also higher order effects involving two control factors and a noise factor. The overall conclusion is that adaptive one-factor-at-a-time, used in concert with factorial outer arrays, is demonstrated to be an effective approach to robust parameter design providing significant practical advantages as compared to commonly used alternatives.