101. Model-based Constrained MDP for Budget Allocation in Sequential Incentive Marketing
- Author
-
Jiang Zaifan, Le Guo, Zhu Jun, Lei Lv, Shuai Xiao, Chen Yuanbo, and Shuang Yang
- Subjects
Counterfactual thinking ,FOS: Computer and information sciences ,Computer Science - Artificial Intelligence ,Computer science ,010501 environmental sciences ,01 natural sciences ,Dual (category theory) ,010104 statistics & probability ,symbols.namesake ,Variable (computer science) ,Incentive ,Artificial Intelligence (cs.AI) ,Lagrangian relaxation ,symbols ,Reinforcement learning ,Markov decision process ,0101 mathematics ,Marketing ,Budget constraint ,0105 earth and related environmental sciences - Abstract
Sequential incentive marketing is an important approach for online businesses to acquire customers, increase loyalty and boost sales. How to effectively allocate the incentives so as to maximize the return (e.g., business objectives) under the budget constraint, however, is less studied in the literature. This problem is technically challenging due to the facts that 1) the allocation strategy has to be learned using historically logged data, which is counterfactual in nature, and 2) both the optimality and feasibility (i.e., that cost cannot exceed budget) needs to be assessed before being deployed to online systems. In this paper, we formulate the problem as a constrained Markov decision process (CMDP). To solve the CMDP problem with logged counterfactual data, we propose an efficient learning algorithm which combines bisection search and model-based planning. First, the CMDP is converted into its dual using Lagrangian relaxation, which is proved to be monotonic with respect to the dual variable. Furthermore, we show that the dual problem can be solved by policy learning, with the optimal dual variable being found efficiently via bisection search (i.e., by taking advantage of the monotonicity). Lastly, we show that model-based planing can be used to effectively accelerate the joint optimization process without retraining the policy for every dual variable. Empirical results on synthetic and real marketing datasets confirm the effectiveness of our methods., Comment: Published at CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management
- Published
- 2023
- Full Text
- View/download PDF