Search

従来研究では，推薦システムを表現するための確率モデルとしてマルコフ決定過程が採用されている．他方，多くの分野において，ニューラルネットワークを用いた強化学習方法が提案されている．しかし，推薦システムにおけるニューラルネットワークを用いた強化学習方法は提案されていない．そこで，本研究では，マルコフ決定過程の真のパラメータが未知の仮定のもとで推薦システムにおける全結合ニューラルネットワークを用いた強化学習方法を提案する．提案方法では顧客の性質を表現するために顧客の履歴情報を利用する．シミュレーションによって提案方法の有効性を示す．シミュレーション結果では，提案方法の出力が最適解と一致した．, [ENG] Markov decision processes are applied to recommender system in previous research. Reinforcement learning methods using neural networks have been proposed in many fields. But a reinforcement learning method using neural networks has not been proposed in recommender system. In this research we propose a reinforcement learning method using fully connected neural networks in recommender system under the condition that the true parameters of Markov decision processes are unknown. The proposed method uses historical data of customers to represent customers' properties. The effectiveness of the proposed method is shown by some simulations. The output of the proposed method is equal to the optimal solution in the simulation result.

従来研究では，ヘルスケア支援方法を表現するための確率モデルとしてマルコフ決定過程が採用され，マルコフ決定過程の真のパラメータが既知の仮定のもとで検討されている．本研究では，より現実に近い真のパラメータが未知の仮定のもとでヘルスケア支援のための半教師付き学習方法を提案する．学習データは完全データと不完全データによって構成される．提案方法ではEMアルゴリズム(expectation-maximization algorithm) を用いる．数例のシミュレーションによって提案方法の有効性を示す．シミュレーション結果より，学習データが大きくなるにつれて学習精度が高くなることが確認できる．, [ENG] Markov decision processes are applied to a healthcare support method in previous research. In the previous research the true parameters of Markov decision processes are known. In this research we propose a semi-supervised learning method for the healthcare support method under the condition that the true parameters of Markov decision processes are unknown. Learning data consist of complete data and incomplete data. In the proposed method EM(expectation-maximization) algorithm is used. The effectiveness of the proposed method is shown by some simulations. The result shows that the learning accuracy becomes higher as the learning data becomes bigger.

従来研究では，顧客クラスが変化する推薦システムを表現する確率モデルとしてマルコフ決定過程が採用され，マルコフ決定過程の真のパラメータが既知の仮定のもとで検討されている．本研究では，真のパラメータが未知の仮定のもとで顧客クラスが変化する推薦システムにおける半教師付き学習方法を提案する．学習データは完全データと不完全データによって構成される．提案方法ではEMアルゴリズムを用いる．シミュレーションによって提案方法の有効性を示す．, Markov decision processes(MDP) are applied to recommender system with transitions of user classes. In previous research the true parameters of MDP are known. In this research we propose a semi-supervised learning method for recommender system with transitions of user classes under the condition that the true parameters are unknown. Learning data consist of complete data and incomplete data. In the proposed method EM algorithm is used. The effectiveness of the proposed method is shown by some simulations.

従来研究では，推薦システムの新規顧客問題を表現するための確率モデルとしてマルコフ決定過程が採用され，マルコフ決定過程の真のパラメータが既知の仮定のもとで検討されている．本研究では，より現実に近い真のパラメータが未知の仮定のもとで推薦システムの新規顧客問題における半教師付き学習方法を提案する．学習データは完全データと不完全データによって構成される．提案方法ではEMアルゴリズム（expectation-maximization algorithm）を用いる．数例のシミュレーションによって提案方法の有効性を示す．, Markov decision processes(MDP) are applied to a new customer problem of recommender system in previous research. In the previous research the true parameters of MDP are known. In this research we propose a semi-supervised learning method for the new customer problem of recommender system under the condition that the true parameters of MDP are unknown. Learning data consist of complete data and incomplete data. In the proposed method EM(expectation-maximization) algorithm is used. The effectiveness of the proposed method is shown by some simulations.

推薦システムにおける従来研究では確率モデルとしてマルコフ決定過程が採用されている．当該従来研究ではベイズ基準のもとで総利得が最大化されているが，新規顧客に対する質問方法は検討されていない．また，新規顧客に対する質問方法を検討している従来研究もあるが，当該従来研究では総利得の最大化は検討されていない． そこで，本研究ではマルコフ決定過程を推薦システムにおける新規顧客に対する質問に適用し，総利得をベイズ基準のもとで最大化する質問方法を提案する．提案アルゴリズム中では，最適化手法として動的計画法を用いる．, In previous research of recommender system Markov decision processes are used in order to represent recommender system. Total reward is maximized with reference to a Bayes criterion. In the previous research a questionnaire method for a new customer is not studied. In other previous research a questionnaire method for a new customer is studied, but the total reward is not maximized. In this research we apply Markov decision processes to questionnaire for a new customer in recommender system. We propose a new questionnaire method which maximizes the total reward with reference to the Bayes criterion. In the proposed method dynamic programming is used.

本研究では，マルコフ決定過程を用いて講義をモデル化し，統計的決定理論に基づく2つの教授戦略算出方法を提案する．従来研究で算出される教授戦略は1回の講義の効果を最大化するのに対して，本研究で算出する教授戦略は複数の講義で構成される科目全体での効果を最大化する．提案方法では動的計画法を利用している．1つ目の提案方法は個別指導用の方法で， 2つ目の提案方法は複数指導用の方法である．教育目的はマルコフ決定過程の利得関数によって表現される．また，教授戦略はマルコフ決定過程の政策によって表現される．提案方法によって算出される最適な教授戦略はマルコフ決定過程の期待利得を最大化する．提案方法の有効性を示すために数値計算例を紹介する．, In this research Markov decision processes are used in order to represent lectures. The best teaching strategy computed in previous research maximizes the effectiveness of a lecture. The best teaching strategy computed in this research maximizes the effectiveness of a subject which is composed of multiple lectures. We propose two computation methods based on statisticαl decision theory for teaching strategies. Dynamic programming algorithm is used in the proposed methods. The first proposed method selects teaching materials for α learner. The second proposed method selects teaching materials for multiple learners. A purpose of education is represented by the reward function of Markov decision processes. A teaching strategy is represented by the policy of Markov decision processes. The best teaching strategies computed by the proposed methods maximize the expected reward of Markov decision processes. We show the effectiveness of the proposed methods by some numerical calculation examρles.

従来からマルコフ決定過程（MDP）を用いたロールプレイングゲーム（RPG）のモデル化が行われている．しかし，RPGの攻略法を能動的に学習する研究は行われていない．そこで，本研究では，真のパラメータが未知のMDPで表現されたRPGにおける期待総利得をベイズ基準のもとで最大にする攻略法を求める能動的な学習方法を提案する．シミュレーションをとおして，提案方法の有効性を確認する．, In previous research a role-playing game(RPG) is represented with Markov decision processes(MDP). But active learning method for RPG hαs not been studied yet. In this research we propose an active learning method which mαximizes an expected total reward with respect to a Bayes criterion under the condition that the true parameter of MDP is unknown. We recognize the effectiveness of our proρosed method by some simulations.

従来からマルコフ決定過程（MDP）を用いたロールプレイングゲーム（RPG）のモデル化が行われている．従来研究ではRPGが部分的にモデル化されている．本研究では，MDPを用いてより一般的なRPGのモデル化を行う．最初にMDPの真のパラメータ既知の場合に相当するRPGについて，報酬の期待値を最大にするアルゴリズムを提案する．次にMDPの真のパラメータ未知の場合に相当するRPGについて，ベイズ基準のもとで報酬を最大にするアルゴリズムを提案する．次にMDPの真のパラメータ未知の場合に相当するRPGについて，学習データを用いて報酬を近似的に最大にするアルゴリズムを提案する．In previous research a part of role-playing game (RPG) is represented with Markov decision processes (MDP). In this research we represent a more general PRG with MDP. We maximize an expected total reward under the condition that the true parameter of MDP is known in the first proposition. We maximize an expected total reward with respect to a Bayes criterion under the condition that the true parameter of MDP is unknown in the second proposition. We approximately maximize an expected total reward using learning data under the condition that the true parameter of MDP is unknown in the third proposition.

In previous research Markov decision processes has been applied to managing credit lines. And an expected total discounted reward was maximized. But an outer environment (e.g. economic growth) was not represented in the model. So in this research we apply Markov decision processes with the outer environment to managing credit lines. And we propose a new managing credit lines method which maximizes expected total discounted reward in the Markov decision processes with the outer environment.

This paper investigates properties of the optimaiity equation and optimal policies in discrete time Markov decision processes with expected discounted total rewards. Under conditions where the model is well defined and the optimaiity equation is true, it is shown that as a solution of the optimaiity equation, the solution called optimal value function is always the smallest one, and is also the unique one under another weak condition. Moreover, a structure of optimal policies is discussed. Finally, these properties are applied to state feedback control of discrete event systems with a numerical example.

One most common problem in reinforcement learning systems (e.g. Q-learning) is to reduce the number of trials to converge to an optimal policy. As one of the solution to the problem, k-certainty exploration method was proposed. Miyazaki reported that this method could determine an optimal policy faster than Q-learning in Markov decision processes (MDPs). This method is very efficient learning method. But, we propose an improvement plan that makes this method more efficient. In k-certainty exploration method, in case there is no k-uncertainty rule (rules which aren't selected more than k times) in current state, an agent sometimes walks randomly until it finds a state where it can select a k-uncertainty rule. We think that this behavior is not efficient. To reduce this useless behavior, we propose combining kcertainty exploration method with Dynamic programming (DP). Miyazaki's system uses DP after all rules are executed at least k times. But, our method uses k-certainty exploration method along with DP during learning. Our method, takes two pattern actions. In case an agent can select k-uncertainty rules, one of these rules is selected at random as k-certainty exploration method. In another case there is no k-uncertainty rule, behavior of agent is different from the behavior of k-certainty exploration method. In that case, our method uses DP to compute an optimal policy for moving from a current state to a state in which some k-uncertainty rules remain. The model for DP is constructed by using only known states. The outline will be described below. First, an agent makes a map constructed by using only known states. In the map, goals are states in which there are k-uncertainty rules and arbitrary state values are set in these states. Point to which attention should be paid is that the map is not given from outside. It is made from only experience. Next, each state values of states in which there are only k-certainty rules in the map are computed by DP (we used Policy Iteration). Finally, an agent continues to select greedy action until it arrives at a state in which it can select a k-uncertainty rule. By this improvement, we expect it can determine an optimal policy faster than k-certainty exploration method. And we have verified that our exploration method can determine an optimal policy faster than k-certainty exploration method by computer simulation.