荒井 幸代

In recent years, inverse reinforcement learning has attracted attention as a method for estimating the intention of actions using the trajectories of various action-taking agents, including human flow data. In the context of reinforcement learning, “intention” refers to a reward function. Conventional inverse reinforcement learning assumes that all trajectories are generated from policies learned under a single reward function. However, it is natural to assume that people in a human flow act according to multiple policies. In this study, we introduce an expectation-maximization algorithm to inverse reinforcement learning, and propose a method to estimate different reward functions from the trajectories of human flow. The effectiveness of the proposed method was evaluated through a computer experiment based on human flow data collected from subjects around airport gates.

Sequential decision-making under multiple objective functions includes the problem of exhaustively searching for a Pareto-optimal policy and the problem of selecting a policy from the resulting set of Pareto-optimal policies based on the decision maker’s preferences. This paper focuses on the latter problem. In order to select a policy that reflects the decision maker’s preferences, it is necessary to order these policies, which is problematic because the decision-maker’s preferences are generally tacit knowledge. Furthermore, it is difficult to order them quantitatively. For this reason, conventional methods have mainly been used to elicit preferences through dialogue with decision-makers and through one-to-one comparisons. In contrast, this paper proposes a method based on inverse reinforcement learning to estimate the weight of each objective from the decision-making sequence. The estimated weights can be used to quantitatively evaluate the Pareto-optimal policies from the viewpoints of the decision-makers preferences. We applied the proposed method to the multi-objective reinforcement learning benchmark problem and verified its effectiveness as an elicitation method of weights for each objective function.

<jats:title>Abstract</jats:title><jats:p>The effective utilization of regenerative power generated by trains has attracted the attention of engineers due to its promising potential in energy conservation for electrified railways. Charge control by wayside battery batteries is an effective method of utilizing this regenerative power. Wayside batteries requires saving energy by utilizing the minimum storage capacity of energy storage devices. However, because current control policies are rule‐based, based on human empirical knowledge, it is difficult to decide the rules appropriately considering the battery's state of charge. Therefore, in this paper, we introduce reinforcement learning with an actor‐critic algorithm to acquire an effective control policy, which had been previously difficult to derive as rules using experts’ knowledge. The proposed algorithm, which can autonomously learn the control policy, stabilizes the balance of power supply and demand. Through several computational simulations, we demonstrate that the proposed method exhibits a superior performance compared to existing ones.</jats:p>

In this paper, we introduce an intelligent vehicle in traffic flow where a phantom traffic jam occurs for ensuring traffic-flow stability. The intelligent vehicle shares information on the speed and gap of the leading vehicle. Furthermore, the intelligent vehicle can foresee changes in the leading vehicles through shared information and can start accelerating faster than human-driven vehicles can. We propose an intelligent vehicle model, which is a generalized Nagel-Schreckenberg model can arbitrarily set the number of leading vehicles to share information with and set maximum distance of inter-vehicle communication. We found that phantom traffic jams are suppressed by an intelligent vehicle that can share information with two or more vehicles in front and information at least 30 meters away.

Multi-Objective Reinforcement Learning (MORL) can he divided into two approaches according to the number of acquired policies. One approach learns a single policy that makes the agent reach a single arbitral Pareto optimal solution, and the other approach learns multiple policies that correspond to each Pareto optimal solution. The latter approach finds the multiple policies simultaneously; however, it incurs significant computational cost. In many real-world cases, learning a single solution is sufficient in the multi-objective context. In this paper, we focus on the former approach where a suitable weight of each object must be defined. To estimate the weight of each object as parameters, we utilize Q-values on the expert's trajectory, which indicates the optimal sequence of actions. This approach is an analogy obtained from apprenticeship learning via inverse reinforcement learning. We evaluate the proposed method using a well-known MORL benchmark problem, i.e., the Deep Sea Treasure environment.

In this paper, we introduce an intelligent vehicle in traffic flow where a phantom traffic jam occurs for ensuring traffic-flow stability. The intelligent vehicle shares information on the speed and gap of the leading vehicle. Furthermore, the intelligent vehicle can foresee changes in the leading vehicles through shared information and can start accelerating faster than human-driven vehicles can. We propose an intelligent-vehicle model, which is a generalized Nagel-Schreckenberg model that allows sharing information with leading vehicles. The generalized Nagel-Schreckenberg model can arbitrarily set the number of leading vehicles to share information with, and we found that phantom traffic jams are resolved by an intelligent vehicle that shares information with two or more vehicles in front.

In this paper, we propose a method to diminish the state space explosion problem of a multiagent reinforcement learning context, where each agent needs to observe other agents' states, and previous actions at each step of its learning process. However, both the number of state and action become exponential in the number of agents, leading to enormous amount of computation and very slow learning. In our method, the agent considers other agents' statuses only when they interfere with one another to reach their goals. Our idea is that each agent starts with its state space which does not include information of others'. Then, they automatically expand and refine their state space when agents detect interference. We adopt the information theory measure of entropy to detect the interference status where agents should take into account the other agents. We demonstrate the advantage of our method over the properties of global convergence in a time efficient manner.

This study is intended to encourage appropriate social norms among multiple agents. Effective norms, such as those emerging from sustained individual interactions over time, can make agents act cooperatively to optimize their performance. We introduce a "social learning" model in which agents mutually interact under a framework of the coordination game. Because coordination games have dual equilibria, social norms are necessary to make agents converge to a unique equilibrium. As described in this paper, we present the emergence of a right social norm by inverse reinforcement learning, which is an approach for extracting a reward function from the observation of optimal behaviors. First, we let a mediator agent estimate the reward function by inverse reinforcement learning from the observation of a master's behavior. Secondly, we introduce agents who act according to an estimated reward function in the multiagent world in which most agents, called citizens, have no way to act. Finally, we evaluate the effectiveness of introducing inverse reinforcement learning. © 2014 Information Processing Society of Japan.

道路交通施策の意思決定者にとって，道路環境を変化させた直後の過渡的な交通を予測することは，環境に関する知識が十分に浸透し収束した後の定常状態を予測することと同様に重要である．著者らはこれまで学習を用いた経路探索アルゴリズムの一つであるQ-routingを応用しながら過渡的な交通状況の再現を試みてきた．本講演では現実の問題にQ-routingを適用した際の経路選択行動の遷移の様子を報告する．

野球が契約料や広告料など巨額の金銭を動かすビジネスになっている.この背景では,期待得点値を最大にするラインナップを構成することは野球チームにとって大きな課題である.期待得点値は,野球の攻撃をマルコフ連鎖として捉えた期待得点値算出モデルにより計算可能である.しかし,n名の選手集合から構成され得る全通りのラインナップの期待得点値を計算するためには,O(n^9)の計算量が必要である.そこで,本稿では打番の要求機能の定量化法,および得られた要求機能を用いて,最適ラインナップ構成問題をマッチング問題に定式化する方法を提案する.また,提案法を,1.打番の要求機能の定量化法の妥当性,2.ラインナップ構成法としての評価の2段階から既存手法との比較により議論する.

<jats:p>Although a large number of reinforcement learning algorithms have been proposed for the generation of cooperative behaviors, the question of how to evaluate mutual benefit or loss among them is still open. As far as we know, an emerged behavior is regarded as a cooperative behavior when embedded agents have finally achieved their global goal, regardless of whether or not mutual interference has had any effect during the course of the learning process of each agent. Thus, we cannot detect any harmful interaction on the way to achieving a fully-converged policy. In this paper, we propose a measure based on information theory for evaluating the degree of interaction during the learning process from the viewpoint of information sharing. In order to discuss the bad effects of concurrent learning, we apply our proposed measure to a situation in which there exist conflicts among the agents, and we show the availability of our measure.</jats:p>

Although a large number of algorithms have been proposed for generating cooperative behaviors, the question of how to evaluate mutual benefit among them is still open. This study provides a measure for cooperation degree among the reinforcement learning agents. By means of our proposed measure, that is based on information theory, the degree of interaction among agents can be evaluated from the viewpoint of information sharing. Here, we show the availability of this measure through some experiments on "pursuit game", and evaluate the degree of cooperation among hunters and prey.

In this paper, we discuss guidelines for a reward design problem that defines when and what amount of reward should be given to the agents, within the context of reinforcement learning approach. We take keepaway soccer as a standard task of multiagent domain which requires skilled teamwork. The difficulties of designing reward for good teamwork are due to its features as follows: i) since it is a continuing task which has no explicit goal, it is hard to tell when reward should be given to the agents, ii) since it is a multiagent cooperative task, it is hard to make a fair share of the reward for each agent's contribution. Through some experiments, we show that reward design have a major effect on the agent's behavior, and introduce the reward function that makes agents perform keepaway successfully.

Reasoning plays a central role in intelligent systems that operate in complex situations that involve time constraints. In this paper, we present the Adaptive Logic Interpreter, a reasoning system that acquires a controlled inference strategy adapted to the scenario at hand, using a variation on relational reinforcement learning. Employing this inference mechanism in a reactive agent architecture lets the agent focus its reasoning on the most rewarding parts of its knowledge base and hence perform better under time and computational resource constraints. We present experiments that demonstrate the benefits of this approach to reasoning in reactive agents, then discuss related work and directions for future research.