Reinforcement Learning by Policy Search

Item

Title
en_US Reinforcement Learning by Policy Search
Creator
en_US Peshkin, Leonid
Date
2004-10-20T20:31:39Z
Date Available
2004-10-20T20:31:39Z
Date Issued
en_US 2003-02-14
Identifier
en_US AITR-2003-003
uri
http://hdl.handle.net/1721.1/7101
Abstract
en_US One objective of artificial intelligence is to model the behavior of an intelligent agent interacting with its environment. The environment's transformations can be modeled as a Markov chain, whose state is partially observable to the agent and affected by its actions; such processes are known as partially observable Markov decision processes (POMDPs). While the environment's dynamics are assumed to obey certain rules, the agent does not know them and must learn. In this dissertation we focus on the agent's adaptation as captured by the reinforcement learning framework. This means learning a policy---a mapping of observations into actions---based on feedback from the environment. The learning can be viewed as browsing a set of policies while evaluating them by trial through interaction with the environment. The set of policies is constrained by the architecture of the agent's controller. POMDPs require a controller to have a memory. We investigate controllers with memory, including controllers with external memory, finite state controllers and distributed controllers for multi-agent systems. For these various controllers we work out the details of the algorithms which learn by ascending the gradient of expected cumulative reinforcement. Building on statistical learning theory and experiment design theory, a policy evaluation algorithm is developed for the case of experience re-use. We address the question of sufficient experience for uniform convergence of policy evaluation and obtain sample complexity bounds for various estimators. Finally, we demonstrate the performance of the proposed algorithms on several domains, the most complex of which is simulated adaptive packet routing in a telecommunication network.
Extent
en_US 144 p.
26942112 bytes
1735254 bytes
Format
application/postscript
application/pdf
Language
en_US
Relation
en_US AITR-2003-003
Subject
en_US AI
en_US POMDP
en_US policy search
en_US adaptive systems
en_US reinforcement learning
en_US adaptive behavior
Item sets
AI Technical Reports (1964 - 2004)