Header logo is

Active Exploration for Robot Parameter Selection in Episodic Reinforcement Learning

2011

Conference Paper

ei


As the complexity of robots and other autonomous systems increases, it becomes more important that these systems can adapt and optimize their settings actively. However, such optimization is rarely trivial. Sampling from the system is often expensive in terms of time and other costs, and excessive sampling should therefore be avoided. The parameter space is also usually continuous and multi-dimensional. Given the inherent exploration-exploitation dilemma of the problem, we propose treating it as an episodic reinforcement learning problem. In this reinforcement learning framework, the policy is defined by the system's parameters and the rewards are given by the system's performance. The rewards accumulate during each episode of a task. In this paper, we present a method for efficiently sampling and optimizing in continuous multidimensional spaces. The approach is based on Gaussian process regression, which can represent continuous non-linear mappings from parameters to system performance. We employ an upper confidence bound policy, which explicitly manages the trade-off between exploration and exploitation. Unlike many other policies for this kind of problem, we do not rely on a discretization of the action space. The presented method was evaluated on a real robot. The robot had to learn grasping parameters in order to adapt its grasping execution to different objects. The proposed method was also tested on a more general gain tuning problem. The results of the experiments show that the presented method can quickly determine suitable parameters and is applicable to real online learning applications.

Author(s): Kroemer, O. and Peters, J.
Journal: Proceedings of the 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL 2011)
Pages: 25-31
Year: 2011
Month: April
Day: 0
Publisher: IEEE

Department(s): Empirical Inference
Bibtex Type: Conference Paper (inproceedings)

DOI: 10.1109/ADPRL.2011.5967378
Event Name: IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL 2011)
Event Place: Paris, France

Address: Piscataway, NJ, USA
Digital: 0
ISBN: 978-1-4244-9887-1
Language: en
Organization: Max-Planck-Gesellschaft
School: Biologische Kybernetik

Links: PDF
Web

BibTex

@inproceedings{7050,
  title = {Active Exploration for Robot Parameter Selection in Episodic Reinforcement Learning},
  author = {Kroemer, O. and Peters, J.},
  journal = {Proceedings of the 2011 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL 2011)},
  pages = {25-31},
  publisher = {IEEE},
  organization = {Max-Planck-Gesellschaft},
  school = {Biologische Kybernetik},
  address = {Piscataway, NJ, USA},
  month = apr,
  year = {2011},
  doi = {10.1109/ADPRL.2011.5967378 },
  month_numeric = {4}
}