Skip to content

Learning Robust Control for LQR Systems with Multiplicative Noise via Policy Gradient

Authors: Benjamin Gravell, Peyman Mohajerin Esfahani, Tyler Summers

Published: 2019 (Journal Paper)

Source: IEEE Transactions on Automatic Control

arXiv: 1905.13547

DOI: 10.1109/TAC.2020.3037046

Summary

Provides non-asymptotic finite-sample convergence results for policy gradient algorithms applied to the problem of optimal control of linear systems with multiplicative noise when the dynamics and noise covariances are unknown.

Abstract

The linear quadratic regulator (LQR) problem has reemerged as an important theoretical benchmark for reinforcement learning-based control of complex dynamical systems with continuous state and action spaces. In contrast with nearly all recent work in this area, we consider multiplicative noise models, which are increasingly relevant because they explicitly incorporate inherent uncertainty and variation in the system dynamics and thereby improve robustness properties of the controller. Robustness is a critical and poorly understood issue in reinforcement learning; existing methods which do not account for uncertainty can converge to fragile policies or fail to converge at all. Additionally, intentional injection of multiplicative noise into learning algorithms can enhance robustness of policies, as observed in ad hoc work on domain randomization. Although policy gradient algorithms require optimization of a non-convex cost function, we show that the multiplicative noise LQR cost has a special property called gradient domination, which is exploited to prove global convergence of policy gradient algorithms to the globally optimum control policy with polynomial dependence on problem parameters. Results are provided both in the model-known and model-unknown settings where samples of system trajectories are used to estimate policy gradients.

Primary

Paper arxiv.org

Standard

arXiv Abstract 1905.13547 arXiv PDF 1905.13547 arXiv HTML 1905.13547 DOI 10.1109/TAC.2020.3037046

Alternate

personal.utdallas.edu PDF personal.utdallas.edu ieeexplore.ieee.org ieeexplore.ieee.org GitHub: TSummersLab/polgrad-multinoise github.com

Tags

  • Gradient methods

  • Gradient descent

  • Natural gradient

  • Reinforcement learning

  • Policy gradient

  • Linear systems

  • Uncertain systems

  • Optimal control

  • Stochastic systems

  • Multiplicative noise

  • Non-convex

  • Dynamics

  • Global convergence

  • Linear quadratic regulator