Skip to content

A Large Deviations Perspective on Policy Gradient Algorithms

Authors
Wouter Jongeneel, Daniel Kuhn, Mengmeng Li
Year
2023
Preprint
Source
arXiv
Identifiers
arXiv 2311.07411

Abstract

Motivated by policy gradient methods in the context of reinforcement learning, we identify a large deviation rate function for the iterates generated by stochastic gradient descent for possibly non-convex objectives satisfying a Polyak-Łojasiewicz condition. Leveraging the contraction principle from large deviations theory, we illustrate the potential of this result by showing how convergence properties of policy gradient with a softmax parametrization and an entropy regularized objective can be naturally extended to a wide spectrum of other policy parametrizations.

Knowledge Base

Map Tree Search

External

Document arXiv Abstract arXiv PDF arXiv HTML Google Scholar OpenAlex

Tags

Gradient descent Stochastic gradients Reinforcement learning Learning Policy gradients Stochastic gradient descent Rate function Gradient method Large deviations theory

Similar Papers