Simple Statistical Gradient-following Algorithms for Connectionist Reinforcement Learning

Abstract¶

This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units. These algorithms, called REINFORCE algorithms, are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reinforcement tasks, and they do this without explicitly computing gradient estimates or even storing information from which such estimates could be computed. Specific examples of such algorithms are presented, some of which bear a close relationship to certain existing algorithms while others are novel but potentially interesting in their own right. Also given are results that show how such algorithms can be naturally integrated with backpropagation. We close with a brief discussion of a number of additional issues surrounding the use of such algorithms, including what is known about their limiting behaviors as well as further considerations that might be used to help develop similar but potentially more powerful reinforcement learning algorithms.

Summary¶

Introduces the REINFORCE family of algorithms, showing that simple stochastic weight updates can follow the gradient of expected reinforcement without explicitly estimating or storing that gradient. The paper is a foundational policy-gradient reference and connects reinforcement learning in stochastic neural networks with backpropagation.

Links¶

Tags¶

Similar Papers¶