Stochastic Differential Dynamic Programming
Abstract¶
Although there has been a significant amount of work in the area of stochastic optimal control theory towards the development of new algorithms, the problem of how to control a stochastic nonlinear system remains an open research topic. Recent iterative linear quadratic optimal control methods iLQG handle control and state multiplicative noise while they are derived based on first order approximation of dynamics. On the other hand, methods such as Differential Dynamic Programming expand the dynamics up to the second order but so far they can handle nonlinear systems with additive noise. In this work we present a generalization of the classic Differential Dynamic Programming algorithm. We assume the existence of state and control multiplicative process noise, and proceed to derive the second-order expansion of the cost-to-go. We find the correction terms that arise from the stochastic assumption. Despite having quartic and cubic terms in the initial expression, we show that these vanish, leaving us with the same quadratic structure as standard DDP.
Summary¶
Generalizes differential dynamic programming to nonlinear stochastic systems with state- and control-dependent process noise. The derivation shows how the stochastic correction terms collapse back to a quadratic local dynamic-programming structure, preserving the practical backward/forward sweep form while extending DDP beyond deterministic or purely additive-noise settings.