The Power of Learned Locally Linear Models for Nonlinear Policy Optimization

Authors: Daniel Pfrommer, Max Simchowitz, Tyler Westenbroek, Nikolai Matni, Stephen Tu

Published: 2023 (Conference Paper)

Source: International Conference on Machine Learning (ICML)

arXiv: 2305.09619

DOI: 10.5555/3618408.3619563

Summary

Provides theoretical grounding for why locally linear models learned from data combined with model-based trajectory optimization (iLQR) can achieve sample efficiency for control of nonlinear systems with unknown dynamics.

Abstract

A common pipeline in learning-based control is to iteratively estimate a model of system dynamics, and apply a trajectory optimization algorithm - e.g. iLQR - on the learned model to minimize a target cost. This paper conducts a rigorous analysis of a simplified variant of this strategy for general nonlinear systems. We analyze an algorithm which iterates between estimating local linear models of nonlinear system dynamics and performing iLQR-like policy updates. We demonstrate that this algorithm attains sample complexity polynomial in relevant problem parameters, and, by synthesizing locally stabilizing gains, overcomes exponential dependence in problem horizon. Experimental results validate the performance of our algorithm, and compare to natural deep-learning baselines.

Links

Primary

Paper arxiv.org

Standard

arXiv Abstract 2305.09619 arXiv PDF 2305.09619 arXiv HTML 2305.09619 DOI 10.5555/3618408.3619563

Alternate

proceedings.mlr.press PDF proceedings.mlr.press dl.acm.org dl.acm.org nikolaimatni.github.io nikolaimatni.github.io

Tags

• Reinforcement learning

• Trajectory optimization

• Nonlinear control

• Linear models

• iLQR

• Data-driven control

• Model-based