An Analytical Update Rule for General Policy Optimization
Document Type
Conference Proceeding
Date of Original Version
1-1-2022
Abstract
We present an analytical policy update rule that is independent of parametric function approximators. The policy update rule is suitable for optimizing general stochastic policies and has a monotonic improvement guarantee. It is derived from a closed-form solution to trust-region optimization using calculus of variation, following a new theoretical result that tightens existing bounds for policy improvement using trust-region methods. The update rule builds a connection between policy search methods and value function methods. Moreover, off-policy reinforcement learning algorithms can be derived from the update rule since it does not need to compute integration over on-policy states. In addition, the update rule extends immediately to cooperative multi-agent systems when policy updates are performed by one agent at a time.
Publication Title, e.g., Journal
Proceedings of Machine Learning Research
Volume
162
Citation/Publisher Attribution
Li, Hepeng, Nicholas Clavette, and Haibo He. "An Analytical Update Rule for General Policy Optimization." Proceedings of Machine Learning Research 162, (2022). https://digitalcommons.uri.edu/ele_facpubs/1744