PAC-Bayesian Soft Actor-Critic Learning

Research output: Contribution to journalConference articleResearchpeer-review

Abstract

Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably against the state-of-the-art SAC implementation on multiple classical control and locomotion tasks in terms of both sample efficiency and regret.

Original languageEnglish
Book seriesProceedings of Machine Learning Research
Volume253
Pages (from-to)127-145
ISSN2640-3498
Publication statusPublished - 2024
Event6th Symposium on Advances in Approximate Bayesian Inference, AABI 2024 - Vienna, Austria
Duration: 21. Jul 2024 → …

Conference

Conference6th Symposium on Advances in Approximate Bayesian Inference, AABI 2024
Country/TerritoryAustria
CityVienna
Period21/07/2024 → …

Fingerprint

Dive into the research topics of 'PAC-Bayesian Soft Actor-Critic Learning'. Together they form a unique fingerprint.

Cite this