What type of study is this?

This is a Quantitative Study study.

September 19, 2025

Using Reward Machines for Offline Reinforcement Learning With Non-Markovian Reward Functions

Key Points

C-QRM outperforms the baseline method with fewer training steps while learning optimal policies.
The method utilizes non-Markovian reward functions to provide a more realistic reward structure in reinforcement learning.
C-QRM aims to learn a conservative Q-function, helping mitigate overestimation bias associated with reward calculations.
Improved sampling efficiency is a significant benefit of using the offline dataset in the C-QRM approach.

Abstract

We investigate the problem of offline reinforcement learning using non-Markovian reward functions, which allows for the incorporation of more realistic and intricate reward structures in the learning process. Offline reinforcement learning has shown promising potential in learning optimal policies when the agent has access to previously collected static datasets. Reward machines offer a way to encode the high-level structure of non-Markovian reward functions. We introduce C-QRM, an offline reinforcement learning approach that employs non-Markovian reward functions specified as reward machines to accomplish complex tasks and learn an optimal policy more efficiently by utilizing the offline dataset. Our objective is to learn a conservative Q-function that decomposes complex high-level reward machines and whose expected value of a policy under this Q-function provides a lower bound to its actual value. C-QRM learns these lower-bounded Q-values, mitigating overestimation bias and improving sampling efficiency. We evaluate the performance of the proposed C-QRM algorithm by comparing it to QRM as a baseline method. The results indicate that C-QRM outperforms QRM with fewer training steps and benefits from the offline dataset.

Mark Helpful

Bookmark

Relay