Retaining Naturally Fragmented Experience for Offline Reinforcement Learning: Evidence From Real Operational Logs

Offline reinforcement learning (RL) learns policies from fixed datasets, but real operational logs often contain missing or invalid measurements. Because offline RL requires multiple variables to be valid simultaneously, these imperfections can divide continuous records into disconnected fragments. This study preserves the naturally imperfect structure of real operational logs rather than reconstructing missing observations, which is a more faithful way to represent the recorded experience. The resulting natural fragments are retained as complete units and progressively added to investigate how much valid experience beyond the longest contiguous fragment should be used for offline RL. Commercial heating control telemetry collected over 4 months yields valid offline RL transitions across 28 naturally occurring fragments, with no transitions constructed across the gaps. IQL and CQL are trained with matched random seeds as natural fragments are added in descending order of duration, with retention measured as a percentage of eligible transitions. Policies are evaluated using ARX and XGBoost dynamics models, with each evaluated fragment excluded from the corresponding model fitting data. Under both evaluators, the highest mean evaluated values for IQL and CQL occur at intermediate retention rather than at either the longest fragment baseline or full retention. Increasing retention also brings policy behavior closer to the support of the logged data, although the improvement diminishes at higher retention levels. These findings indicate that retaining more valid experience from naturally fragmented operational logs does not necessarily improve evaluated policy performance, highlighting the importance of examining retention decisions without reconstructing the original gaps.