Bridging data and adaptation in agent-based modelling: A comparative reinforcement learning approach to urban food retail choice

Agent-based models (ABMs) are powerful tools for simulating complex socioeconomic systems. Reinforcement learning (RL), especially online RL, is increasingly used to model agent behaviour, allowing agents to explore and learn optimal actions from the model environment. However, standard online RL approaches learn from scratch and often fail to incorporate valuable real-world data. Further, while purely offline RL can use this data, it does not adapt to new dynamics. To address this limitation, this study develops a hybrid offline-to-online (O2O) RL model and compares it with online and offline learning in an ABM of customer store choice. This allows us to address a critical gap, as most research on customer grocery shopping behaviour is static and fails to capture dynamic interactions between customers and food retailers. The result shows that the hybrid mode achieves the highest performance in customer rewards, successfully leveraging historical data for a strong start, while using online learning to adapt and optimize. The hybrid method also produced a distinct spatial pattern of competition, retaining a realistic baseline from its offline training while also showing emergent areas of intensified competition. Furthermore, the study assesses how well each model replicates observed customer behaviours, finding that pre-training with offline data is crucial for capturing specific shopping patterns, for example, specific temporal shopping patterns. Notably, store agents converged on a stable pricing strategy regardless of the customer learning model. This research highlights that the choice of RL training approach critically shapes the outcomes of ABM simulation. The O2O approach represents a powerful and adaptable framework that balances empirical data with learning ability, allowing its application to diverse socioeconomic systems by defining custom state, action, and reward structures.