Dyna-ESN: Efficient Deep Reinforcement Learning for Partially Observable Dynamic Spectrum Access

This paper focuses on advancing reinforcement learning for challenging environments characterized by partial observability and non-stationarity, such as dynamic spectrum access (DSA). In the literature, the Deep Recurrent Q-Network was introduced to capitalize on the inherent temporal correlations present in DSA. Nevertheless, its practicality is still questionable due to sample inefficiency and slow convergence. We introduce Dyna-ESN, leveraging both model-based and model-free methods by employing Reservoir Computing for generative modeling. Specifically, we utilize Echo State Networks (ESNs) to synthesize samples for enhancing the sample efficiency of a model-free Deep Echo State Q-network, enabling effective operation of agent given limited genuine relevant samples obtained through interaction with environment. To mitigate potential adverse effects of synthetic samples, an evaluation algorithm guides the sample selection process, ensuring reliability. A sample augmentation technique is also introduced to allow agents to collect adequate samples despite controlling the sensing rate and duration of secondary transmissions. Our analysis explores trade-offs between data evaluation and sample efficiency, as well as the bias-variance trade-off of the model, identifying optimal design parameters. Evaluating the performance of Dyna-ESN in DSA scenarios demonstrates its performance benefits over existing methods, paving the way for more efficient and effective techniques in complex dynamic environments.