Answers are grounded in this paper and cite their sources.
Shows effective spoken interaction learning in multimodal environments, indicating practical applications for training agents.
We propose Spolacq-GDS, a spoken dialogue simulator designed to support spoken interaction learning in multimodal environments. Dialogue scenarios are flexibly defined using human-readable finite-state automata, in which each state represents a dialogue scene and triggers dynamic generation of speech and visual context via pretrained generative models. This enables richly interactive simulations without requiring task-specific datasets. While large language models like ChatGPT are often fine-tuned using reinforcement learning from human feedback (RLHF), rewards are typically assigned to isolated, single-turn outputs or single-step actions. As a result, such models lack interactive learning loops with the environment and user state, limiting their ability to acquire goal-directed behavior through experience. Simulators such as EmoUS enhance user realism through emotion modeling but remain restricted to text-only interactions and fixed task flows. Spolacq-GDS addresses these limitations by providing a data-free, customizable spoken-dialogue simulation environment implemented within the Gymnasium reinforcement learning framework. It supports integration with both reinforcement learning and self-supervised learning agents. We demonstrate that previously studied spoken language acquisition tasks and newly designed multimodal scenarios can be implemented using only text-based scenario descriptions, making Spolacq-GDS a practical platform for exploring motivation-driven spoken interaction in richly simulated environments.
No takes yet. Share an insight, caveat, or question.
Awashima et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: