VISUALHINTS:A Visual-Lingual Environment for Multimodal Reinforcement Learning

We present VisualHints, a novel environment for multimodal reinforcement (RL) involving text-based interactions along with visual hints(obtained from the environment). Real-life problems often demand that agents with the environment using both natural language information and perception towards solving a goal. However, most traditional RL either solve pure vision-based tasks like Atari games or-based robotic manipulation; or entirely use natural language as a mode of, like Text-based games and dialog systems. In this work, we aim to this gap and unify these two approaches in a single environment for RL. We introduce an extension of the TextWorld cooking environment the addition of visual clues interspersed throughout the environment. The is to force an RL agent to use both text and visual features to predict language action commands for solving the final task of cooking a meal. enable variations and difficulties in our environment to emulate various real-world scenarios. We present a baseline multimodal agent for such problems using CNN-based feature extraction from visual hints and for textual feature extraction. We believe that our proposed-lingual environment will facilitate novel problem settings for the RL.