Understanding and following directions provided by humans can enable robots navigate effectively in unknown situations. We present FollowNet, an-to-end differentiable neural architecture for learning multi-modal policies. FollowNet maps natural language instructions as well as and depth inputs to locomotion primitives. FollowNet processes using an attention mechanism conditioned on its visual and depth to focus on the relevant parts of the command while performing the task. Deep reinforcement learning (RL) a sparse reward learns the state representation, the attention function, and control. We evaluate our agent on a dataset of complex natural language that guide the agent through a rich and realistic dataset of homes. We show that the FollowNet agent learns to execute previously instructions described with a similar vocabulary, and successfully along paths not encountered during training. The agent shows 30% over a baseline model without the attention mechanism, with 52% rate at novel instructions.
No takes yet. Share an insight, caveat, or question.
Shah et al. (2018) studied this question.