Learning to plan and execute subgoals with a hierarchical policy for long-horizon instruction following tasks

This is the authors' abstract. We don't add key points for this paper.

With the increasing significance of autonomous agents capable of executing long-horizon tasks through natural language instructions, the complexity of planning and executing a series of interrelated subgoals has been brought to the forefront. Such long-horizon tasks demand accurate and adaptable reasoning based on instructive languages. Prior studies plan subgoals primarily based on the language instructions, harming the adaptability of the plan and leading to discrepancies between the plan and real-time status. To alleviate this problem, we introduce EOPBiSE (Episodic On-the-Fly Planner and Bifurcate Subgoal Executor), a novel hierarchical policy framework. Its planner leverages both linguistic instructions and holistic visual observations from a task’s entire episode history, ensuring adaptive subgoal predictions. Concurrently, it minimizes the discrepancy between plan and status by bifurcating subgoal information in the executor for effective plan execution. Evaluation on the ALFRED benchmark, which includes complex household tasks based on egocentric visions, shows that EOPBiSE can effectively handle a wide range of long-horizon tasks and exhibits generalization capabilities in unseen scenarios.