Answers are grounded in this paper and cite their sources.
Language-conditioned tabletop manipulation requires identifying the object to be grasped and the spatial region for its placement relative to a reference object. This study presents an end-to-end model that predicts grasp position, grasp angle, and placement position jointly from an RGB image and a relational instruction. Role-specific attention maps weight a shared visual feature grid separately for grasping and placement, preserving spatial structure for action prediction. The maps are conditioned on the instruction through low-rank bilinear fusion, and the resulting model predicts operation parameters directly at inference. Existing scene-instruction pairs from the previous study are re-annotated with two position maps and grasp-angle classes to supervise the three output branches jointly. The model reaches full-pipeline success rates of 94.70% across 2000 simulated trials and 85.50% across 400 real-robot trials. In simulation and on the real robot, respectively, these rates exceed CLIPort-adapted by 4.15 and 4.25 percentage points and the modular baseline by 7.35 and 6.75 percentage points. Separate attention improves simulated full-pipeline success by 10.55 percentage points over shared attention; domain randomization improves real-robot success by 11.00 percentage points. Paraphrase and held-out-combination tests reveal remaining sensitivity to linguistic and compositional variation. These results indicate that role-specific attention supports joint learning of instruction-conditioned grasp and placement parameters in planar manipulation.
No takes yet. Share an insight, caveat, or question.
Xiao et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: