A Humanoid Automation Framework Integrating Task and Object Awareness With Model-based Control for Process-oriented Tasks

Manufacturing and logistics processes often involve settings in which the type, quantity, and sequence of target objects can vary, yet widely used visual servoing techniques may not accommodate such variability. To address this limitation, recent studies have explored end-to-end learning-based VLA (Vision Language Action) models; however, maintaining high success rates and motion precision under changing object placements and task sequences may require demonstration data covering diverse process conditions, thereby increasing data-collection effort. To reduce this burden, this study proposes a hybrid framework for process automation composed of two complementary modules. The interpretation of variable task instructions and object pose estimation are performed by a learning-based task and object awareness module, while motion generation, which requires precision and reliability, is handled by a model-based manipulation module. Based on the estimated pose, kinematic feasibility is evaluated in terms of inverse kinematics, joint limits, and manipulability to determine task-specific motion targets. The model-based manipulation module also provides the required workspace in confined environments through torso-inclusive motion without base movement. The proposed framework was evaluated in two process-simulating tasks, achieving part-level success rates of 94.4% for part sorting and 98.0% for part insertion. Compared with a VLA-based method on the insertion task under the same conditions, it improved part-level success from 89.0% to 98.0% and reduced the average task time from 206.6 s to 173.2 s.