Visuo-tactile physical-state fusion and stage-aware closed-loop control for manipulating containers with unknown internal payloads in resource-recovery scenarios

Robotic grasp-lift-dump manipulation of containers with visually unobservable internal payloads presents a coupled perception-control problem. External RGB-D geometry can localize the container and initialize the grasp, but hidden payload mass and load distribution determine post-contact gripping demand, eccentric moment, slip tendency, and dumping behavior. The relevance of these variables changes across closing, stabilization, lifting, and dumping, creating a need for explicit post-contact physical-state estimation and stage-conditioned feedback. This study introduces a stage-aware visuo-tactile physical-state fusion and closed-loop control approach for this problem. The proposed method organizes pre-contact grasping cues from red-green-blue–depth (RGB-D) data, post-contact bilateral tactile load inference, and raw tactile feedback into grasp-relevant physical states. The tactile inference module estimates the internal payload mass and a one-dimensional projected center-of-mass (COM) offset, while the real-time closed-loop controller relies on an independently computed lateral load-bias proxy, contact confidence, slip risk, and grasp stability. The RGB-D module predicts bounding boxes, yaw angles, and depths through depth-quality-aware gated fusion, transformer-based object modeling, and region-of-interest refinement. These geometric estimates are then used to define the initial task-space grasp reference, specifying three-dimensional position and yaw expressed in the robot base frame. The remaining tool-orientation components are constrained by the predefined gripper approach configuration. The physics-guided bilateral tactile module infers load properties from aligned tactile maps and analytical features, including pressure integration, center of pressure, contact area, and bilateral balance. The stage-aware controller subsequently adjusts the grip command, lateral pose, and dumping motion throughout the approach, contact, closure, stabilization, lifting, and dumping stages. In a 600-trial four-method system-level comparison, the proposed method achieved an 80.7% complete-task success rate, outperforming the RGB-D-only fixed-grip baseline by 46.0 percentage points. It also lowered the slip and drop rates from 38.7 and 18.0% to 7.3 and 2.7%, respectively. These results support the effectiveness of the proposed closed-loop framework under the evaluated payload conditions, with physics-inspired tactile descriptors supplying explicit physical cues for load-state estimation and control.