Vision-Based Multimodal Pressure Estimation for Robotic Gripper Manipulators

Precise perception and control of gripper pressure are essential for dexterous robotic manipulation. Conventional methods relying on force/torque or tactile sensors increase system complexity, incur higher costs, and offer limited adaptability. Vision-based approaches provide a promising non-contact alternative, but existing methods typically use single RGB images, restricting exploitable information. To overcome these limitations, we propose a hierarchical multimodal representation alignment and fusion framework. The framework integrates complementary cues from RGB images, segmentation masks, depth maps, and textual prompts describing gripper actions. A modality-aware feature interaction mechanism adaptively emphasizes the most informative modalities, and pressure-relevant features are consolidated across spatial scales and hierarchical levels, capturing both fine-grained details and high-level semantics. This design enables robust modeling of subtle visual–force correlations, fully leveraging the complementary potential of multimodal perception. Experiments on a publicly available benchmark demonstrate that the framework achieves highly competitive and stable performance. For tendon-driven grippers, our approach improves temporal accuracy to 96.57% and reduces the mean absolute error to 4.33 Pa; for soft grippers, it achieves 95.52% temporal accuracy with robust volumetric consistency. These results validate the statistical stability, effectiveness and robustness of the framework for non-contact pressure sensing and establish a solid foundation for dexterous manipulation and broader robotic applications.