Object Recognition Using Deep Learning‐Powered Glove With Flex and MPU Sensors Fusion

The integration of multimodal sensor data is critical for developing intelligent systems capable of extracting complex, insightful information. We introduce a robust platform for high‐fidelity and real‐time hand posture replication, as well as object classification, centered on a novel tactile‐sensing glove. The glove design incorporates 10 flex sensors spanning the PIP and MCP joints, complemented by two MPU6050 IMUs strategically mounted on the thumb’s TM and the hand’s dorsal CMC bones for comprehensive motion capture. We leverage this rich sensory data to drive a digital twin—constructed in Blender and visualized in the Unity3D engine—achieving precise real‐time hand posture reproduction, necessary for visual feedback and sensor anomaly detection during sensor calibration and data collection by the operator. Furthermore, we evaluate the glove’s classification capability on a custom dataset of 15 objects. Through a grid search optimization, we trained One‐dimensional Convolutional Neural Network (CNN‐1D), Long‐Short Term Memory (LSTM), and Temporal Convolutional Network (TCN) architectures. The models achieved classification testing accuracies confidence intervals of 95.40%, 96.50%, 92.55%, 93.84%, and 93.47%, 94.76%, respectively, validating the high‐performance and utility of our multimodal sensing approach.