Egocentric Industrial Human–Object Interaction Recognition Based on Hand Interaction Point Modeling

In intelligent manufacturing environments, first-person view imagery has become increasingly important for equipment maintenance, operation monitoring, and intelligent assistance systems.However, frequent hand occlusions, small interaction targets, and complex operational procedures make fine-grained human-object interaction (HOI) recognition challenging.In this paper, we propose an egocentric industrial HOI recognition method based on hand interaction point modeling, which integrates hand segmentation, object detection, a minimum-distance interaction search mechanism, and temporal analysis to establish handobject associations and recognize interaction events.Experiments were conducted on the ENIGMA-51 industrial egocentric dataset and evaluated using the Egocentric HOI Detection task.Experimental results demonstrate that the proposed contour-based interaction point modeling significantly improves the HOI recognition performance, achieving 59.60% average precision for the hand and contact state (AP H.+State), 78.60% mean average precision for the hand and object state (mAP H.+Obj), and 45.63% mean average precision for the hand, object, and contact state (mAP H.+All), outperforming the conventional center-based interaction modeling approach.The results demonstrate that contour-based interaction modeling more accurately captures real hand-object contact relationships and provides more reliable interaction cues for industrial HOI recognition, offering an effective solution for intelligent factory operation comprehension and standard operating procedure analysis.Sensors and Materials, Vol.38, No. 10 (2026) errors can not only reduce operational efficiency and product quality but may also result in equipment damage, personal injury, and even serious industrial accidents.Therefore, the development of intelligent and proactive safety management mechanisms has become an important research topic in the fields of smart manufacturing and industrial safety.(1) With the rapid advancement of wearable sensing technologies, various sensing devices have been introduced into occupational monitoring and safety management applications.For example, smartphones equipped with accelerometers and gyroscopes can be utilized to track human motion patterns and posture changes.(1) Smartwatches can monitor workers' physiological conditions through heart-rate and motion sensors.Furthermore, smart glasses, which provide real-time image acquisition and information visualization capabilities, have emerged as a promising platform for both operational assistance and workplace monitoring.(1) Among these wearable devices, smart glasses are capable of directly capturing first-person-view (FPV) visual information from the user's perspective.(2) Such visual data not only provide a complete record of the operator's observations and actions but also accurately reflect interactions among workers, equipment, tools, and the surrounding environment.Consequently, FPV-based monitoring systems offer significant potential for real-time observation, behavior analysis, and the safety supervision of industrial operations.However, in FPV imagery, accurately characterizing the contact relationship between hands and manipulated objects remains a challenging task owing to frequent hand occlusions, rapid movements, and continuous viewpoint changes.This research area in computer vision is commonly referred to as egocentric or first-person vision.Although early studies have explored FPV-based visual understanding, (3) existing approaches primarily rely on hand keypoint, (4) bounding boxes, (5) or segmentation masks (6) to determine interaction relationships.In industrial operating environments, however, these methods often fail to precisely represent actual contact locations and operator intentions.To address this limitation, we propose an egocentric industrial human-object interaction (HOI) recognition method based on hand interaction point modeling (HIPM), aiming to improve the accuracy of hand-object interaction analysis and our understanding of operation behavior.To facilitate research in industrial FPV applications, several public datasets have been introduced in recent years, including EPIC-KITCHENS, (7) Ego4D, (7) and Assembly101.(7) However, most of these datasets focus on kitchen activities, dailylife scenarios, or assembly tasks, which differ significantly from real-world industrial maintenance environments.Because of safety regulations, privacy concerns, and proprietary information restrictions, acquiring data from actual industrial settings remains difficult.Consequently, many existing studies rely on simulated equipment or simplified environments, limiting their ability to fully capture the characteristics of industrial operations.The ENIGMA-51 dataset (4) was developed to bridge this gap by simulating realistic industrial maintenance scenarios.It contains recordings of participants performing electrical control board maintenance tasks while interacting with equipment such as electric screwdrivers, power supplies, oscilloscopes, and control buttons.In addition, the dataset provides detailed annotations for hands, objects, and interaction relationships, making it an important benchmark for egocentric HOI research in industrial environments.On the basis of these fine-grained annotations, the proposed method is positioned as the perception layer within the overall SOPs