Transformer-Based Recognition of Fine-Grained Assembly Actions from Skeleton Data

Human Action Recognition (HAR) plays a crucial role in numerous application domains. In industrial environments, this task becomes particularly challenging. Operators are required to perform complex tasks in close proximity to machinery, collaborate with cobots, and comply with stringent safety and well-being requirements. In such scenarios, distinguishing between actions that are often very similar is an extremely complex problem that is attracting increasing attention within the scientific community. Traditional image-based and sensor-based approaches face limitations such as sensitivity to environmental conditions or user discomfort. The recent availability of low-cost RGB-D cameras has enabled the use of 3D skeleton data, offering a privacy-preserving and robust alternative for action recognition. In this study, we use a Transformer-based approach to recognize complex assembly actions using skeleton data from the HA4M dataset, which provides multimodal data on very challenging assembly actions that exhibit fine-grained movements and strong intra-action similarities. We investigate various input-tensor configurations of raw skeletal joint coordinates and assess their impact on spatial and temporal representation learning in attention-based models, an aspect that is largely underexplored in the literature. Extensive experiments compare performance with different embedding dimensions, with and without positional encodings, while analyzing the influence of architectural hyperparameters. Our findings highlight the importance of input structure in Transformer-based HAR and demonstrate that our best-performing configuration achieves state-of-the-art performance on a challenging industrial HAR task.