Exploration of Pointing Forms in Customer-Shopkeeper Interactions

To investigate the use of different pointing forms in service scenarios, we collected the ShopPoint 1 dataset, a skeleton-based dataset of pointing gestures from customer-shopkeeper interactions in a camera shop scenario. 13 participants took part in the data collection, including 3 shopkeepers with real-world customer service experience and 10 customers. We recorded 61 one-to-one role-played interactions. Coders annotated pointing gestures from videos of these interactions, emphasizing pointing arm forms (straight-arm, bent-arm, and hand-only pointing) and hand forms (index-finger and open-hand pointing). This annotation process resulted in 2959 pointing gestures. We conducted statistical analysis on the annotated data. The analysis revealed that bent-arm pointing was used more frequently than other arm forms. Straight-arm pointing was used more for far targets than for close targets, and hand-only was used more for close targets. Shopkeepers used bent-arm pointing more frequently than customers when referring to far targets. To evaluate the recognition of these pointing gestures, we tested several existing Skeleton-based Action Recognition (SAR) methods on the dataset. The highest accuracy was achieved at 72.51% by using transfer learning (i.e., pre-training and fine-tuning). This evaluation indicates that though transfer learning aids performance, recognizing pointing with diverse forms remains challenging.