Unobtrusive in-situ measurement of behavior change by deep metric similarity learning of motion patterns

This paper introduces an unobtrusive in-situ measurement method to measure user behavior changes during arbitrary exposures in XR systems. Here, such behavior changes can be elicited by the avatar’s appearance that the users embody. We present a biometric user model based on deep metric similarity learning, which utilizes high-dimensional embeddings as reference vectors to detect behavioral changes in individual users. Nevertheless, this approach can only inform about behavior change, but not about the psychological processes behind it. We evaluate our model against two alternative approaches: a (non-learned) motion analysis based on central tendencies of movement patterns and subjective post-exposure embodiment questionnaires frequently used in various XR exposures. In a within-subject study, participants performed a fruit collection task while embodying avatars of different body heights (short, actual-height, and tall). Subjective assessments confirmed the effective manipulation of perceived body schema, while the (non-learned) objective analyses of head and hand movements revealed significant differences across conditions. Our similarity learning model, trained on motion data, successfully identified the elicited behavior change for various query and reference data pairings of the avatar conditions. A cross-validation showed the robustness of our approach across different user splits. The approach has several advantages in comparison to existing methods: 1) In-situ measurement without additional user input, 2) generalizable and scalable motion analysis for various use cases, 3) user-specific analysis on the individual level, and 4) future work can use this approach with a trained model, to add users and evaluate their behavior in real-time to study how avatar changes affect behavior. Future work must validate our approach beyond the single scenario studied, assess its generalizability, and evaluate its real-time applicability.