Extending Gaze and Pinch with Thumb-to-Finger Microgestures for Cross-Context Interaction

Gaze+Pinch is a simple and effective model for XR interaction, using gaze for pointing and pinch for selection. Yet, for many tasks, its expressivity is lacking. To address this problem, we extend the pinch gesture with additional thumb to finger microgestures to leverage the expressivity afforded by the hand. Rather than designing a gesture set that tackles an individual problem, this project’s approach is to design a system-level gesture set that maps gestures to application-specific functionality, depending on the application in focus. Utilising a system-level gesture set aims to provide interactions that are learnable and discoverable across diverse contexts, while application-specific mappings aim to provide task-specific utility, while leaving the original Gaze+Pinch interaction untouched. To support users across contexts changes, we employed two established desktop metaphors that describe how gestures will behave across contexts (the buttons and scroll wheel of a mouse, and function-key shortcuts on a keyboard). In a 16 participant user study across two practical tasks (travel planning and spreadsheeting), we compared Gaze+Pinch against Gaze+Microgestures (a condition which restricted participants to the extended gesture set) and Combined (where participants were able to freely choose between Gaze+Pinch and Gaze+Microgesture interactions). Results from the study revealed the viability of our approach, with the Combined offering the fastest, most preferred, and most satisfying performance. Participants engaged with the extended gesture set in the combined condition, but preference fell when restricted to the additional gestures in Gaze+Microgestures. This, in conjunction with qualitative feedback, underscores the importance of user agency and customisability, especially for gestural input, where differences in hand dimensions, finger dexterity, and personality can result in significant impacts on user experience. These findings demonstrate that optional microgestures can enhance expressivity without compromising the usability of the original interaction, pointing towards a design strategy that enables more expressive and flexible gaze-based input systems.