Distant Mid-Air Interaction via Accurate Gaze and Hand Posture Estimation with an Ultrafast Pan-Tilt Camera

We propose a distant mid-air interaction system that uses an ultrafast pan-tilt camera to enable accurate selection and manipulation of objects across the entire large display, without large arm movements. By simultaneously capturing high-resolution images of the user's face and both hands using the ultrafast pan-tilt camera, we achieve the same level of gaze and hand posture estimation accuracy at 5 m away from the camera as that obtained at close range. In this system, the screen is divided into multiple regions, and one is selected as the region of interest (ROI) based on the gaze direction. This allows the user to perform interactions within the ROI with only slight hand movements. Our demonstration showed that users can perform multi-touch interactions across the entire large display while maintaining a natural posture with their hands in front of their chest.