Answers are grounded in this paper and cite their sources.
Existing research has demonstrated the benefits and envisioned broader applications of using extended reality (XR) head-mounted displays (HMDs) to create virtual display extensions for personal computers (PCs), offering a more flexible and cost-effective alternative to traditional physical monitors. This combined setup, with inputs from both sources available, provides users with more options for interaction. In this work, we aim to understand the advantages and disadvantages of these inputs and their interaction techniques in the fundamental window placement tasks in virtual displays enabled by the combined use of a PC and an HMD. With their standard inputs, we identified five potential hand-based window placement techniques: Mouse Cursor, Pinch Drag, Keyboard Shortcut, Touchpad Gesture, and Wrist Bending. We conducted a mixed-design user study (N = 40) to evaluate user performance and experience with these window placement techniques in two display setups: a fully virtual setup where the HMD displays all windows, and a hybrid setup involving a laptop screen in addition to the HMD’s virtual windows. All five techniques were evaluated in a structured layout task, while Mouse Cursor and Pinch Drag were additionally compared in a freeform placement task. Across both setups, Mouse Cursor was faster than Pinch Drag, while Keyboard Shortcut and Touchpad Gesture were faster and produced fewer failed trials than Wrist Bending. Laptop-based techniques also received higher usability ratings and lower perceived workload ratings than mid-air techniques. Nevertheless, some participants preferred mid-air techniques, valuing their naturalness. Our empirical findings can guide the future design of cross-device and cross-reality interactions between XR HMDs and PCs.
No takes yet. Share an insight, caveat, or question.
Shi et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: