Abstract In VR environment, the method of text-input by creating virtual surrogates for the keyboard and hands not only overcomes spatial constraints but also enhances users' sense of immersion. However, the input efficiency of virtual keyboards based on virtual surrogates is hindered by the accuracy of hand tracking. Especially in small vision-based tracking devices, hand occlusion leads to a decline in input efficiency. To enhance the input efficiency of virtual keyboards, we propose an optimization approach for virtual keyboards based on multimodal character prediction. This method integrates historical text and hand - motion trajectory information to build a multimodal character-prediction model. Based on this, a probability fusion strategy grounded in dynamic weights is designed. Probability serves as the driving force to update the key shapes, guiding users to input text efficiently. Compared with virtual keyboards in standard form, this method increases the text-input speed of virtual keyboards from 10.99 WPM to 14.08 WPM, and reduces the total error rate from 0.11 to 0.06.
Chen et al. (Wed,) studied this question.