Multi-Modal Interactions of Mixed Reality Framework

Mixed Reality (MR) has potential to be used not only in the entertainment industry and the work industry, but also for assistive technology. Mixed Reality is suited for assistive technology because of its utilization of other forms of physical feedback to the human senses besides vision while still making use of visual feedback. We propose a glove-based MR system framework that will use finger and hand movement tracking along with tactile feedback so that the blind and visually-impaired (BVI) can interact tactily with virtual objects. In addition to touch, our proposed framework will include robust interactions through other modalities such as a custom voice assistant and an audio interface as well as visual interfaces that tailor to the visual needs of BVI users. Through the various modalities of interaction in our proposed framework, BVI users can obtain a more detailed sense of virtual objects of any 3D model and their experiences will not be limited by vision. The customizable features and modalities available in our system framework will allow for a more individual experience that can be tailored to the variety of the different needs of the BVI as well as general users.