Enabling Real-World Assistive Agents: From Live Vision to Proactive Context-Aware Information Delivery

Interacting with the real world is a fundamental part of daily life, yet it remains challenging for individuals who are blind or visually impaired (BVI). It demands live, contextual understanding of dynamic environments, along with interactive, multimodal communication to fulfill their sensory and cognitive needs. To address this, my dissertation develops assistive AI systems and frameworks that observe the real world through multimodal sensing, reason about essential information in response to user contexts, and deliver human-like verbal communication to support real-world understanding. First, I explored the design insights of assistive AI agents by investigating how BVI users interact with human-like video AI systems across diverse real-world contexts. Second, with the identified insights, such as a lack of proactivity, I developed a mobile application that analyzes live camera feeds to generate real-time visual descriptions aligned with user goals, delivering them in harmony with the audio environment. Lastly, I extend it with a set of human-like capabilities, such as memory, spatial understanding, and the ability to infer intent from natural interactions, to act as a long-term assistive companion. Ultimately, my dissertation advances a paradigm shift from digital agents to real-world assistive agents that enhance the independence and agency of BVI individuals.