ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by People Who Are Blind

Searching for objects in unfamiliar scenarios is a fundamental challenge for people with blindness, requiring target specification, detection, and intent-specific details, such as navigating toward the object and understanding its surroundings. However, existing description- and detection-based assistive technologies fail to address this multifaceted nature of interactive object search. We present ObjectFinder, a wearable prototype with smart glasses (RGB-D sensing), a bone conduction headset, and an edge computing unit for interactive object search by people with blindness. Users query target objects with flexible wording; once detected, ObjectFinder provides real-time egocentric localization information and lets usersinitiate different branches to gather intent-specific details. ObjectFinder integrates open-vocabulary object detection with a multimodal large language model into an intent-driven pipeline, developed iteratively with a blind co-designer. In a study with eight participants with blindness, ObjectFinder’s flexible target specification, egocentric and allocentric cues, and targeted context around the target were valued, with lower NASA-TLX task load than the baseline.