Poster: Translating Vision into Words: Advancing Object Recognition with Visual-Language Models

Poster: This study focuses on automatically identifying and classifying objects within indoor environments. Traditional methods struggle with this task due to the high cost of manually labeling each object and the inherent ambiguity of written descriptions. To overcome these limitations, we propose a novel instance segmentation approach that utilizes a visual-language model. This system is trained on extensive indoor environment data, including detailed point clouds (3D representations) and RGB images, readily collected by modern smartphone sensors. By eliminating the need for pre-defined labels, the system allows users to search for items using natural language. We evaluate the system's effectiveness through experiments analyzing performance metrics like accuracy and efficiency with public datasets, demonstrating its practicality and potential usefulness.