Learning what you hold: A personalized object registration and recognition framework via handheld interaction

This is the authors' abstract. We don't add key points for this paper.

Personalized object recognition is a critical capability for home service robots to understand and execute user-specific requests within idiosyncratic domestic environments. However, existing approaches to object recognition predominantly treat targets as generic category instances, thereby failing to capture the subjective semantic attributes assigned by humans—such as ownership, emotional value, or specific functional roles. We propose an interactive personalized object learning and recognition framework that enables robots to acquire user-specific object knowledge through natural handheld demonstrations and verbal descriptions. The framework constructs multimodal object templates comprising fine-grained visual embeddings extracted via Mask-Guided Feature Aggregation (MGFA) with DINOv3, and semantic attribute embeddings encoded through SentenceTransformer. This dual-channel representation bridges visual appearance with personalized meaning. During the recognition phase, a hybrid retrieval mechanism first utilizes natural language queries to perform semantic filtering within the template database. Target localization is then achieved by matching scene-level proposals against the visual features retrieved from the identified templates, ensuring precise instance identification in cluttered environments. We further integrate the framework with a large language model agent, enabling end-to-end conversational interaction for both object registration and retrieval. Experiments on the iCubWorld dataset demonstrate that our approach effectively handles both explicit visual queries and implicit functional queries, achieving robust few-shot recognition with minimal user effort. The proposed framework provides a practical pathway toward deploying personalized visual knowledge systems in real-world home environments.