Visual focus of attention in adaptive language acquisition

The authors present a study of an adaptive language acquisition system that has multisensory input as opposed to just a message input. They describe and evaluate a device that acquires language through interaction with an environment that provides both keyboard and visual input. In particular, the machine action is to focus its attention, by directing its eyeball toward one of many blocks of different colors and shapes, in response to a message such as 'look at the red square'. The attention focus is controlled by minimizing a time-varying potential function that correlates the message and visual input. This correlation is factored through color and shape sensory primitive subnetworks in an information-theoretic connectionist network, allowing the machine to generalize between different objects having the same color or shape. The system runs in a conversational mode where the user can provide clarifying messages and error feedback until the system responds correctly. During the course of performing its task, a vocabulary of 431 words was acquired from 11 users in over 1000 unconstrained natural language conversations. The average number of inputs for the machine to respond correctly with only 1.4 sentences, and it retained 98% of what it was taught.>