The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns concepts, words, and semantic parsing of sentences without explicit on any of them; instead, our model learns by simply looking at and reading paired questions and answers. Our model builds an-based scene representation and translates sentences into executable, programs. To bridge the learning of two modules, we use a-symbolic reasoning module that executes these programs on the latent representation. Analogical to human concept learning, the perception learns visual concepts based on the language description of the object referred to. Meanwhile, the learned visual concepts facilitate learning words and parsing new sentences. We use curriculum learning to guide the over the large compositional space of images and language. Extensive demonstrate the accuracy and efficiency of our model on learning concepts, word representations, and semantic parsing of sentences. , our method allows easy generalization to new object attributes, , language concepts, scenes and questions, and even new program. It also empowers applications including visual question answering and image-text retrieval.