Hierarchical optimization predicts plasticity in the macaque inferior temporal cortex following object training

Abstract How does the primate brain coordinate plasticity when learning to discriminate new objects? We measured consequences of object learning on inferior temporal (IT) cortex, a key waypoint supporting object recognition in the ventral visual stream, in male macaques. Neural activity in task-trained monkeys’ IT showed increased object selectivity, enhanced linear separability across objects, and more object-invariant representations compared to task-naïve monkeys. To model these differences, we developed a computational framework using anatomically-mapped artificial neural network (ANN) models of the ventral stream with various learning algorithms. Simulations revealed that gradient-based, performance-optimizing updates of ANN internal representations accurately approximated observed IT cortex changes. These models predict novel training-induced phenomena in the IT cortex, including changes independent of object identity and IT’s alignment with behavior. This convergence between empirical measurements and model predictions suggests ventral stream plasticity follows task optimization principles well-approximated by gradient descent, enabling accurate predictions about visual plasticity and generalization to test images.