Response dynamics in macaque ventral stream recapitulate the visual hierarchy

Chronic recording in macaque inferotemporal cortex (IT) revealed a progression of feature selectivity over the duration of the response, with different images evoking strongest responses in different time bins. Neurons responded briskly and transiently to early best images and gave slower and more sustained responses to later best images. Correlations between IT selectivity and deep neural network (DNN) activations in early layers were faster and more transient than correlations with deeper DNN layers, suggesting a hierarchical representation unfolding. Analysis of best images in different time bins showed that early best images were simpler, or made up of fewer parts, than later best images. This simple-to-complex selectivity dynamic is consistent with, but more universal than, previously reported dynamics of global-to-local or category-to-exemplar selectivity. In parallel with the simple-to-complex selectivity progression, the image selectivity and the population activity became sparser during the response, consistent with the hypothesis of a canonical cortical microcircuit that pools inputs and incorporates neighboring-neuron inhibition. We propose that this temporal cascade in IT recapitulates the multiple antecedent stages in the visual hierarchy, which also show a simple-to-complex progression of feature selectivity.