Decoding and reconstructing color from responses in human visual cortex

Introduction How is color represented by the distributed spatial patterns of activity in visual cortex? We determined the accuracy with which stimulus color could be decoded and reconstructed from fMRI measurements of activity in visual cortex. Methods We used eight isoluminant colors, equally spaced in the CIE L*a*b color space, surrounding and equidistant from a central gray point. Each stimulus was a concentric grating modulating from central gray to one of the eight colors, presented in pseudo-randomized order (1.5 s duration, 3–6 s ISI). fMRI responses were analyzed with multivariate techniques: pattern classification and principal component analysis (PCA). Results Stimulus color was accurately decoded from spatially distributed patterns of responses within areas V1, V2, V3, V4 and VO, but not LO1, LO2, V3A/B or MT. In a complementary analysis, we used PCA to reconstruct a color space from activity in each visual area. PCA extracted the covariation between voxels' responses to the different colors. In V4, the first two principal components (the main source of variation) of the responses constituted a vector space that resembled perceptual color space, with similar colors evoking the most similar responses. By contrast, although decoding was more accurate in V1 than V4, the PCA of V1 activity did not reveal a similar progression. The mean responses, averaged across voxels in each visual area, were the same for all colors. Although voxels in each visual area exhibited different color preferences, no consistent spatial organization or topography was found. Conclusion Each stimulus color was associated with a unique spatially distributed pattern of activity within early visual areas. We assume that this reflects the color-selectivity of the neurons in those areas. If so, activity in V4 (but not V1) appears to depend primarily on the responses of color-selective neurons with color tuning that reflects perceptual color space.