Talking about other people: an endless range of possibilities

Image description datasets, such as Flickr30K and MS COCO, show a high degree of variation in the ways that crowdworkers talk about the world. Although this gives us a rich and diverse collection of data to work with, it also introduces uncertainty about how the world should be described. This paper shows the extent of this uncertainty in the PEOPLE domain. We present a taxonomy of different ways to talk about other people. This taxonomy serves as a reference point to think about how other people should be described, and can be used to classify and compute statistics about labels applied to people.