Langmark: annotations for scenes with semantic inconsistencies connecting distributional semantic models to vision science

We introduce Langmark: object annotations (outlines and labels) for 124 scenes containing violations of environmental regularities (e.g. semantic inconsistencies, i.e., objects that are unexpected in a given context, like a hammer in a kitchen). Langmark enables analyzing these scenes, which are popular stimuli in scene-perception research, using distributional semantic models (DSMs) – powerful tools capturing human linguistic knowledge. To demonstrate the potential of this approach, we consider scenes with the aforementioned semantic inconsistencies. The dominant conceptualization of object-scene inconsistency is the dichotomy between consistent and inconsistent objects. Given that DSMs capture relationships between object names on a continuous scale, they allow for a change leading to new, testable hypotheses, namely, conceptualizing the object-context consistency as a continuous variable. However, DSMs operate on words, so applying them to image analysis is challenging because it requires objects shown in an image to be labelled (named) and often outlined. Langmark addresses this challenge. It enables analyzing data from experiments using scenes with inconsistencies (including semantic inconsistencies) that we annotated with DSMs. We showcase this approach by demonstrating that an off-the-shelf DSM is sensitive to semantic inconsistencies. Additionally, we supplemented Langmark with human ratings, which are an alternative way of quantifying object-context relationships. We compared them to the outputs of a DSM and thereby identified challenges associated with both methods. For example, the ratings changed depending on the details of the procedure used for their collection. These insights, coupled with the Langmark annotations, should enable a wider adoption of DSMs within vision science.