Challenges in Annotating Gesture-Based Cognitive Status in Human-Robot Collaboration Datasets

For robots to be effective at collaborating with humans, they must be able to effectively communicate about entities in open-world tasks. Existing research on natural language generation and referring expression generation has yet to address how gesture and cognitive status impact how humans or robots decide how to refer to entities, a process known as Referring Form Selection. To address these issues we present a novel experimental testbed that leverages the Givenness Hierarchy to produce an entity's cognitive status. We also discuss challenges in developing this testbed and how we surmounted them.