CopycHats: Question Sequencing with Artificial Agents

Schema Matching, the task of finding correspondences among attributes of different schemata, plays an important role in data integration. The task has been extensively researched, leading to the development of multiple algorithmic approaches, many of which incorporate humans to some extent, by performing the matching, validating algorithmic solutions, or generating reliable ground truth for algorithms to be trained against. Human matching is a temporal process, in which previous decisions and ongoing cognitive processes influence future behavior. Therefore, planning a human matching task necessitates careful consideration, for example, the ordering of questions posed to the matcher. Various strategies exist for optimally choosing the sequence of matching questions and evaluating them may be limited by notable inherent human limitations. In this work, we propose to leverage Large Language Models (LLMs) to create artificial agents that emulate human agents in order to evaluate question sequencing strategies. We offer an alternative to traditional human-based evaluations, which overcome those limitations. We test our suggested evaluation framework and discuss the similarities and differences between artificial and human agents in the context of schema matching evaluation.