What does “I will, yeah!” mean? Can chatbots do pragmatics?

Abstract We consider the compatibility of artificial intelligence (AI) chatbots with human pragmatics and whether they have utility as a tool in the understanding of human communications, and language in use. We claim that, precisely because pragmatics reflects a human communication-oriented endeavour, there are a number of cognitive processes to do with processing context, speech acts, common ground, implicatures, and inference, that are potentially difficult for an AI chatbot to accurately handle and mimic. Another claim is that an AI chatbot manages common ground differently to a person. In order to advance our knowledge of the pragmatic behaviour of the AI chatbots, we undertake empirical testing of Microsoft’s Copilot, Google Gemini, and OpenAI’s ChatGPT. We test against: a) the behaviour of these AI chatbots for how they treat local, linguistic, encyclopaedic and contextual knowledge; b) the AI chatbots’ knowledge and understanding of common ground; c) the AI chatbots’ understanding of different kinds of speech acts; d) reference resolution across a discourse; e) implicatures and inferencing. We address the following questions: 1) To what degree does an AI chatbot engage with pragmatic processes? 2) Can an AI chatbot mimic the human cognitive architecture? 3) To what extent can an AI chatbot be trusted with the quality of the knowledge it produces?