Does Long-Term Memory Improve Chatbot Performance? An Empirical Evaluation

This study investigates whether long-term memory improves chatbot performance across different types of tasks. Rather than assuming memory automatically makes a chatbot better, the paper directly compares a memory-enabled version of ChatGPT against a non-memory version across 2,000 prompts covering comprehension, reasoning, descriptive, and writing tasks. To evaluate this, the paper introduces a three-dimensional framework — Deciphering, Deciding, and Describing — along with a fourth category, Essay and Writing, for tasks requiring longer written responses. Each prompt was scored across all three dimensions by 30 independent raters from six countries, along with a custom Personalization Factor designed to measure how much the memory-enabled model adapted to a user's writing style and prior interactions. The results show that memory provides little to no benefit for comprehension and reasoning tasks (Deciphering and Deciding), where both models performed nearly identically. However, memory did show measurable advantages in descriptive and writing-oriented tasks (Describing and Essay/Writing), where personalization also had a positive effect on response quality. The paper argues that long-term memory should be evaluated on a task-specific basis rather than assumed to universally improve chatbot performance, and discusses the tradeoffs between personalization, computational cost, and performance in memory-enabled AI systems.