Task-Aligned Prompts Enable Effective LLM-Based Memory Selection for Conversational QA

Long-term conversational memory selection is critical for AI assistants. Current approaches rely on LLM-based importance scoring using prompts adapted from Stanford’s Generative Agents. We present the first systematic evaluation comparing this approach against heuristic baselines across three benchmarks spanning different conversation lengths. Our experiments reveal that the widely-adopted “importance” formulation exhibits misalignment with retrieval utility: it achieves only 53% of full-context performance at 30% budget on LoCoMo, underperforming simple TF-IDF (81%). However, prompt objective matters dramatically: a task-aligned “factual content” prompt achieves 146% of full-context at 10% budget and 114% at 30%. We validate this finding across three domains—personal conversations (LoCoMo), multi-session chat (MSC), and technical support (Molweni)—revealing a critical insight: filtering benefits depend on conversation length. Our results demonstrate that task-aligned prompts combined with appropriate length-aware filtering enable LLM-based selection to substantially outperform both heuristics and full-context baselines.