Output-Level Evaluation Limits in Profile-Conditioned LLM Responses: A Case of Indistinguishable Explanatory Framing under Memory Settings

LLM responses under persistent user profile conditions may exhibit differences in explanatory framing even when identical prompts are used. However, such differences are difficult to evaluate from output-level observations alone, as they may be indistinguishable from generic personalization behavior. This paper presents a small observational dataset across five domains and four session conditions (persistent memory, memory-off, chat history disabled, and temporary chat). While differences in explanatory structure appear in several cases, informal external review indicates that these differences cannot be readily classified using output-only information. This suggests a potential limitation in current evaluation approaches: output-level observations may be insufficient to distinguish between generic personalization and deeper changes in explanatory pathways. This observation is consistent with recent findings that interaction context can significantly alter LLM behavior in long-term settings (Jain et al., 2026), while further suggesting that such changes may not be distinguishable from output-level observations alone.