As AI systems become increasingly personalized, remembering past conversations and tailoring responses to individual users, they are beginning to reflect users back to themselves in ways that feel deeply meaningful. This paper introduces the CASK framework (Computational Assessment of Self-Knowledge) to distinguish between genuine self-knowledge and what appears to be self-knowledge but is, in fact, an illusion generated by the AI. CASK identifies five key factors that, when present, can mislead users into believing they have gained authentic self-knowledge: anthropomorphic misunderstanding (misattributing normal AI behavior to unique personal insight), the Barnum effect (accepting vague personality descriptions as uniquely accurate), sycophancy (AI reinforcing existing beliefs), the richness of extended conversation creating a sense of depth, and the inherently positive framing of most responses (the Pollyanna effect). Drawing on literature from persuasion, philosophy of self-knowledge, and human-AI interaction, CASK offers a practical multi-factor assessment to help users evaluate the epistemic reliability of AI-generated self-knowledge. It also outlines key challenges for future research in this area.
Kail Lennard Patruck (Wed,) studied this question.