Living with an Agent: A Privacy-Preserving System for Longitudinal Assessment of Family Social Dynamics in Smart Homes

Recent advances in ambient sensing, speech processing, and smart-home technologies have enabled the passive monitoring of human activities in domestic environments. Existing systems primarily focus on detecting speech presence, social interaction occurrence, emotional states, or individual behavioral indicators. However, limited work has explored how continuous in-home sensing can support reflection on family-level social dynamics while preserving privacy and avoiding judgmental or diagnostic interpretations. We present and evaluate a privacy-preserving, locally processed household sensing architecture that integrates distributed audiovisual sentinels, speech analytics, structured interaction abstractions, and longitudinal evidence retrieval. The study evaluates selected components under controlled and household deployment conditions and identifies the limitations of automated relational interpretation, particularly in the presence of incomplete observations and generative language-model errors. This work aims to gradually bring AI agents at home towards a more intimate human-machine relationship. We initiate this co-existence through a privacy-preserving platform that utilizes distributed room-level audiovisual sentinels and on-premises speech analytics to derive longitudinal indicators of household relational dynamics. These indices include Warmth, Joy, Tension, and Withdrawal. Rather than attempting to infer subjective constructs such as happiness, depression, or marital satisfaction, it focuses on observable conversational behaviors including participation balance, responsiveness, conflict intensity, repair after conflict, and social inclusion. The system introduces a smartwatch-like visualization framework that communicates family-level and dyadic social dynamics through ambient and interpretable visualizations. Weekly reports summarize patterns of connection while maintaining a non-judgmental approach. Our aim is to establish an evidence-grounded relational representation of the family system, in which software agents coexist with human residents as tools for observation, communication, and reflection that will follow them for decades. In this work, we narrow our aim as to whether the system can identify observable interactional behaviors under clearly defined conditions. We have the ambition to initiate a trend where everyone is allowed to design their personal in-home agent, therefore we provide a large open codebase that enables researchers and developers to implement, examine, and extend the proposed framework within their own homes.