Beyond apps: a method to model full distributions of screen content over time to study media effects

Research on smartphone media use has progressed from aggregate screen-time measures to app-level logs, yet these approaches still obscure the diversity and structure of the content users encounter. We introduce smartphone screen content patterns (content patterns) as a new construct and propose an analytical framework that represents smartphone use as distributions of experienced screen content rather than summaries of time spent on devices or in apps. We operationalize content patterns using deep learning embeddings of moment-by-moment screenshots and formalize use periods as probability distributions over embedding space. This distribution-based representation supports two analytic strategies: Mantel tests to assess covariation between changes in content patterns and outcomes, and k-nearest neighbor regression to evaluate whether content patterns contain sufficient information to reconstruct outcomes. We illustrate the framework with six months of screenomics data from 53 adolescents, including 21 million passively collected screenshots and 542 fortnightly mental health self-reports. Results show that most content variability occurs within applications and time periods, highlighting the limits of app- and time-based measures. Content patterns outperform screen-time measures and provide complementary insights to app-level logs, advancing a content-centric approach to modeling digital media exposure.