Scrutinizing Systemic Risks in Personalized Recommender Systems Through Sock-Puppet Auditing of VLOPs

Very Large Online Platforms (VLOPs) use personalized recommender systems to optimize their main performance metric: attention-based user engagement. In doing so, these systems might however amplify systemic risks by promoting controversial or polarizing content, thereby exacerbating issues such as misinformation, societal polarization, and the manipulation of civic discourse. To mitigate these risks, regulations such as the European Union’s Digital Services Act (DSA) mandate increased data access and transparency, including for the auditing of personalized recommender systems. However, the data access provided by VLOPs remains limited—often restricted to specific user demographics, aggregate statistics, or curated datasets—hindering meaningful oversight. Consequently, new methods are needed to audit recommender systems effectively at the user level. In this paper, based on an analysis of the legal context and technical alternatives for data access, we present SOAP, the System for Observing and Analyzing Posts. SOAP is an open-source framework for auditing recommender systems using sock-puppet accounts. It enables fine-grained user-level analysis beyond the constrained data access typically provided by platforms. We detail SOAP’s technical implementation and evaluate its ability to scrutinize systemic risks. Additionally, we tested SOAP in a workshop with over 100 participants and observed a measurable increase in participants’ algorithmic literacy. This demonstrates SOAP’s potential not only for research and regulatory auditing, but also as an educational framework to foster public awareness of algorithmic influence.