Black box measures? How to study people’s algorithm skills

Considerable scholarship has established that algorithms are an increasingly important part of what information people encounter in everyday life. Much less work has focused on studying users’ experiences with, understandings of, and attitudes about how algorithms may influence what they see and do. The dearth of research on this topic globally with diverse populations may be in part due to the difficulty of studying a subject about which there is no known ground truth given that details about algorithms are proprietary and rarely made public. This paper explicitly takes on the methodological challenges of studying people’s algorithm skills to shed light on the special considerations required when studying a topic about which even the researchers possess limited know-how. The paper advocates for more such scholarship to accompany existing system-level analyses of algorithms’ social implications and offers a blueprint for how to do so.