Exploring Privacy and Accuracy Trade-Offs in Crowdsourced Behavioral Video Coding

Coding behavioral video is an important method used by re-searchers to understand social phenomenon. Unfortunately, traditional hand-coding approaches can take days or weeks of time to complete. Recent work has shown that these tasks can be completed quickly by leveraging the parallelism of large online crowds, but using the crowd introduces new concerns about accuracy, reliability, privacy, and cost. To explore these issues, we conducted interviews with 12 researchers who fre-quently code behavioral video, to investigate common prac-tices and challenges with video coding. We find accuracy and privacy to be the researchers ’ primary concerns. To explore this more concretely, we used sample videos to investigate whether crowds can accurately recognize instances of com-monly coded behaviors, and show that the crowd yields accu-rate results. Then, we demonstrate a method for obfuscating participant identity with a video blur filter, and find, as ex-pected, that workers ’ ability to identify participants decreases as blur level increases. The workers ’ ability to accurately and reliably code behaviors also decreases, but not as steeply as the identity test. This trade-off between coding quality and privacy protection suggests that researchers can use online crowds to code for some key behaviors in video without com-promising participant identity. We conclude with a discussion of how researchers can balance privacy and accuracy on their own data using a system we introduce called Incognito.