Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis

It is widely thought that news organizations exhibit ideological bias, but rigorously quantifying such slant has proven methodologically challenging. Through a combination of machine-learning and crowdsourcing techniques, we investigate the selection and framing of political issues in fifteen major US news outlets. Starting with 803,146 news stories published over twelve months, we first used supervised learning algorithms to identify the 14 percent of articles pertaining to political events. We then recruited 749 online human judges to classify a random subset of 10,502 of these political articles according to topic and ideological position. Our analysis yields an ideological ordering of outlets consistent with prior work. However, news outlets are considerably more similar than generally believed. Specifically, with the exception of political scandals, major news organizations present topics in a largely nonpartisan manner, casting neither Democrats nor Republicans in a particularly favorable or unfavorable light. Moreover, again with the exception of political scandals, little evidence exists of systematic differences in story selection, with all major news outlets covering a wide variety of topics with frequency largely unrelated to the outlet’s ideological position. Finally, news organizations express their ideological bias not by directly advocating for a preferred political party, but rather by disproportionately criticizing one side, a convention that further moderates overall differences.