Computational Identification of Media Frames: Strengths, Weaknesses, and Opportunities

With the availability of large volumes of electronic communications data and the increasing sophistication of computational techniques, the development of automated approaches for different kinds of framing analysis is an important goal of researchers. There is as yet no standard method for the “unsupervised” inductive identification of frames based upon the content of articles. Three groups of core approaches underlie a wide range of work in this area, and we compare three techniques based on these approaches against each other and against manual human analysts. The three techniques are a k-means clustering algorithm together with a sophisticated natural language processing (NLP)-based feature selection process; evolutionary factor analysis (EFA), a factor analysis approach; and the structural topic model (STM). We use two datasets – one very focused and one extremely broad – as examples of the kind of frame analysis problems readers may wish to attempt. Even in a highly targeted dataset, we find some distance between the frames generated by computational analysis and those manually produced by our human analysts. The details of each method have a substantial impact on frame quality and interpretation. We find that the STM approach is the most effective for our narrow-scope dataset, but that it returns definite topics, and not frames, when working on our very broad dataset. We also show that we can get results emphasizing different parts of the framing problem by combining parts of different methods as a multi-stage process, rather than viewing available methods as simple plug-and-play models.