Can Algorithms Efficiently Identify Interpretable and Persuasive Message Features? An Agnostic Causal Machine Learning Approach

Argument strength and message persuasiveness are key constructs in message effects research. Yet, researchers still lack a systematic and efficient approach to uncover the "recipe" for these message-level latent features. We applied an agnostic causal machine learning approach that integrates the supervised Indian Buffet Process (sIBP) algorithm with AI-facilitated researcher refinement, train/test set splitting, and crowdsourcing in two multiple-message experiments, each with a large stimulus pool. We conducted message-level analyses on (a) textual tobacco control messages (K = 377) among Chinese men who smoke (N = 1,206) and (b) COVID-19 vaccine promotional messages (K = 1,759) among a national sample of U.S. adults (N = 819). This agnostic approach discovered that messages emphasizing negative health consequences increased argument strength and message persuasiveness. In contrast, politicizing cues reduced the message persuasiveness of social media messages promoting COVID-19 vaccines. We discussed the strengths and limitations of this approach for future message effects research.