Beyond beyond standardization: studying robustness of empirical claims based on topic modeling through multiverse analysis

While topic modeling is widely used, some scholars have already announced that the application of topic modeling in social science research is impossible to standardize. Meanwhile, researchers have to make numerous methodological decisions, yet base their empirical claims on just one topic model. This raises the question of robustness about those claims. In this study, we apply the framework of preregistered multiverse analysis to evaluate the robustness of previous empirical claims from six studies that were based on topic modeling. Based on the Open Science materials of these six studies, we slightly modify the original topic modeling procedures, such as preprocessing, number of topic clusters, and topic modeling algorithm, to other defensible choices and determine whether the empirical claims remain consistent. Even though we observe that certain empirical claims are more robust, most claims are distorted considerably by changes to the modeling options. The contributions of this study are twofold. First, we confirm that empirical claims based on just one topic model might not be robust against numerous researcher degrees of freedom. Second, we advocate a wider adoption of preregistered multiverse analysis in social science research for checking the robustness of empirical findings.