Using LLMs as sentiment analyzers to predict review helpfulness: first insights to open the black box

Abstract This study examines the potential of large language models for sentiment analysis in marketing. Using the empirical setting of online customer reviews, we further explore implications for prediction of review helpfulness. Relying on a dataset of 28,900 product reviews from a consumer platform and an experiment with N = 1063 participants, we find that the LLM’s accuracy in assessing intended meaning (as in the star-rating) in customer-written text depends on the degree of emotionality, as in purchases of hedonic (vs. utilitarian) goods. We further demonstrate that deviations between LLM classification and original star rating predict lower review helpfulness. This effect is mediated by the deviation of human readers’ assumption on the intended star rating from the actual star rating and moderated by the degree of information asymmetry before the purchase; that is, a greater deviation between the LLM classification and the original star rating indicates lower review helpfulness for search goods than for experience goods.