Simplicity over sophistication? Rule-based chatbots lead to higher return rates than LLMs in retail

Purpose – This study examines associations between chatbot efficiency, effectiveness and customer revisit rates, and compares efficiency and effectiveness across rule-based (RB) and large language model (LLM) chatbot deployments in retail. Design – A mixed-methods approach combined two studies. Study 1 analysed nine RB chatbots across four industries in Portugal and Brazil, covering 108,566 sessions in 2023. Pearson correlations and an exploratory multiple regression examined chatbot-level associations with revisit rates. Study 2 combined six interviews with retail managers in Portugal and descriptive data from three RB and two LLM retail chatbots, covering 11,320 sessions and 50,321 interactions in 2024. Findings – Among the nine RB chatbots, efficiency was positively associated with revisit rate after adjustment for effectiveness (β = .92, p = .004; R2 = .77); effectiveness showed no statistically detectable independent association. Within the five retail deployments examined in Study 2, RB chatbots recorded higher efficiency and effectiveness rates than the LLM systems. Study 2 did not compare revisit rates between architectures. Originality – The study connects log-derived operational metrics with observed continued use and examines their interpretation through the IS Success Model. It also provides a descriptive comparison of RB and LLM retail deployments. The findings motivate further investigation of interaction effort, while the small chatbot-level samples and observational design limit generalisation and causal interpretation.