BeigeSage: sentiment classification in economic texts utilizing LLMs and fine-tuned BERT models

Large language models (LLMs) possess the contextual awareness and scale of pre-training needed to perform sentiment analysis on complex economic texts with human-like accuracy and computer-like speed. This study compares the ability of five LLMs (ChatGPT, Llama, Phi, Gemma, and a fine-tuned BERT model named BeigeSage) alongside a traditional rule-based method (VADER) to classify sentiment in the Federal Reserve’s Beige Book reports. Human annotators created a labelled dataset of 1000 Beige Book text chunks, and each model was evaluated using accuracy, macro F1 score, and Matthews correlation coefficient across three sentiment classes: positive, negative, and mixed. While all LLMs substantially outperformed the traditional method, BeigeSage achieved the highest overall performance, demonstrating both strong general accuracy and balance across all sentiment categories, especially for difficult passages containing mixed sentiment. Results also show that smaller open-source models like Phi and Llama can match or exceed ChatGPT’s accuracy while offering faster or lower-cost deployment. These findings suggest that fine-tuning LLMs with domain-specific labelled data significantly enhances classification quality.