Chatbots are widely used in the healthcare sector, making their accuracy and reliability essential. Beyond providing factually correct information, chatbots must also consider the human aspect of their responses. Large language models (LLMs) can be utilized to evaluate chatbot responses, employing prompting strategies such as chain-of-thought and few-shot prompting to enhance reasoning and optimize output quality. This study evaluates a chatbot’s answers to medical questions using both objective and subjective assessments. Different prompting techniques were applied: objective evaluation used baseline, chain-of-thought (COT), and chain-of-thought with few-shot (COTF) prompting, while subjective evaluation used baseline and baseline with few-shot (Baseline-f) prompting. The results revealed that COTF prompting with both models improved the performance of objective evaluation, while few-shot prompting enhanced subjective evaluation.
Tamayo et al. (Fri,) studied this question.