Evaluating Group Fairness and Relevance in Conversational Search: An Alternative Formulation

The Conversational Search (CS) Subtask of the NTCIR-18 FairWeb-2 Task used Sakai's GFRC (Group Fairness and Relevance for Conversations) measure for evaluating the participating systems. As the Relevance and Group Fairness components were not directly integrated in GFRC and the measure lacked a clear user model, the present pilot study discusses an alternative called GFRC2. By directly transferring the general idea of the GFR (Group Fairness and Relevance) framework for web search to the task of evaluating generated conversations, we formulate GFRC2 as a form of expected user experience for a population of users who go through the words within the conversation. This also lets us visualise the Relevance and Group Fairness component scores for each cluster of users who are assumed to abandon the conversation at a particular relevant nugget. We demonstrate the steps of computing GFRC2 using real runs from the FairWeb-2 CS Subtask.