What question did this study set out to answer?

This study aims to extend the evaluation criteria of conversational search systems to include group fairness.

April 1, 2026Open Access

Fairness-based Evaluation of Conversational Search: A Pilot Study

Key Points

This study aims to extend the evaluation criteria of conversational search systems to include group fairness.
Utilized the FairWeb-1 test topic set for analysis
Harvested user-system conversations from New Bing and Google Bard
Developed the Group Fairness and Relevance of Conversations (GFRC) framework for evaluation
Demonstrated the application of GFRC to evaluate multiple conversational search turns
Identified significant aspects of fairness and relevance in the conversation flow
Highlighted open questions regarding human-in-the-loop evaluation methods

Abstract

NTCIR-17 introduced the FairWeb-1 task,which evaluated web page rankingsin terms of both relevance and group fairness.The present study shows how their evaluation frameworkcan be extended for the evaluation ofmulti-turn, textual conversational search systems.By using the full test topic set of FairWeb-1to harvest actual user-system conversations from the New Bing and Google Bard,we demonstrate how a series of system turns can be evaluatedusing our evaluation framework,which we callGFRC (Group Fairness and Relevance of Conversations).In addition, based onobservations from our pilot experiment,we briefly discuss a few open questions in human-in-the-loopevaluation of conversational search in general.

Read Full Paperexternally

Ask AI

Helpful

Bookmark

View Full Paper