Evaluating interactive systems in TREC

The TREC (Text REtrieval Conference) experiments were designed to allow large-scale laboratory testing of information retrieval techniques. As the experiments have progressed, groups within TREC have become increasingly interested in finding ways to allow user interaction without invalidating the experimental design. The development of an “interactive tract” within TREC to accommodate user interaction has required some modifications in the way the retrieval tasks is designed. In particular there is a need to simulate a realistic interactive searching task within a laboratory environment. Through successive interactive studies in TREC, the Okapi team at City University London has identified methodological issues relevant to this process. A diagnostic experiment was conducted as a follow-up to TREC searches which attempted to isolate the human and automatic contributions to query formulation and retrieval performance. © 1996 John Wiley & Sons, Inc.