STAIRS redux: Thoughts on the STAIRS evaluation, ten years after

The test of retrieval effectiveness performed on IBM's STAIRS and reported in Communications of the ACM ten years ago, continues to be cited frequently in the information retrieval literature. The reasons for the study's continuing pertinence to today's research are discussed, and the political, legal, and commercial aspects of the study are presented. In addition, the method of calculating recall that was used in the STAIRS study is discussed in some detail, especially how it reduces the five major types of uncertainty in recall estimations. It is also suggested that this method of recall estimation may serve as the basis for recall estimations that might be truly comparable between systems. © 1996 John Wiley & Sons, Inc.