Answers are grounded in this paper and cite their sources.
Retrieval system effectiveness can be measured in two quite different ways: by monitoring the behavior of users and gathering data about the ease and accuracy with which they accomplish certain specified information-seeking tasks; or by using numeric effectiveness metrics to score system runs in reference to a set of relevance judgments. In the second approach, the effectiveness metric is chosen in the belief that user task performance, if it were to be measured by the first approach, should be linked to the score provided by the metric.
No takes yet. Share an insight, caveat, or question.
Moffat et al. (2013) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: