Pairwise Comparison and Ranking: Optimum Properties of the Row Sum Procedure

The present paper is concerned with those pairwise comparison experiments for which ranking in descending order of the "row sums" is the appropriate ranking procedure. More precisely, assume that n items are compared pairwise, omitting no pairs, and that the result of the comparison between item i and j is expressed as a real number xᵢⱼ satisfying {equation*}{1}xᵢⱼ + xⱼᵢ = 0 all i, j.{equation*} xᵢⱼ might be some measured difference between i and j, or it might take the values $1, 0, -1$ according as item i is judged to be superior, equal or inferior to item j, or it might be a statistic summarizing the results of several comparisons between the two items, etc. It will be shown that ranking in descending order of the scores {equation*}{2}s_i = ∑^nⱼ₌₁ xᵢⱼ{equation*} (row sum procedure) uniformly minimizes the risk among all permutation invariant procedures, and for all "reasonable" loss functions, provided the xᵢⱼ (i < j) are independent random variables distributed according to an exponential distribution of the type {equation*}{3}P(xᵢⱼ t) = c(ϑ_i - ϑ_j) ∫^t-∞ e(ϑ_i-ϑ_j)τ μ(dτ),{equation*} where μ is a symmetric probability measure on the real line. Then s = (s₁, ⋯, sₙ) is a sufficient statistic for the joint distribution of the xᵢⱼ. Under suitable regularity conditions, the converse also holds: if the distributions of the xᵢⱼ are not of this form, then s is not sufficient, and the row sum procedure is not optimal. Thus, the model just described seems to constitute the natural domain of the row sum ranking procedure. Fortunately, this domain contains the important case where the xᵢⱼ (i < j) are independent normal variables with mean ϑᵢ - ϑⱼ and equal variance σ². Another particular case--the case of tournaments without draws--where the xᵢⱼ can take only two values, has been treated in the joint paper [1]. Incidentally, the present paper grew out of an attempt to cover the case of tournaments with draws (see example (iv) in Section 4 below). Although one cannot expect strict optimality properties, one may still ask whether the row sum procedure has "nice" (e.g., minimax) properties even outside the above model. A modest result in this direction is the following. Take a particular joint distribution of the xᵢⱼ belonging to the above model, and consider the class of all joint distributions of the xᵢⱼ that lead (up to permutations) to the same joint distribution of the sᵢ as this particular one. Then this latter will be least favorable for the ranking problem, and the row sum procedure will have minimax properties in each such class (modified minimax principle of Wesler [2]). For instance, if the xᵢⱼ (i < j) are independent normal variables with mean ϑᵢⱼ and equal variance, then the joint distribution of the sᵢ depends only on the ϑᵢ = (1/n) ∑ⁿⱼ₌₁ ϑᵢⱼ, and the linear model ϑ'ᵢⱼ = ϑᵢ - ϑⱼ is least favorable, if the ϑᵢ are kept fixed (up to permutations). There is an immediate generalization of the results to the case of comparison in k-tuples instead of in pairs, as indicated in the last section of this paper.