Key points are not available for this paper at this time.
Most popular strategies to capture subjective judgments from humans involve construction of a unidimensional relative measurement scale, representing preferences or judgments about a set of objects or conditions. This is generally captured by means of direct scoring, either in the of a Likert or cardinal scale, or by comparative judgments in pairs or. In this sense, the use of pairwise comparisons is becoming increasingly because of the simplicity of this experimental procedure. However, this requires non-trivial data analysis to aggregate the comparison ranks a quality scale and analyse the results, in order to take full advantage the collected data. This paper explains the process of translating pairwise data into a measurement scale, discusses the benefits and of such scaling methods and introduces a publicly available in Matlab. We improve on existing scaling methods by introducing analysis, providing methods for computing confidence intervals and testing and introducing a prior, which reduces estimation error the number of observers is low. Most of our examples focus on image assessment.
Pérez‐Ortiz et al. (Mon,) studied this question.