Do users anchor on large language model rankings? How do they adjust?

This paper presents the results of an experiment about humans using a large language model (LLM), analyzing the extent to which humans anchor on numeric rankings provided by an LLM, the extent to which they adjust those rankings and the extent to which those adjustments are ‘systematic’. The paper finds that users appear to anchor on those LLM numeric rankings, but users appear to adjust either by decreasing high rankings or increasing low rankings – few took the LLM ranking exactly, in contrast to expectations based on the notion of ‘least effort’. This suggests a ‘semi-autonomous’ bias and hybrid intelligence, whereby people’s judgments are affected by the anchor bias. They make a change in the LLM information to generate their own ranking. Although a self-assessment of their ‘knowledge’ was not rated highly, that knowledge was positively and statistically significantly related to their rankings.