AI-driven assessment of semantic similarity and association of responses between the 'Big Five' and DSM-5/ICD-11 personality traits

Rating scales developed with different purposes are often found to provide similar scores. For similar scores to reflect associations with the respective underlying constructs, one must ensure that the questions contained in the scales do not contain items with similar content. We evaluated the use of large language models to identify parts of the NEO inventory and the DSM-5/ICD-11 personality disorders scale whose semantics accounted for the pattern of association of responses of participants (N = 502), and candidate items that reflected ‘genuine’ associations, i.e. those that cannot be predicted by their semantic relatedness. We found that the performance of the language models we examined differed considerably. The best performing models, however, could predict response correlations between subscales of these two inventories from their semantic similarity, reveal the internal semantic structure of the subscales, and identify candidate items responsible for the semantic origin of responses. Future improvements in the size of language models may provide greater accuracy in the characterization of the semantic similarity at the granularity of individual items.