Key points are not available for this paper at this time.
Large Language Models (LLMs) especially ChatGPT have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. Existing works study the virtual personalities of LLMs but rarely explore the possibility of analyzing human personalities via LLMs. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers–Briggs Type Indicator (MBTI) tests. Specifically, we first devise unbiased prompts by randomly permuting options in MBTI questions and adopt the average testing result to encourage more impartial answer generation. Then, we propose to replace the subject in question statements to enable flexible queries and assessments on different subjects from LLMs. Finally, we re-formulate the question instructions in a manner of correctness evaluation to facilitate LLMs to generate clearer responses. The proposed framework enables LLMs to flexibly assess personalities of different groups of people. We further propose three evaluation metrics to measure the consistency, robustness, and fairness of assessment results from state-of-the-art LLMs including ChatGPT and GPT-4. Our experiments reveal ChatGPT’s ability to assess human personalities, and the average results demonstrate that it can achieve more consistent and fairer assessments in spite of lower robustness against prompt biases compared with InstructGPT.
Building similarity graph...
Analyzing shared references across papers
Loading...
Haocong Rao
Cyril Leung
Chunyan Miao
University of British Columbia
Nanyang Technological University
Building similarity graph...
Analyzing shared references across papers
Loading...
Rao et al. (Sun,) studied this question.
www.synapsesocial.com/papers/6a08b721113ba5b476de6c90 — DOI: https://doi.org/10.18653/v1/2023.findings-emnlp.84
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: