Development and validation of a scale assessing perceived trustworthiness in large language models

Large language models (LLMs) are increasingly part of everyday life, yet there is no established way to measure how users evaluate their trustworthiness. This study introduces the Perceived Trustworthiness of LLMs scale (PT-LLM-8), a self-report psychometric instrument measuring users’ perceived trustworthiness of their primary LLM. PT-LLM-8 is developed from the TrustLLM framework and adapted as a human-centred measure. The PT-LLM-8 comprises eight items representing truthfulness, safety, fairness, robustness, privacy, transparency, accountability, and compliance with laws. Psychometric properties of the scale were tested with 752 LLM users in the United Kingdom (Mean age = 28.58, SD = 6.11, 50.3% males, 48.8% females). Exploratory and confirmatory factor analyses supported a one-factor structure, with high internal consistency (Cronbach’s alpha = 0.90, Composite Reliability = 0.91, strong item-total correlations (ranged between 0.62 and 0.75), and measurement invariance across gender. The total score measures overall perceived trustworthiness, while individual item responses may also be examined descriptively for information about the content represented by each item. For researchers, practitioners, and developers, the PT-LLM-8 offers a practical instrument for evaluating interventions, comparing groups and contexts, and examining whether technical safeguards are reflected in users’ perceived trustworthiness of LLM. The scale can also be applied to guide system design, support policy development, and help organisations monitor shifts in user trust toward LLMs over time, making it applicable across research, practice, and governance.