Generative AI in Human-AI Collaboration: Validation of the Collaborative AI Literacy and Collaborative AI Metacognition Scales for Effective Use

Advancements in AI's conversational capabilities and situational awareness mean that now, humans work with AI collaboratively. However, these developments affect the skills needed to use AI effectively. In this study, we address the need to update measures of AI-related knowledge and skills to reflect the collaborative capability of advanced AI tools by developing and validating two new scales focusing on collaboration and metacognition. A survey of 292 users of collaborative AI tools was conducted. Both the Collaborative AI Literacy and Collaborative AI Metacognition scales showed good internal consistency and predictive validity. Structural equation modeling supported their convergent and discriminant validity. Both measures correlated with users' assessments of the benefits from working with collaborative AI tools. As predicted, Collaborative AI Metacognition explained significant variance beyond that explained by general Metacognition. These validated scales provide an important resource for assessing and researching knowledge and skills for working with Collaborative AI tools.