Do Large Language Models Contain Software Architectural Knowledge? An Exploratory Case Study with GPT

Key Points

• The study aims to assess whether large language models can accurately assist software engineers in discovering Architectural Knowledge.
• Exploratory case study design
• Evaluation of GPT-3.5 responses to architecture-related questions
• Assessment criteria focused on accuracy, quality, and trustworthiness
• GPT-3.5's responses varied in accuracy and quality
• Identified instances where GPT provided useful insights into architectural concepts
• Trustworthiness of GPT's answers influenced by context and complexity of questions