A comparative study on retrieval-augmented generation and chain-of-thought applications for LLM-assisted engineering design ideation

This study examines the comparative impacts of Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) reasoning on Large Language Models (LLMs) in the context of generative engineering design ideation. Using a mixed-methods approach, six prompt strategies were evaluated across 40 product design tasks through quantitative metrics (Design Comprehensiveness, Intra-design Richness, and Solution Elaboration), and blind expert assessments (Requirement Identification, Requirement Fulfilment, Actionability, Novelty). Results showed convergence between quantitative analysis and expert evaluations. Non-RAG methods excelled in Design Comprehensiveness, aligning with higher expert scores for Requirement Identification, while CoT-enhanced approaches improved Solution Elaboration, correlating with superior ratings for Requirement Fulfilment and Actionability. Non-RAG methods exhibited greater internal design variety (Intra-Design Richness), whereas RAG-based outputs scored higher for external originality (Novelty). Spearman correlation, Jaccard similarity analyses and factorial analyses confirm RAG as the primary driver of conceptual domain shifts, with CoT serving as a secondary mechanism for structuring outputs. These results elucidate the trade-offs between RAG and CoT, providing a foundation for their strategic application in LLM-assisted engineering design. This work contributes to the understanding of the distinct roles and trade-offs of these techniques, providing a systematic foundation for their strategic deployment in LLM-assisted engineering design ideation.