Evolution of a Survey for Self-Reported Engineering Design Space Exploration Tendency

Abstract develop a survey consisting of statements that provide insight into an individual's design space exploration tendency. There are formal exercises to evaluate design space exploration, but these exercises are resource intensive, time consuming, challenging to deploy and difficult to process the results. The survey instrument is intended to address several of these challenges. To develop the survey instrument, the Shah-Vargas (SV) metrics of engineering ideation effectiveness were used as a basis for quantifying engineering Design Space Exploration (DSE). These metrics are 1) Quantity – the number of ideas generated, 2) Quality – the conformance of each idea to engineering requirements, 3) Variety – the dissimilarity of an idea within an individual's set of generated ideas, and 4) Novelty – the dissimilarity of an idea within the collectively exhaustive set of ideas. With these metrics as a guide, an initial list of statements was developed using two approaches. First, literature was reviewed for statements that have been used to collect self-reported data on the four metrics. Second, the definitions of the four metrics from Shah and colleagues were reviewed and converted to question form. This resulted in four statements per metric, totaling 16 statements. Next, to assess question clarity regarding the four metrics and to ensure survey respondents accurately grasped the metric each statement pertained to, Latent Semantic Analysis (LSA) was employed to evaluate overlap. In addition, the statements were processed by a Large Language Model which was asked to assess overlap. Based on the findings from these analyses, the statements were modified to reduce overlap. A final verification of mutual exclusivity will be where participants are going to be asked to categorize each question into one of the four metrics. The result of this work is a survey with statements which allows an individual to self-report their DSE tendency. In the future, this validity of self-reported data will be assessed by comparing it with direct assessment of DSE tendency. Once validated, the DSE survey is intended for researchers to gain a deeper understanding about DSE tendencies without having the resource-intensive, subjective task of performing direct assessments. Additionally, the survey can be used as a pre-screening if/when design exercises are deployed.