Abstract Incorporating user needs into design strategies is a promising approach for successful product design. Numerous studies extract design implications from user-generated data through supervised and unsupervised learning techniques. While supervised learning methods generally deliver superior performance, they require extensive data labeling, which is time-consuming and labor-intensive. This paper introduces a novel framework aimed at automating the labeling of user data, thereby addressing this critical limitation. The proposed framework consists of two pseudo-labeling mechanisms, keyword detection and Large Language Model (LLM) application. The first stage extracts keywords for the target topic and then labels datasets by checking if the data contains these keywords. The second stage employs an LLM and labels the remainder of the first stage based on their context. This paper presents two applications of LLMs tailored to the characteristics of the target data. (i) Prompting LLM: This approach appends a task-specific template to the input text (reviews) and predicts the masked token representing the label. (ii) Fine-tuned LLM: Leveraging domain knowledge, this method involves fine-tuning the LLM to classify the input data (reviews) with improved accuracy and contextual relevance. The proposed method was evaluated using a case study with real-world datasets. The study validates the effectiveness of the novel framework by comparing pseudo-labeled results for smartphone sub-features against ground-truth labels. The results reveal that the framework achieves F1 scores ranging from 87% to 97%, significantly outperforming a baseline model, which yield F1 scores between 70% and 89%.
Jiang et al. (Thu,) studied this question.