Enhancing Data Retrieval with Custom Embedding Models and ChatGPT

This research aims to study data fragmentation and inaccessibility challenges due to large company growth such as with Eltronic A/S. A real use case is being experimented upon, using the proposed methodology. The devised solution is a search platform that utilizes a fine-tuned embedding model trained on company data to index and fetch relevant information. The model was trained using the HuggingFace Sentence transformer library. The training loss utilised is Multiple Negatives Ranking Loss on augmented data, generated using ChatGPT. Combining the embedding model with the integration of ChatGPT allows for a more seamless way of performing information search. In this study, a fine-tuning model on internal company data increases the Top-1 Retrieval errors by 10 % compared to other state-of-the-art models. The solution also increases employees' ability to find relevant information compared to the company's previous solution.