In e-commerce platforms, millions of product–offer matchings are performed daily, which requires scalable and efficient solutions beyond traditional methods. This study aims to improve the deployment performance of Large Language Models (LLMs) in high-volume data matching processes. In this context, the Turkish BERT model was converted into the ONNX format, and the model size was reduced from 1.2 GB to 200–300 MB, thereby enhancing deployment efficiency. The performance of the model was comparatively evaluated across five different deployment infrastructures: NVIDIA Triton Inference Server, BentoML, Jina AI, LightningLite, and FastAPI (GPU). The results demonstrate that Triton Inference Server provides superior performance compared to other solutions, with its high throughput capacity and low latency. Furthermore, migrating from CPU to GPU achieved an approximately 70% improvement in response times and a reduction in operational costs. Future work will focus on automating model updates and orchestrating multiple models.
Tekelioğlu et al. (Tue,) studied this question.