This work presents a comparative experimental study of selected transformer-based language model architectures: GPT (Generative Pre-trained Transformer), BERT (Bidirectional Encoder Representations from Transformers), and BLOOM (BigScience Large Open-science Open-access Multilingual Language Model). The evaluation spans execution time, accuracy, computational efficiency, cost-effectiveness, and task-specific performance across six key Natural Language Processing (NLP) benchmarks: Stanford Question Answering Dataset (SQuAD), General Language Understanding Evaluation (GLUE), Super General Language Understanding Evaluation (SuperGLUE), Cross-lingual Natural Language Inference (XNLI), AG News Text Classification Dataset (AG News), and the Massive Multitask Language Understanding benchmark (MMLU). BERT-Large demonstrates strong performance in classification-oriented tasks, achieving 94. 1% accuracy on AG News and an 89. 2% F1 score on SQuAD. GPT-3 excels in high-quality text generation but carries the highest training cost (4. 6M) and inference cost (0. 060 per 1K tokens). BLOOM-7B provides relatively stronger multilingual capabilities with a 91. 2% accuracy on XNLI while maintaining moderate computational requirements. GPT-2 offers the most cost-efficient option for resource-constrained applications with a training cost of only 40K. Model selection should therefore be guided by the intended task requirements, computational budget, and deployment constraints. This analysis provides experimentally grounded insights under a controlled yet task-aligned evaluation setting for researchers, practitioners, and organizations aiming to determine the most suitable LLM architecture for their specific use cases, while acknowledging practical differences in evaluation methodology across model types rather than a fully uniform benchmarking pipeline.
Deshpande et al. (Wed,) studied this question.