Hybrid Optimization of NLP and Vision Transformers using Recursive Knowledge Distillation and QLoRA

This paper presents a unified framework for compressing large language and vision transformer models usingRecursive Knowledge Distillation (RKD), QLoRA, and pruning.Our experiments on the SST-2 and Beans datasets show that itis possible toachieve up to 10x model size reduction with only aminor drop in accuracy. The study benchmarks Straightforward,Successive, andMulti-Agent Distillation techniques and appliesquantization and structural pruning postdistillation to achievehighly efficient models suitable for real-world deployment.