Fine-tuning large language models (LLMs) with backpropagation even for a subset of parameters such as LoRA can be much more memory-consuming than inference and is often deemed impractical for resource-constrained mobile devices. Alternative methods, such as zeroth-order optimization (ZO), can greatly reduce the memory footprint but come at the cost of significantly slower model convergence (10 to 100 more steps than backpropagation). We propose a memory-efficient implementation of backpropagation (MeBP) on mobile devices that provides better trade-off between memory usage and compute time, while converging faster and achieving better performance than the ZO baseline. We verify the effectiveness of MeBP on an iPhone 15 Pro Max and show that various LLMs, ranging from 0. 5B to 4B parameters, can be fine-tuned using less than 1GB of memory. We release an example of the MeBP implementation at https: //github. com/apple/ml-mebp.
Song et al. (Fri,) studied this question.