LLM Inference Acceleration on Edge Devices via Dual-Model Parallel Speculative Decoding

This project builds on the finding that the draft model and target model in the speculative decoding architecture can run in parallel to a certain extent. The two models will be deployed on two separate processing units of edge devices such as mobile phones — for example, the draft model will run on the NPU and the target model on the GPU. Systematic design will be adopted to minimize hardware resource contention between the two models, so as to reduce LLM inference latency.