Benchmarking Quantized Edge LLMs for Natural Language Mobile Robot Control

Deploying large language models on limited edge hardware remains an open challenge for natural language robot control. This study compares three quantized sub-1.5B models, namely SmolLM2-135M, Gemma3-1B, and TinyLlama-1.1B, for velocity-command generation on a TurtleBot3 Burger in ROS/Gazebo using llama.cpp. The evaluation uses a two-tier protocol: an atomic benchmark with 900 inference trials (50×6×3) measuring parse success, semantic accuracy, and latency, and a mission benchmark with 150 multi-step navigation runs and 870 total steps measuring waypoint, heading, and endpoint error from Gazebo ground-truth pose. A clear model-load cold-start effect is observed: the first warmup inference after loading is roughly 6–9× slower than steady-state warm-start inference because of KV-cache initialization and memory-mapped weight paging. Warm-start latencies differ significantly across models (Kruskal–Wallis H=682.3; p<10−148): 0.506 s for SmolLM2, 1.661 s for TinyLlama, and 2.903 s for Gemma3. Semantic accuracy also differs across models under the trial-level independence approximation used in this benchmark (χ2=57.6; p<10−12), with TinyLlama-1.1B achieving the highest score of 72.7%, ahead of Gemma3-1B with 51.7% and SmolLM2-135M with 42.7%. On mission aggregates, Gemma3-1B yields the lowest overall waypoint error of 0.654 m and endpoint error of 1.022 m, while TinyLlama-1.1B remains close and retains the strongest atomic command fidelity. These results suggest that quantized small LLMs are promising for supervised, edge-local natural language robot control, but the observed error rates and simulation-only evaluation indicate that they are not yet reliable enough for unsupervised real-world deployment.