A multimodal LLM-driven robotic control system for adaptive industrial manipulation: Integrating vision-language models for enhanced manufacturing flexibility

This research presents a multimodal Large Language Model (LLM)-driven robotic control system designed for practical industrial applications. Background: Traditional robot control systems rely on pre-programmed rules, limiting adaptability in complex environments, while recent LLM applications in robotics remain largely confined to simulations or structured scenarios. Methods: We developed a three-layer architecture (perception-reasoning-execution) implemented on an Elephant Robotics myCobot platform, featuring specialized prompt engineering strategies and an adaptive execution module with self-correction capabilities. Results: Experimental evaluation demonstrated the system's ability to understand complex instructions (82.7% accuracy), significantly outperform traditional methods in environmental adaptability (maintaining 85–92% task completion rates under variable conditions), and reduce deployment time by 98.3%. Comprehensive ablation studies validated that each proposed innovation contributes meaningfully to system performance, with the complete system achieving 94.7% task success rate representing an 18.4% improvement over traditional vision-only approaches. The multimodal approach improved performance by 16.4% over single-modality methods, with particularly strong advantages in ambiguous instruction parsing (41.4% improvement). Conclusions: This research establishes the feasibility of multimodal LLMs in industrial robotics, providing empirical evidence that such systems can combine high adaptability with practical deployment requirements, creating a foundation for more intuitive and accessible manufacturing automation systems.