The of personal computer (PC) tasks represents a systems-level challenge that integrates natural language processing, visual perception and mouse–keyboard action control. While existing approaches mainly focus on the application programming interface (API)-based or terminal-based automation, which are incompatible with the majority of applications for the lack of accessible interface. In this article, we propose PCLLM, a novel end-to-end system that automates PC operations by integrating large language models (LLMs) with computer vision techniques to directly control the mouse and keyboard. First, a software knowledge-based prompt engineering method is developed to comprehend software architecture and operational sequences. Second, template matching techniques are integrated for precise element localization, allowing the system to accurately identify and interact. Third, a dual-LLM pipeline is designed to automatically generate the test data, where a questioner LLM generates diverse task commands and the PCLLM executes these tasks, the corresponding process data are recorded automatically for performance evaluation. Finally, PCLLM is further validated through three typically PC applications (Notepad, Wordpad and Calculator), demonstrating its flexible and robust performance towards intelligent PC automation. To evaluate the proposed system, we adopt task completion rate as the primary metric. Experimental results show that PCLLM achieves the highest completion rates of 98.59%, 95.77%, and 52.11% on Notepad for basic, intermediate, and advanced tasks respectively when powered by GPT-4o, outperforming the CogAgent baseline. These results demonstrate the effectiveness of our approach for PC task automation.
Wang et al. (Sat,) studied this question.