Integrating Large Language Models with near Real-Time Web Crawling for Enhanced Job Recommendation Systems

This study addresses the limitations of traditional job recommendation systems that rely on static datasets, making them less responsive to dynamic job market changes. While existing job platforms address job search with an untransparent logic following their business goals, job seekers may benefit from a solution actively and dynamically crawling and evaluating job offers from a variety of sites according to their objectives. To address this gap, a hybrid system was developed that integrates large language models (LLMs) for semantic analysis with near real-time data acquisition through web crawling. The system extracts and ranks job-specific keywords from user inputs, such as resumes, while dynamically retrieving job listings from online platforms. User evaluations indicated strong performance in keyword extraction and system usability but revealed challenges in web crawler performance, affecting recommendation accuracy. Compared with a state-of-the-art commercial tool, user tests indicate a smaller accuracy of our prototype but a higher functionality satisfaction. Test users highlighted its great potential for further development. The results highlight the benefits of combining LLMs and web crawling while emphasizing the need for improved near real-time data handling to enhance recommendation precision and user satisfaction.