Don't Touch the Power Line - A Proof-of-Concept for Aligned LLM-Based Assistance Systems to Support the Maintenance in the Electricity Distribution System

As a possible solution to the demographic change and the resulting knowledge loss due to retirements in the Energy sector, this study aimed to develop a generic pipeline to implement and evaluate proof-of-concepts (PoCs) for LLM-based assistance systems in new domains. Our pipeline contains an LLM-based data generation strategy based on documents, a retrieval-augmented generation (RAG) architecture utilizing prompting techniques on existing German LLMs, and an LLM-based automatic evaluation strategy. We leverage our pipeline to evaluate five LLMs using data from a German DSO. We found that the Llama3 and the Mistral model are appropriately aligned for the task. We plan to pilot the RAG architecture in the DSO's infrastructure for future research and continuously research improvements using the generated human demonstrations.