Extracting Legal Norm Analysis Categories from German Law Texts with Large Language Models

The digitization of public services in Germany is always based on a legal basis (e.g., laws). In the digitization process, first relevant entities in law documents (e.g., actors) are detected, then a list of possible process steps of their interactions is derived. The final process is constructed and transformed to a digital service for citizens and companies. Today, the discovery of custom entities in German law documents is still manual high effort work. In our study, we investigate the capabilities of Large Language Models (LLMs) to automate this task, choose five LLMs from 61 evaluated candidates, and perform prompt engineering to create five different prompt variants with differing parts. We examine the automatic annotation by two LLMs (LeoLM and BLOOM CLP German) in detail and find that the inclusion of more information in the prompts as well as an increased number of examples per prompt are beneficial. We report micro F1-scores for the optimal scenario of 0.91 for BLOOM CLP German, and 0.82 for LeoLM, with a higher balanced accuracy for LeoLM. The results indicate that LLMs have a good potential to perform named entity recognition, especially for supporting legal norm analysis in the context of the digitization of public administration.