Extracting structured information from complex legal texts poses a significant challenge due to the ambiguity and precision required in the legal domain. While Large Language Models (LLMs) have shown promise for such tasks, the effectiveness of Reinforcement Learning from Human Feedback (RLHF) in improving their performance on formal information extraction, such as identifying ontology-based facts, acts, and preconditions, has not been thoroughly explored. In this study, we examine methods to enhance the performance of smallerscale LLMs on legal information extraction by using 1) RLHF and 2) Reinforcement Learning from AI Feedback (RLAIF). We construct a dataset of 282 expert-annotated legal text segments, each labelled with preconditions or subfacts linked to specific acts or facts based on a legal-domain ontology. For instance, if borrowing a book requires library membership, the model should identify this precondition and the corresponding section in the legal document. We collect feedback on model-generated answers from six domain experts and a GPT-4.1-based LLM-as-a-judge, using evaluation criteria consistent across both sources. The human feedback shows moderate inter-annotator agreement (Fleiss’ Kappa ≈ 0.5), indicating the inherent difficulty of the task, even among experts. Our findings demonstrate that RLHF leads to better model performance than RLAIF. Human-in-the-loop training yields more accurate and coherent extractions, while models fine-tuned on AI feedback tend to produce shorter responses that are often correct but less comprehensive. Notably, human feedback promotes gradual, structured improvements in output quality, reinforcing the value of expert evaluation for complex NLP tasks. While promising, our method requires scaling to larger datasets to validate its effectiveness fully. Nevertheless, our study provides early evidence that RLHF, particularly with expert input, is a powerful tool for aligning LLMs with high-precision information extraction goals in complex domains such as law.
Jacques Fürst (Wed,) studied this question.