Shot Selection to Determine Legality of Actions in a Rule-Heavy Environment

Given a set of rules expressed in real-world regulatory language and an action one wants to perform, can state-of-the-art large language models determine whether the action is permissible? If not, why not, and what additional information is needed before they can perform this task well? These questions have been of significant interest recently, given the use of rules to constrain language models (LMs) behaviors and ensure that they align with human values. However, existing work studying how well LMs can interpret those rules focuses on a narrow set of domains, with little focus on cases where those rules have the complexity of real-world legal systems. We investigate these questions by constructing a benchmark dataset from a board game selected for the complexity of its rule set. We study the effects of shot size, function calls, and example rationales on the quality of reasoning about action permissibility. We then discuss the implications of these findings to work in automated legal reasoning.