Answers are grounded in this paper and cite their sources.
We introduce RODEO , a corpus of 5,111 royalty offset and deduction clause spans from material-contract exhibits filed with the U.S. Securities and Exchange Commission (709 exhibits, 417 filers), labelled for their economic function under a seven-function taxonomy. Because exhaustive human annotation is costly, we build the resource with a protocol assisted by a large language model (LLM): independent double classification, adjudication of substantive disagreements, and a stratified human spot-check. We release the corpus with source identifiers and privacy-screened excerpts, together with the per-pass and adjudicated labels, prompts and code. As a quality-control analysis we examine how far the model can be trusted. It is highly self-consistent on the surface (82.4% exact-set agreement across passes) yet poorly calibrated: 82% of predictions carry the maximal confidence score, even that top bin flips label across passes 13.8% of the time, and the model’s own uncertainty flag recovers only 1% of genuine disagreements. On a deliberately hard subset, a model from an independent family agrees with the adjudicated label only 36.6% of the time, whereas on a broad stratified sample the two LLM annotators agree substantially (macro Cohen’s κ = 0.78). Inter-pass disagreement, not self-reported confidence, is therefore the more useful signal for triaging hard cases when full human labelling is infeasible. Supervised baselines show that the labelling is learnable (a bag-of-words logistic-regression classifier reaches macro-F1 0.80; the notice-and-documentation condition is hardest). The released labels are LLM-generated with a human spot-check (“silver”), and we set out the path to a fully human-adjudicated version.
No takes yet. Share an insight, caveat, or question.
Y Kim (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: