Benchmark Validity in Financial Language-Model Evaluation

Benchmark results are increasingly used to compare language models for financial decision-support tasks, but such comparisons can confound domain learning with compliance with task-specific output contracts. This study examines that measurement problem using FinVector-Market-4B, a LoRA adaptation of Qwen3.5-4B for structured financial reasoning. The model is evaluated using a matched 2×2 design that crosses model state — base versus fine-tuned — with prompt-contract visibility — implicit versus explicit schema. This design makes it possible to distinguish improvements associated with learning the response contract from gains that remain after the base model is given the same output structure. The results show that prompt specification materially changes measured performance: supplying the explicit schema raises base-model JSON validity from 0% to 91.3%. Substantial task-level gains nevertheless remain after this control. Under the matched explicit-schema condition, FinQA exact match improves from 14.7% to 40.0%, correct calculator-expression accuracy from 48.0% to 82.7%, scenario-direction agreement from 20.1% to 89.5%, and structural market-implication direction agreement from 52.4% to 87.2%. Performance is not uniformly improved: policy-tone macro-F1 declines from 77.4% to 62.3% despite a small rise in overall policy accuracy. The study treats benchmark design as a measurement issue in financial language-model evaluation and shows why prompt-contract visibility, task-level error analysis, negative results, and explicit claim boundaries are important when interpreting evidence of domain adaptation.