Answers are grounded in this paper and cite their sources.
Frozen language models can produce plausible smart-contract reports despite generation failures and weak discrimination. We benchmark three approximately 7-billion-parameter models on SmartBugs-Curated, ScrawlD, and balanced BCCC under a fixed prompt family and documented recovery procedures. Across 75,957 model–record outputs, final validity is 100%, but native first-attempt record validity ranges from 3.7% to 85.3%; all three datasets fail a declared recovery-sensitivity criterion. The reported scores are conditional on the evaluated model snapshots, prompt, and recovery protocol rather than estimates of general Code-LLM performance on these benchmarks. The best raw micro-F1 is 0.961 on SmartBugs-Curated and 0.108 on ScrawlD; the best BCCC balanced accuracy is 0.520. We examine the Taxonomy-Aware Calibrated Ensemble (TACE) using nested cross-fitting with prevalence-only and logistic-stacking controls. On ScrawlD, their micro-F1 values are 0.5985, 0.5978, and 0.5986; 80.1% of TACE’s positive decisions occur when all three models predict negative. Source-family-grouped analysis gives qualitatively similar conclusions. Confidence-based abstention can retain mostly negative decisions while discarding almost all true positive alerts. These findings support TACE as an interpretable reliability diagnostic, not evidence of superior semantic detection or model complementarity. The study separates native validity, recovery dependence, prevalence effects, calibration, discrimination, and selective positive-alert utility; prompt robustness and cross-dataset transfer remain untested.
No takes yet. Share an insight, caveat, or question.
Le et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: