Answers are grounded in this paper and cite their sources.
AI-assisted development weakens the meaning of a green continuous-integration (CI) check: generated tests may execute realistic code and pass while replacing semantic oracles with tautologies or coarse smoke checks. We introduce CI-DeceptiveTest, a Python execution-paired benchmark and Gatekeeper protocol for detecting such low-assurance tests. Its 600 pairs originate from real tests in 12 open-source repositories; each source-control test is paired with a candidate produced by controlled oracle weakening, yielding 1200 label-free static-binary views. Construction combines deterministic validation, execution evidence, a 20% manually annotated slice, random inspection of 30% of AI-assisted records. The evidence is GPT-5.5-first: GPT-5.5 supports construction, review, and full-dataset validation. Labels are transformation-defined, not model-assigned. On the full benchmark, transparent static rules achieve 0.928 low-assurance recall but a 0.458 false-block rate on accepted source-control tests. mimo-v2.5-pro is retained as an auxiliary external cross-check; it reaches 0.885 macro F1 and reduces false blocking to 0.048. Bounded mutation analysis over 42 pre-specified source/candidate pairs further supports the paired construction. The central stress result is that weak-oracle detection remains hard: plausible coarse assertions retain execution while losing semantic constraint. The benchmark enables CI gatekeeper evaluation in a controlled, reproducible setting.
No takes yet. Share an insight, caveat, or question.
Qining Li (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: