Beyond Flip Rate: A Multi-Criteria Comparative Benchmark of Counterfactual Explanation Methods for Indonesian Text Classification

Purpose – A counterfactual explanation for Indonesian text classification must simultaneously flip the classifier’s decision, remain a minimal and fluent edit, and preserve content that has no bearing on the label. This study asks whether a more powerful generator yields a better Indonesian counterfactual and benchmarks five generators under a single multi-criteria framework. Design/methods/approach – Five generators, a masked-language substitution baseline, two zero-shot LLM pipelines, a gradient-guided variant, and a second-family LLM, were evaluated on two Indonesian test sets across validity, edit quality, format compliance, and cost, combined into a single composite index, and tested for statistical significance. Findings: The masked-language baseline is the strongest and cheapest generator, leading on both flip rate and composite quality while running approximately 55 to 198 times faster than the LLM pipelines. The LLM pipelines are competitive on edit quality but not on validity, and the second-family model degenerates on roughly two-thirds of its outputs, a failure that a flip-rate-only evaluation would have masked. Research implications/limitations: For practitioners building trustworthy AI systems, the practical implication is that a cheaper, simpler generator can be a safer choice, and that generator selection should rest on multi-axis, cost-aware evidence rather than on flip rate or model size alone. All axes here are automated proxies evaluated on two tasks without human judgement. Originality/value – To the best of our knowledge, this study presents the first multicriteria counterfactual benchmark for Indonesian text classification, showing that generator size is a poor proxy for explanation quality and offering a reusable, cost-aware protocol for selecting trustworthy counterfactual methods in lower-resourced languages.