An Interpretable Multi-Signal Scam Detection System Using Machine Learning & LLMs

Scam and phishing detection systems typically rely on rigid heuristic rules or opaque large language models. This paper presents ScamShield: a hybrid, interpretable pipeline combining a 24-feature supervised machine learning model, an LLM semantic safety layer, and rule-based heuristics in a unified ensemble. Evaluated on a synthetic benchmark of 19,992 messages spanning 17 scam categories and validated on the UCI SMS Spam Collection (5,574 real-world messages), ScamShield achieves cross-validated F1 = 0.9969 ± 0.0004 on the synthetic benchmark and F1 = 0.9303 ± 0.0098 on real-world data — competitive with fine-tuned DistilBERT (estimated F1 ≈ 0.97–0.99) while requiring 125× less storage, operating at sub-5 ms inference latency, and providing full per-prediction interpretability via coefficient attribution. Statistical significance is confirmed by McNemar's test against all four baselines (p < 0.001). Adversarial evaluation reveals recall drops of 71–82% under obfuscation attacks, identifying semantic embeddings and character-level robustness as the primary improvement directions. Index Terms— Scam Detection, Interpretable ML, Feature Engineering, Gradient Boosting, Ensemble Systems, LLM Safety, Adversarial Robustness, McNemar's Test, Phishing Classification, Probability Calibration.