LLM Safety as a Quality Gate: Integrating Bias, Fairness, and Robustness Evaluation into CI/CD via GitHub Apps

This paper presents LLM Eval Agent, a production-deployed GitHub App that automates bias, fairness, and robustness evaluation of large language models (LLMs) directly within the pull request review process. Applying the shift-left principle from DevSecOps to LLM safety, the system intercepts PR events via GitHub webhooks, executes a configurable five-test evaluation harness (LangTest) across multiple model architectures, and surfaces results as GitHub Check Runs that block merge when safety thresholds are not met. We evaluate three SST-2 fine-tuned models — DistilBERT, BERT-base, and RoBERTa — across five test categories. All three architectures achieve strong bias consistency (98–100%) and robustness (82–100%), yet every model fails the fairness gate (0% gender F1 pass rate), identifying a systemic SST-2 training data problem surfaced automatically at merge time. The system provides confidence scoring, append-only audit logging with a longitudinal trend API, adversarial red-teaming, and scheduled drift detection. End-to-end validation on a live Railway deployment (PR #4, kavitaj11/llm-eval-demo) confirmed 21-second evaluation latency and correct block-merge enforcement under both normal and timeout conditions. Source code: https://github.com/K11-Software-Solutions/llm-eval-agent-appDemo repository: https://github.com/kavitaj11/llm-eval-demo