A comparative study on textual and multimodal PII detection using large language models

Personally identifiable information (PII) detection is foundational to privacy-preserving analytics, compliant data sharing, and safe model training. Traditional rule-based and supervised named entity recognition (NER) pipelines excel on templated fields but are brittle under domain drift and long-tail variants, motivating large language models (LLMs) that generalize via in-context learning. Despite rapid LLM progress, we lack systematic and cross-domain evaluations that reflect redaction-centric error costs and practical deployment constraints. This paper presents an LLM-centric benchmark of PII detection across three textual domains and introduces a carefully annotated multimodal PII dataset to extend beyond text-only settings. Our experiments examine zero-shot and few-shot prompting, quantify prompt sensitivity, and characterize privacy-relevant error modes. We further assess a privacy-oriented, cost-aware baseline that combines a small open-source model with a multi-agent retrieval-augmented workflow, analyzing performance and feasibility for resource-constrained environments. Results highlight domain and modality effects, consistent gains from few-shot prompting (with saturation effects), and precision–recall trade-offs that vary across tasks and models. The proposed agentic retrieval augmented generation (RAG) pipeline approaches, and in some cases exceed the performance of large proprietary models while improving privacy posture and operational cost. The prompts and the multimodal dataset are made publicly available to support reproducibility and future benchmarking.