Listed but Not Vetted: An Empirical Security Audit of 511 Public MCP Servers

This record contains an independent empirical security study of the Model Context Protocol (MCP) server ecosystem, along with the accompanying open-source scanning tool and dataset. Paper: "Listed but Not Vetted: An Empirical Security Audit of 511 Public MCP Servers." The MCP ecosystem, which lets AI agents such as Claude connect to external tools and data sources, has grown to tens of thousands of publicly listed servers with minimal security vetting in many discovery directories. This study audits 511 randomly sampled, publicly listed MCP server repositories for three concrete risks: hardcoded credential exposure, over-scoped tool capability, and static indicators of indirect prompt-injection surface. Key findings: 79.3% of sampled repositories exercise at least one high-risk capability (network egress, filesystem mutation, shell execution, or dynamic code evaluation), and 47.0% specifically exercise shell execution, dynamic evaluation, or destructive filesystem capability. A naive secret-pattern scan flags 13.1% of repositories for candidate hardcoded credentials; full manual review — cross-checked against an independent rule-based second rater (83.3% agreement, Cohen's κ = 0.37) — reduces this to 3 repositories (0.59%) with a genuine hardcoded-credential concern, a 95.5% false-positive rate reported as a central methodological finding in its own right. Included in this record: Full paper (DOCX) Original static-analysis pipeline (scanner.py) for credential, capability, and injection-surface scanning Independent second-rater classifier (rater2.py) and agreement-statistics script (agreement.py) Redacted dataset of scan results across all 511 repositories (scan_results_public.json) Full list of sampled repositories (scanned_repo_ids.txt) Two repositories flagged during manual review as containing plausible real credentials are anonymized in the public dataset, consistent with the responsible-disclosure handling documented in the paper's Ethics section. Independent research, not affiliated with any institution or funded program.