Evaluating sparse and dense sentence encoders for vocabulary governance in emerging-domain taxonomies: A cross-family robustness approach

Registries in emerging technology domains accumulate semi-controlled short-text fields whose vocabularies fragment faster than governance bodies can normalize them. While methodological proposals — fine-tuned entity matching, instruction-tuned embeddings, and generative LLM classifiers — continue to advance benchmark accuracy, registry operators face a different problem: deciding which off-the-shelf neural or sparse similarity method to deploy under operational constraints (auditability, transparency, no fine-tuning budget, no labeled training data). No existing framework integrates structural taxonomy diagnostics, multi-method semantic-similarity benchmarking, and human validation into a single deployable pipeline that prioritises real-world auditability over benchmark optimization. We close this gap by presenting an empirical evaluation framework — structural, semantic, and validation — applied to a curated AI-blockchain corpus (N = 845 project cards) with cross-domain replication on MovieLens tags (N = 699). Five similarity operationalizations spanning sparse (TF-IDF) and dense neural (Sentence-BERT MiniLM, 22M; mpnet, 110M; E5-large, 335M parameters) representational families are compared. Within-family overlap exceeds cross-family overlap by 80–150% (means 0.57–0.59 vs. 0.23–0.32) consistently across all evaluated dense encoders (22M–335M parameters), establishing representational-family choice — rather than scale within the evaluated encoder set — as the first-order determinant of candidate sets. A two-wave annotation study (n = 196 pairs, Cohen’s κ = 0.844, blinded to stratum) confirms 100% synonym precision for cross-family-stable pairs in the validated stratum (27/27; 95% CI 0.872–1.000) versus 2% for negative controls and 18.8% sensitivity for a deployment-ready zero-shot configuration of Claude Opus 4.6 (Anthropic, 2026; accessed via Cursor IDE) on the same stratum. The resulting cross-family stability criterion enables registry operators to auto-merge validated synonyms at full precision using compact 22M-parameter models, without fine-tuning, without large-scale LLM infrastructure, and with full audit trails.